X-structured Steiner minimum tree construction method based on deep reinforcement learning

Abstract

This paper proposes an X-structured Steiner minimum tree construction method based on deep reinforcement learning. First, a routing topology initialization strategy is designed, using the prim algorithm to construct a minimum spanning tree to determine the pin topology. Second, a DRL algorithm-based XSMT problem processing strategy is proposed. This relaxes the 45° and 135° edges that appear in the XSMT problem and uses wire length as a reward, making the results obtained through DRL algorithm training more intuitive. Furthermore, a routing topology refinement strategy is designed. Because the results obtained through DRL algorithm training contain redundant wire lengths, the 45° and 135° edges are rotated into horizontal and vertical edges to eliminate duplicates. Compared to the pre-refinement results, this refinement strategy significantly optimizes wire lengths. When used in integrated circuit wiring design, this invention can produce routing solutions that minimize wire lengths.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated circuit computer-aided design, and in particular to an X-structured Steiner minimum tree construction method based on deep reinforcement learning. Background Art

[0002] For very large scale integration (VLSI), a crucial step is overall wiring between transistors. Traditional wiring methods often rely solely on the Manhattan structure, but this simple Manhattan structure wastes significant wiring resources and limits optimization space. This led to the development of the X-Architecture Steiner Minimal Tree (XSMT). The XSMT problem achieves this by finding Steiner points between pins to create a minimal tree. However, as the number of pins increases, the wiring problem becomes increasingly complex, and traditional solutions are no longer sufficient to meet the growing demand. Therefore, the XSMT problem has become a highly challenging problem.

[0003] Deep reinforcement learning (DRL) combines the perceptual capabilities of deep learning with the decision-making capabilities of reinforcement learning. These complementary strengths allow it to learn control strategies directly from high-dimensional raw data, making it an AI approach closer to human thinking. In the XSMT problem, DRL, during training, uses a reward function to evaluate all previously successful and unsuccessful network routing attempts to generate the "optimal" action for the routing's dynamic evolution, achieving the XSMT goal and ensuring more effective and accurate routing results. Summary of the Invention

[0004] The present invention proposes an X-structured Steiner minimum tree construction method based on deep reinforcement learning, which can obtain a wiring solution that minimizes line length.

[0005] The present invention adopts the following technical solutions.

[0006] A deep reinforcement learning-based X-structure Steiner minimum tree construction method is used for overall wiring between transistors in ultra-large-scale integrated circuits. The method completes the minimum tree wiring based on the X-structure Steiner minimum tree (XSMT) problem. The method uses deep reinforcement learning (DRL) to generate optimal actions for the dynamic evolution of the wiring state by analyzing all previous successful and unsuccessful network wiring attempts in its training reward function to achieve the goal of the XSMT problem. The method includes the following steps:

[0007] Step S1, data preprocessing and initialization stage: initializing the actual coordinates of the pins and mapping the actual coordinates of the pins to grid coordinates;

[0008] Step S2, wiring topology initialization phase: using the initialization strategy for the pins to generate a topology that is fixed in structure for subsequent steps;

[0009] Step S3, action selection phase based on deep reinforcement learning: design a routing selection strategy based on the DQN algorithm according to XSMT;

[0010] Step S4, wiring topology refinement stage: By optimizing the connection of interconnection lines, the wiring topology is optimized to select the optimal wiring topology.

[0011] In the XSMT problem, the wiring operations between two pins are divided into the following categories:

[0012] Action 1: First connect the starting pin and the pseudo Steiner point through a horizontal edge, and then connect the pseudo Steiner point and the ending pin through a vertical edge;

[0013] Action 2: First connect the starting pin and the pseudo-Steiner point through a vertical edge, and then connect the pseudo-Steiner point and the ending pin through a horizontal edge;

[0014] Action 3: First connect the starting pin and the pseudo Steiner point through a horizontal edge or a vertical edge, and then connect the pseudo Steiner point and the ending pin through a 45° edge or a 135° edge;

[0015] Action 4: First connect the starting pin and the pseudo Steiner point through a 45° edge or a 135° edge, and then connect the pseudo Steiner point and the ending pin through a horizontal edge or a vertical edge.

[0016] In the data preprocessing and initialization of step S1, the data input format is the number of pins and the actual coordinates of each pin; when there is no order between the input pins, the pin coordinate values are first sorted by x-coordinate and y-coordinate respectively to obtain the sorted pin numbers; then the actual coordinates of the pins are mapped to grid coordinates according to the serial numbers to obtain coordinate pairs of two groups of pins.

[0017] In XSMT, n pins have n-1 Steiner points. The Steiner points between two pins can be used to determine the wiring method between the two pins. The structure of each edge in the wiring diagram is described as follows:

[0018] [First pin coordinates, second pin coordinates, connection method between the two pins].

[0019] In the wiring topology initialization of step S2, the prim algorithm is used to construct a minimum spanning tree to determine the topological structure of the pins. That is, before using the DQN algorithm for training, a complete graph of a set of pins is first determined, and the multi-pin problem is converted into multiple two-pin problems. The Prim minimum spanning tree algorithm is used to determine the complete graph for initialization.

[0020] In step S3, in the action selection stage based on deep reinforcement learning, when the wiring result of the XSMT problem is obtained through training, the network structure used for training is defined as: each network is defined as two layers, the first layer has an input tensor and an output tensor, the input tensor is the size of the state space, and the output tensor is the number of neurons in the fully connected layer; the second layer has an input tensor and an output tensor, and the output tensor is the size of the action space; after initializing the weights and using a binary distribution to randomly generate the parameter values, the network obtains the value of the action selection performed in the current state.

[0021] In the DQN algorithm system of step S3: The action selection part of the DQN algorithm adopts the epsilon-greedy algorithm, which selects the action with the maximum action value in each state to make the resulting strategy better. At the same time, to prevent actions in certain states from not being selected, a random factor is set to ensure that there is a certain probability of using a random method to select actions in a small number of cases, so that all states have the opportunity to be visited;

[0022] The DQN algorithm experience replay part is used to store learning data. The target network is responsible for updating after reaching the specified number of learning times, and the evaluation network is responsible for updating once after each learning.

[0023] During the learning process, the network update logic is to randomly extract a certain amount of data from the experience replay part, package the data, and select the maximum action value under all states as the expected value of learning based on the action value of all states obtained.

[0024] During the training process of the DQN algorithm, the environment for each training is independent and completely consistent. Every time a new training session begins, the environment is reset, but the network memory of each training session is retained.

[0025] The training process of the DQN algorithm includes prediction and learning. The prediction process obtains the result of executing the action based on the currently selected action, that is, the eigenvalue matrix of the next state, and returns the reward for executing the action.

[0026] In the above learning, assuming that after a certain training, the accumulated information in the experience replay part reaches the capacity limit, the evaluation network will start learning. Each time the evaluation network learns, it will be updated once. The update idea is to randomly extract data from the experience replay part. After learning once, the data is randomly selected in the experience replay part for packaging.

[0027] The update idea of the DQN algorithm target network is as follows: the target network copies the parameters of the evaluation network and decides whether to complete learning by judging a specific value under the premise that the current amount of data reaches the capacity of the experience playback part;

[0028] The environment settings of the DQN algorithm are responsible for initializing the elements required for subsequent training and calculating the reward value based on different action selections in the entire DQN algorithm. The environment settings are grid initialization, training cycle initialization, and action selection.

[0029] Grid initialization: Build a matrix based on the number of all pins and all possible Steiner points. The overall routing direction is always from left to right and from bottom to top. Each matrix is built with two columns, which are used as markers for subsequent action selection. This allows the constructed matrix to fully represent the position and direction of all routing.

[0030] Initialize the training cycle: Initialize the state of each DQN search process, the structure of the Steiner minimum tree, the starting point of each search, grid information, and the number of search steps. At the same time, after each search, compare the optimal reward value obtained with the previous search to finally obtain the highest reward value;

[0031] Action selection: An action is selected for each state. Since XSMT has a non-Manhattan structure in its wiring method, the ratio of the difference between the horizontal coordinates and the vertical coordinates of two pins is used to distinguish whether the Manhattan edge is horizontal or vertical. At the same time, the reward value of the action is recorded by marking each time in the matrix. Since non-Manhattan edges cannot be simply marked by the matrix, a "relaxation" strategy is adopted. When selecting non-Manhattan edges, the matrix marking is only performed on Manhattan edges, and the reward value is also recorded. At the same time, the line length of the non-Manhattan edge is recorded as the reward value of the non-Manhattan edge. The calculation method of the reward value is shown in the following formula 1:

[0032]

[0033] Formula 1.

[0034] In the wiring topology refinement phase of step S4, a refinement strategy algorithm is set for the redundant wire lengths generated by the overlapping parts of the overall wiring structure, and the wiring tree obtained by training the DQN algorithm is optimized to make the wire length shorter;

[0035] The refining strategy algorithm specifically includes the following methods;

[0036] Method 1: Process the edges to obtain a comparable edge structure. Specifically, the line segment is decomposed into four types of edges according to four actions: horizontal edge, vertical edge, 45° edge, and 135° edge. At the same time, the 45° edge and the 135° edge are rotated clockwise into horizontal edge and vertical edge respectively. The four edges are stored as the following structure:

[0037] [horizontal edge vertical coordinate value, starting point coordinate value, end point coordinate value]

[0038] [horizontal coordinate value of the vertical edge, starting point coordinate value, end point coordinate value]

[0039] Rearrange the starting and ending points of each edge so that horizontal lines start on the left and end on the right, and vertical edges start at the bottom and end at the top. Compare edges of the same type to see if they meet the overlap conditions.

[0040] Method 2: Deduplication: Specifically, sort the edges in ascending order based on the four edge structures obtained in Method 1. If the first values are the same, sort them according to the second value.

[0041] For each type of edge, deduplication operation is performed separately. Since the sorting has been completed and the duplicate edges are adjacent, they can be compared in sequence. When the first value is the same, that is, the horizontal edges are on the same straight line, the overlapping part is related to the end point of the line segment, and the line segment with the smaller end point value is the overlapping part. When the first value is the same but the second value is different, since it has been sorted, the overlapping part is related to the starting point of the first line segment and the end point of the second line segment. When the starting point of the second line segment is smaller than the end point of the second line segment, the line segment between the two points is the overlapping part. The two line segments are merged to complete the deduplication operation. The same applies to vertical edges.

[0042] The process of refining the strategy algorithm includes the following steps;

[0043] Step A1: store four line segments in the minimum tree obtained by training the DQN algorithm according to the four action selections;

[0044] Step A2: Rotate the 45° side and the 135° side clockwise to become horizontal and vertical sides respectively;

[0045] Step A3: Adjust the starting and ending points of the line segment. The left point of the horizontal edge is the starting point and the right point is the ending point. The low point of the vertical edge is the starting point and the high point is the ending point.

[0046] Step A4: Sort each edge type, with horizontal edges sorted by ordinate and vertical edges sorted by abscissa;

[0047] Step A5: merge overlapping line segments, that is, horizontal edges and vertical edges on the same straight line will have overlapping parts;

[0048] Step A6: Calculate the result after line length refinement.

[0049] The XSMT problem processing strategy under the deep reinforcement learning (DRL) algorithm relaxes the 45° and 135° edges that appear in the XSMT problem and uses line length as the reward value, making the results obtained through DRL algorithm training more intuitive.

[0050] In step S2, a minimum spanning tree is constructed using the prim algorithm to determine the topology of the pins;

[0051] In step S4, in order to solve the problem of redundant line lengths in the results obtained through DRL algorithm training, the 45° edges and 135° edges in the results are rotated into horizontal edges and vertical edges to remove duplicates.

[0052] This paper proposes a method for constructing an X-structured Steiner minimum tree based on deep reinforcement learning. First, a routing topology initialization strategy is designed, using the prim algorithm to construct a minimum spanning tree to determine the pin topology. Second, a strategy for processing the XSMT problem using the DRL algorithm is proposed. This relaxes the 45° and 135° edges that appear in the XSMT problem and uses wire length as a reward, making the results obtained through DRL algorithm training more intuitive. Furthermore, a routing topology refinement strategy is designed. Because the results obtained through DRL algorithm training contain redundant wire lengths, the 45° and 135° edges are rotated into horizontal and vertical edges to remove duplicates. Compared with the results before refinement, the refinement strategy achieves a significant optimization effect on wire length.

[0053] When the present invention is used in integrated circuit wiring design, a wiring solution with minimized line length can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0055] Attachment Figure 1 This is a schematic diagram of the classification of the wiring action between two pins in the XSMT problem in the present invention;

[0056] Attachment Figure 2 This is a schematic diagram of mapping the actual coordinates of the pins into grid coordinates according to the serial numbers during data preprocessing and initialization;

[0057] Attachment Figure 3 This is a schematic diagram of the principle of the DQN algorithm;

[0058] Attachment Figure 4It is a flowchart of the algorithm for refining strategy. DETAILED DESCRIPTION

[0059] As shown in the figure, an X-structure Steiner minimum tree construction method based on deep reinforcement learning is used for the overall wiring between various transistors in a very large-scale integrated circuit. The minimum tree wiring is completed based on the X-structure Steiner minimum tree XSMT problem. The method uses deep reinforcement learning (DRL) to generate the optimal action for the dynamic evolution state of the wiring through all previous successful and unsuccessful network wiring attempts in its reward function during training to achieve the goal of the XSMT problem. The method includes the following steps:

[0060] Step S1, data preprocessing and initialization stage: initializing the actual coordinates of the pins and mapping the actual coordinates of the pins to grid coordinates;

[0061] Step S2, wiring topology initialization phase: using the initialization strategy for the pins to generate a topology that is fixed in structure for subsequent steps;

[0062] Step S3, action selection phase based on deep reinforcement learning: design a routing selection strategy based on the DQN algorithm according to XSMT;

[0063] Step S4, wiring topology refinement stage: By optimizing the connection of interconnection lines, the wiring topology is optimized to select the optimal wiring topology.

[0064] like Figure 1 As shown, in the XSMT problem, the wiring actions between two pins are divided into the following categories:

[0065] Action 1: First connect the starting pin and the pseudo Steiner point through a horizontal edge, and then connect the pseudo Steiner point and the ending pin through a vertical edge;

[0066] like Figure 1 As shown in (a), p1(x1, y1) and p2(x2, y2) are the starting and ending pins of an edge, respectively, where x1 < x2. Pin p1 is connected to the pseudo-Steiner point s via a vertical edge, and then point s is connected to pin p2 via a horizontal edge.

[0067] Action 2: First connect the starting pin and the pseudo-Steiner point through a vertical edge, and then connect the pseudo-Steiner point and the ending pin through a horizontal edge;

[0068] like Figure 1 As shown in (b), pin p1 is connected to the pseudo Steiner point s through a horizontal edge, and then point s is connected to pin p2 through a vertical edge.

[0069] Action 3: First connect the starting pin and the pseudo Steiner point through a horizontal edge or a vertical edge, and then connect the pseudo Steiner point and the ending pin through a 45° edge or a 135° edge;

[0070] like Figure 1 As shown in (c), pin p1 is connected to the pseudo Steiner point S through a horizontal edge, and then point S is connected to pin p2 through a 45° edge.

[0071] Action 4: First connect the starting pin and the pseudo Steiner point through a 45° edge or a 135° edge, and then connect the pseudo Steiner point and the ending pin through a horizontal edge or a vertical edge.

[0072] like Figure 1 As shown in (d), pin p1 is connected to the pseudo Steiner point S through a 45° edge, and then point S is connected to pin p2 through a horizontal edge.

[0073] In the data preprocessing and initialization of step S1, the data input format is the number of pins and the actual coordinates of each pin; when there is no order between the input pins, the pin coordinate values are first sorted by x-coordinate and y-coordinate respectively to obtain the sorted pin numbers; then the actual coordinates of the pins are mapped to grid coordinates according to the serial numbers to obtain coordinate pairs of two groups of pins.

[0074] like Figure 2 As shown, Figure 2 (a) is the actual coordinate diagram of the pin, Figure 2 (b) is a coordinate diagram of the pin's actual coordinates mapped to the grid.

[0075] In XSMT, n pins have n-1 Steiner points. The Steiner point between two pins can be used to determine the wiring method between the two pins. The connection method between the two pins is marked as 0 / 1 / 2 / 3. Therefore, the structure of each edge in the wiring diagram is described as:

[0076] [First pin coordinates, second pin coordinates, connection method between the two pins].

[0077] In the wiring topology initialization of step S2, the prim algorithm is used to construct a minimum spanning tree to determine the topological structure of the pins. That is, before using the DQN algorithm for training, a complete graph of a set of pins is first determined, and the multi-pin problem is converted into multiple two-pin problems. The Prim minimum spanning tree algorithm is used to determine the complete graph for initialization.

[0078] In step S3, in the action selection stage based on deep reinforcement learning, when the wiring result of the XSMT problem is obtained through training, the network structure used for training is defined as: each network is defined as two layers, the first layer has an input tensor and an output tensor, the input tensor is the size of the state space, and the output tensor is the number of neurons in the fully connected layer; the second layer has an input tensor and an output tensor, and the output tensor is the size of the action space; after initializing the weights and using a binary distribution to randomly generate the parameter values, the network obtains the value of the action selection performed in the current state.

[0079] like Figure 3 As shown in the DQN algorithm system of step S3: the action selection part of the DQN algorithm adopts the epsilon-greedy algorithm, which selects the action with the maximum action value in each state to make the obtained strategy better. At the same time, in order to avoid actions in certain states not being selected, a random factor is set to ensure that there is a certain probability of taking a random method to select actions in a small number of cases, so that all states have the opportunity to be visited;

[0080] The DQN algorithm experience replay part is used to store learning data. The target network is responsible for updating after reaching the specified number of learning times, and the evaluation network is responsible for updating once after each learning.

[0081] During the learning process, the network update logic is to randomly extract a certain amount of data from the experience replay part, package the data, and select the maximum action value under all states as the expected value of learning based on the action value of all states obtained.

[0082] During the training process of the DQN algorithm, the environment for each training is independent and completely consistent. Every time a new training session begins, the environment is reset, but the network memory of each training session is retained.

[0083] The training process of the DQN algorithm includes prediction and learning. The prediction process obtains the result of executing the action based on the currently selected action, that is, the eigenvalue matrix of the next state, and returns the reward for executing the action.

[0084] In the above learning, assuming that after a certain training, the accumulated information in the experience replay part reaches the capacity limit, the evaluation network will start learning. Each time the evaluation network learns, it will be updated once. The update idea is to randomly extract data from the experience replay part. After learning once, the data is randomly selected in the experience replay part for packaging.

[0085] The update idea of the DQN algorithm target network is as follows: the target network copies the parameters of the evaluation network and decides whether to complete learning by judging a specific value under the premise that the current amount of data reaches the capacity of the experience playback part;

[0086] The environment settings of the DQN algorithm are responsible for initializing the elements required for subsequent training and calculating the reward value based on different action selections in the entire DQN algorithm. The environment settings are grid initialization, training cycle initialization, and action selection.

[0087] Grid initialization: Build a matrix based on the number of all pins and all possible Steiner points. The overall routing direction is always from left to right and from bottom to top. Each matrix is built with two columns, which are used as markers for subsequent action selection. This allows the constructed matrix to fully represent the position and direction of all routing.

[0088] Initialize the training cycle: Initialize the state of each DQN search process, the structure of the Steiner minimum tree, the starting point of each search, grid information, and the number of search steps. At the same time, after each search, compare the optimal reward value obtained with the previous search to finally obtain the highest reward value;

[0089] Action selection: An action is selected for each state. Since XSMT has a non-Manhattan structure in its wiring method, the ratio of the difference between the horizontal coordinates and the vertical coordinates of two pins is used to distinguish whether the Manhattan edge is horizontal or vertical. At the same time, the reward value of the action is recorded by marking each time in the matrix. Since non-Manhattan edges cannot be simply marked by the matrix, a "relaxation" strategy is adopted. When selecting non-Manhattan edges, the matrix marking is only performed on Manhattan edges, and the reward value is also recorded. At the same time, the line length of the non-Manhattan edge is recorded as the reward value of the non-Manhattan edge. The calculation method of the reward value is shown in the following formula 1:

[0090]

[0091] Formula 1.

[0092] In the wiring topology refinement phase of step S4, a refinement strategy algorithm is set for the redundant wire lengths generated by the overlapping parts of the overall wiring structure, and the wiring tree obtained by training the DQN algorithm is optimized to make the wire length shorter;

[0093] like Figure 4 As shown, the refining strategy algorithm specifically includes the following methods:

[0094] Method 1: Process the edges to obtain a comparable edge structure. Specifically, the line segment is decomposed into four types of edges according to four actions: horizontal edge, vertical edge, 45° edge, and 135° edge. At the same time, the 45° edge and the 135° edge are rotated clockwise into horizontal edge and vertical edge respectively. The four edges are stored as the following structure:

[0095] [horizontal edge vertical coordinate value, starting point coordinate value, end point coordinate value]

[0096] [horizontal coordinate value of the vertical edge, starting point coordinate value, end point coordinate value]

[0097] Rearrange the starting and ending points of each edge so that horizontal lines start on the left and end on the right, and vertical edges start at the bottom and end at the top. Compare edges of the same type to see if they meet the overlap conditions.

[0098] Method 2: Deduplication: Specifically, sort the edges in ascending order based on the four edge structures obtained in Method 1. If the first values are the same, sort them according to the second value.

[0099] For each type of edge, deduplication operation is performed separately. Since the sorting has been completed and the duplicate edges are adjacent, they can be compared in sequence. When the first value is the same, that is, the horizontal edges are on the same straight line, the overlapping part is related to the end point of the line segment, and the line segment with the smaller end point value is the overlapping part. When the first value is the same but the second value is different, since it has been sorted, the overlapping part is related to the starting point of the first line segment and the end point of the second line segment. When the starting point of the second line segment is smaller than the end point of the second line segment, the line segment between the two points is the overlapping part. The two line segments are merged to complete the deduplication operation. The same applies to vertical edges.

[0100] The process of refining the strategy algorithm includes the following steps;

[0101] Step A1: store four line segments in the minimum tree obtained by training the DQN algorithm according to the four action selections;

[0102] Step A2: Rotate the 45° side and the 135° side clockwise to become horizontal and vertical sides respectively;

[0103] Step A3: Adjust the starting and ending points of the line segment. The left point of the horizontal edge is the starting point and the right point is the ending point. The low point of the vertical edge is the starting point and the high point is the ending point.

[0104] Step A4: Sort each edge type, with horizontal edges sorted by ordinate and vertical edges sorted by abscissa;

[0105] Step A5: merge overlapping line segments, that is, horizontal edges and vertical edges on the same straight line will have overlapping parts;

[0106] Step A6: Calculate the result after line length refinement.

[0107] The XSMT problem processing strategy under the deep reinforcement learning (DRL) algorithm relaxes the 45° and 135° edges that appear in the XSMT problem and uses line length as the reward value, making the results obtained through DRL algorithm training more intuitive.

[0108] In step S2, a minimum spanning tree is constructed using the prim algorithm to determine the topology of the pins;

[0109] In step S4, in order to solve the problem of redundant line lengths in the results obtained through DRL algorithm training, the 45° edges and 135° edges in the results are rotated into horizontal edges and vertical edges to remove duplicates.

Claims

1. An X-structured Steiner minimum tree construction method based on deep reinforcement learning is used for the overall wiring between transistors in very large-scale integrated circuits. The minimum tree wiring is completed based on the X-structured Steiner minimum tree (XSMT) problem. The characteristics are: The method uses deep reinforcement learning (DRL) to generate the best action for the dynamic evolution state of the wiring through all previous successful and unsuccessful network wiring attempts of its reward function during training to achieve the goal of the XSMT problem; the method includes the following steps: Step S1, data preprocessing and initialization stage: initializing the actual coordinates of the pins and mapping the actual coordinates of the pins to grid coordinates; Step S2, wiring topology initialization phase: using the initialization strategy for the pins to generate a topology that is fixed in structure for subsequent steps; Step S3, action selection phase based on deep reinforcement learning: design a routing selection strategy based on the DQN algorithm according to XSMT; Step S4, wiring topology refinement stage: optimizing the wiring topology by optimizing the connections of the interconnects to select the optimal wiring topology; In the wiring topology refinement phase of step S4, a refinement strategy algorithm is set for the redundant wire lengths generated by the overlapping parts of the overall wiring structure, and the wiring tree obtained by training the DQN algorithm is optimized to make the wire length shorter; The refining strategy algorithm specifically includes the following methods; Method 1: Process the edges to obtain a comparable edge structure. Specifically, the line segment is decomposed into four types of edges according to four actions: horizontal edge, vertical edge, 45° edge, and 135° edge. At the same time, the 45° edge and the 135° edge are rotated clockwise into horizontal edge and vertical edge respectively. The four edges are stored as the following structure: [horizontal edge vertical coordinate value, starting point coordinate value, end point coordinate value] [horizontal coordinate value of the vertical edge, starting point coordinate value, end point coordinate value] Rearrange the starting and ending points of each edge so that horizontal lines start on the left and end on the right, and vertical edges start at the bottom and end at the top. Compare edges of the same type to see if they meet the overlap conditions. Method 2: Deduplication: Specifically, sort the edges in ascending order based on the four edge structures obtained in Method 1. If the first values are the same, sort them according to the second value. For each type of edge, deduplication operation is performed separately. Since the sorting has been completed and the duplicate edges are adjacent, they can be compared in sequence. When the first value is the same, that is, the horizontal edges are on the same straight line, the overlapping part is related to the end point of the line segment, and the line segment with the smaller end point value is the overlapping part. When the first value is the same but the second value is different, since it has been sorted, the overlapping part is related to the starting point of the first line segment and the end point of the second line segment. When the starting point of the second line segment is smaller than the end point of the second line segment, the line segment between the two points is the overlapping part. The two line segments are merged to complete the deduplication operation. The same applies to vertical edges.

2. The method for constructing an X-structured Steiner minimum tree based on deep reinforcement learning according to claim 1, characterized in that: In the XSMT problem, the wiring operations between two pins are divided into the following categories: Action 1: First connect the starting pin and the pseudo Steiner point through a horizontal edge, and then connect the pseudo Steiner point and the ending pin through a vertical edge; Action 2: First connect the starting pin and the pseudo-Steiner point through a vertical edge, and then connect the pseudo-Steiner point and the ending pin through a horizontal edge; Action 3: First connect the starting pin and the pseudo Steiner point through a horizontal edge or a vertical edge, and then connect the pseudo Steiner point and the ending pin through a 45° edge or a 135° edge; Action 4: First connect the starting pin and the pseudo Steiner point through a 45° edge or a 135° edge, and then connect the pseudo Steiner point and the ending pin through a horizontal edge or a vertical edge.

3. The method for constructing an X-structured Steiner minimum tree based on deep reinforcement learning according to claim 1, characterized in that: In the data preprocessing and initialization step S1, the data input format is the pin number and the actual coordinates of each pin. If there is no order between the input pins, the pin coordinate values are first sorted by x-coordinate and y-coordinate to obtain the sorted pin numbers. Then, the actual coordinates of the pins are mapped to grid coordinates according to the numbers to obtain two sets of pin coordinate pairs. In XSMT, n pins have n-1 Steiner points. The Steiner points between two pins can be used to determine the wiring method between the two pins. The structure of each edge in the wiring diagram is described as follows: [First pin coordinates, second pin coordinates, connection method between the two pins].

4. The method for constructing an X-structured Steiner minimum tree based on deep reinforcement learning according to claim 1, wherein: In the wiring topology initialization of step S2, the prim algorithm is used to construct a minimum spanning tree to determine the topological structure of the pins. That is, before using the DQN algorithm for training, a complete graph of a set of pins is first determined, and the multi-pin problem is converted into multiple two-pin problems. The Prim minimum spanning tree algorithm is used to determine the complete graph for initialization.

5. The method for constructing an X-structured Steiner minimum tree based on deep reinforcement learning according to claim 1, wherein: In step S3, in the action selection stage based on deep reinforcement learning, when the wiring result of the XSMT problem is obtained through training, the network structure used for training is defined as: each network is defined as two layers, the first layer has an input tensor and an output tensor, the input tensor is the size of the state space, and the output tensor is the number of neurons in the fully connected layer; the second layer has an input tensor and an output tensor, and the output tensor is the size of the action space; after initializing the weights and using a binary distribution to randomly generate the parameter values, the network obtains the value of the action selection performed in the current state.

6. The method for constructing an X-structured Steiner minimum tree based on deep reinforcement learning according to claim 5, characterized in that: In the DQN algorithm system of step S3: The action selection part of the DQN algorithm adopts the epsilon-greedy algorithm, which selects the action with the maximum action value in each state to make the resulting strategy better. At the same time, to prevent actions in certain states from not being selected, a random factor is set to ensure that there is a certain probability of using a random method to select actions in a small number of cases, so that all states have the opportunity to be visited; The DQN algorithm experience replay part is used to store learning data. The target network is responsible for updating after reaching the specified number of learning times, and the evaluation network is responsible for updating once after each learning. During the learning process, the network update logic is to randomly extract a certain amount of data from the experience replay part, package the data, and select the maximum action value under all states as the expected value of learning based on the action value of all states obtained.

7. The method for constructing an X-structured Steiner minimum tree based on deep reinforcement learning according to claim 6, characterized in that: During the training process of the DQN algorithm, the environment for each training is independent and completely consistent. Every time a new training session begins, the environment is reset, but the network memory of each training session is retained. The training process of the DQN algorithm includes prediction and learning. The prediction process obtains the result of executing the action based on the currently selected action, that is, the eigenvalue matrix of the next state, and returns the reward for executing the action. In the above learning, assuming that after a certain training, the accumulated information in the experience replay part reaches the capacity limit, the evaluation network will start learning. Each time the evaluation network learns, it will be updated once. The update idea is to randomly extract data from the experience replay part. After learning once, the data is randomly selected in the experience replay part for packaging. The update idea of the DQN algorithm target network is as follows: the target network copies the parameters of the evaluation network and decides whether to complete learning by judging a specific value under the premise that the current amount of data reaches the capacity of the experience playback part; The environment settings of the DQN algorithm are responsible for initializing the elements required for subsequent training and calculating the reward value based on different action selections. The environment settings are grid initialization, training cycle initialization, and action selection. Specifically: Grid initialization: Build a matrix based on the number of all pins and all possible Steiner points. The overall routing direction is always from left to right and from bottom to top. Each matrix is built with two columns, which are used as markers for subsequent action selection. This allows the constructed matrix to fully represent the position and direction of all routing. Initialize the training cycle: Initialize the state of each DQN search process, the structure of the Steiner minimum tree, the starting point of each search, grid information, and the number of search steps. At the same time, after each search, compare the optimal reward value obtained with the previous search to finally obtain the highest reward value; Action selection: An action is selected for each state. Since XSMT has a non-Manhattan structure in its wiring method, the ratio of the difference between the horizontal coordinates and the vertical coordinates of two pins is used to distinguish whether the Manhattan edge is horizontal or vertical. At the same time, the reward value of the action is recorded by marking each time in the matrix. Since non-Manhattan edges cannot be simply marked by the matrix, a "relaxation" strategy is adopted. When selecting non-Manhattan edges, the matrix marking is only performed on Manhattan edges, and the reward value is also recorded. At the same time, the line length of the non-Manhattan edge is recorded as the non-Manhattan edge reward value. The calculation method of the reward value is shown in the following formula 1:

8. The method for constructing an X-structured Steiner minimum tree based on deep reinforcement learning according to claim 1, wherein: The process of refining the strategy algorithm includes the following steps; Step A1: store four line segments in the minimum tree obtained by training the DQN algorithm according to the four action selections; Step A2: Rotate the 45° side and the 135° side clockwise to become horizontal and vertical sides respectively; Step A3: Adjust the starting and ending points of the line segment. The left point of the horizontal edge is the starting point and the right point is the ending point. The low point of the vertical edge is the starting point and the high point is the ending point. Step A4: Sort each edge type, with horizontal edges sorted by ordinate and vertical edges sorted by abscissa; Step A5: merge overlapping line segments, that is, horizontal edges and vertical edges on the same straight line will have overlapping parts; Step A6: Calculate the result after line length refinement.

9. The method for constructing an X-structured Steiner minimum tree based on deep reinforcement learning according to claim 1, characterized in that: The XSMT problem processing strategy under the deep reinforcement learning (DRL) algorithm relaxes the 45° and 135° edges that appear in the XSMT problem and uses line length as the reward value, making the results obtained through DRL algorithm training more intuitive. In step S2, a minimum spanning tree is constructed using the prim algorithm to determine the topology of the pins; In step S4, in order to solve the problem of redundant line lengths in the results obtained through DRL algorithm training, the 45° edges and 135° edges in the results are rotated into horizontal edges and vertical edges to remove duplicates.

Images