Method and system for reinforcing learning algorithm to assist A star algorithm in wiring in PCB environment
By introducing the Q-Learning algorithm and double attenuation strategy into the A-star algorithm, the problem that the A-star algorithm is easily trapped in the local optimal solution in the PCB environment is solved, and more efficient and accurate routing path solving is achieved.
Patent Information
- Application Number
- CN202411443883.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-16
- Publication Date
- 2025-02-07
AI Technical Summary
Traditional A-star algorithms are prone to fall into local optimal solutions when routing in PCB environments. Especially when the heuristic function is not completely accurate, it is difficult to find the global optimal solution, resulting in undesirable routing results.
The reinforcement learning algorithm Q-Learning and double attenuation strategy were introduced to optimize the path-finding process of the A-star algorithm. Through iterative training of the Q-Learning algorithm and dynamic adjustment of the dual attenuation strategy, the algorithm is prevented from falling into local optimal solutions and improved wiring efficiency.
It effectively prevents the local optimal solution problem of A-star algorithm when wiring in PCB environment, improves wiring efficiency and accuracy, and ensures that reasonable paths can be found in complex environments.
Smart Images
Figure CN119397988A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of integrated circuit electronic design automation, and in particular to a method and system for routing in a PCB environment using a reinforcement learning algorithm assisted by an A-star algorithm. Background Art
[0002] Printed Circuit Board (PCB) is widely used in consumer electronics, medical equipment, automotive industry, national defense, aerospace and other related fields. The PCB board in electronic equipment is to electrically interconnect various electronic components according to the design intent through physical wires, thereby realizing the specified function. PCB wiring is a crucial part of electronic product design. Wiring planning algorithms include maze algorithm, genetic algorithm, ant colony algorithm, etc. The most classic wiring algorithm is the A-star algorithm.
[0003] With the rapid development of integrated circuit technology and the increasing number of complex applications, integrated circuits are becoming more and more complex, which makes the optimization of the overall layout and routing planning of PCB boards more important. The traditional automatic routing algorithm using the A-star algorithm has a prominent problem: since the classic A-star algorithm is a greedy search algorithm, it tends to move in a seemingly optimal direction. However, this local optimal decision may cause the algorithm to fail to find the global optimal solution, especially when the heuristic function is not completely accurate or misleading, there is a greater probability of unsatisfactory routing results, which leads to a greater probability of failing to find a reasonable path when performing routing planning in a large-scale PCB environment with complex obstacle layouts. Summary of the invention
[0004] The purpose of the present invention is to provide a method and system for routing in a PCB environment using a reinforcement learning algorithm to assist the A-star algorithm, which can assist the routing process of the A-star algorithm through a double attenuation strategy, thereby preventing the routing process from falling into a local optimal solution and improving the routing efficiency of the PCB environment.
[0005] To achieve the above object, the present invention provides the following solutions:
[0006] A method for routing in a PCB environment using a reinforcement learning algorithm to assist an A-star algorithm, comprising:
[0007] The A-star algorithm is optimized by using the Q-Learning algorithm and the double decay strategy; the A-star algorithm includes a total cost function;
[0008] In a PCB simulation wiring environment, the Q-Learning algorithm and the double attenuation strategy are used to assist the A-star algorithm to solve the wiring path and obtain the shortest path; the PCB simulation wiring environment is constructed according to the set starting node, target node and obstacle; the PCB wiring environment is a grid environment.
[0009] Optionally, the total cost function is the sum of an actual cost function and a heuristic function.
[0010] Optionally, the use of the Q-Learning algorithm and the double attenuation strategy to assist the A-star algorithm in solving the wiring path specifically includes:
[0011] In the PCB simulation wiring environment, the Q-Learning algorithm is first used to assist the A-star algorithm for wiring, and then the double attenuation strategy is used to optimize the wiring path to obtain the shortest path.
[0012] Optionally, it is characterized in that the process of solving the wiring path is: using the total cost function to solve the wiring path nodes in the PCB simulation environment to obtain the shortest path.
[0013] Optionally, the calculation formula of the reward value of an action during the training process of the Q-Learning algorithm is:
[0014] Q(s,a)←Q(s,a)+α[r+ymax a′ Q(s′, a′)-Q(s, a)]
[0015] Among them, α is the learning rate, which is used to determine the extent to which newly acquired information will overwrite old information. If the value is 0, it means that the agent stops learning, and if the value is 1, it means that the agent only considers the latest information; γ is the exploration factor, which is used to indicate the impact of further rewards on the current action; s is the state, a is the action, r is the reward, and max a′ Q(s′, a′) means selecting the combination of all states and actions that can be selected in the next step with the highest reward value as the next exploration position, and then obtaining the updated reward value.
[0016] Optionally, the using of the Q-Learning algorithm to assist the A-star algorithm in routing specifically includes:
[0017] Initialize Q-table: First, create a Q-table to store the Q-value of each state and action; the size of the Q-table depends on the number of states and actions in the environment; initially, initialize all Q-values to 0 or random values;
[0018] Select action: At each time step, an action is selected based on the current state. The specific process is: using the epsilon-greedy strategy, randomly select an action with a probability of ε, and select the action with the highest Q-value with a probability of 1-ε;
[0019] Execute actions: Apply the selected action to the environment, and get rewards and the next state based on the feedback from the environment;
[0020] Update Q-value: Use the Q-learning update formula to update Q-value based on the reward and next state.
[0021] Iteration: Repeat the steps of "select action", "execute action" and "update Q-value" until the stopping condition is reached; the stopping condition is reaching the maximum number of iterations or converging to a stable value;
[0022] Path selection: Select the optimal path based on the trained Q-table. The specific process is: starting from the starting state, select the action with the highest Q-value, transfer to the next state, and then repeat the process until the target state is reached to obtain the output path.
[0023] Optionally, it is characterized in that the double decay strategy specifically includes: during the operation of the Q-learning algorithm, as the number of training times increases, the values of the exploration factor and the learning rate are dynamically reduced.
[0024] The present invention also provides a system for routing in a PCB environment using a reinforcement learning algorithm to assist an A-star algorithm, including:
[0025] An algorithm optimization unit, used for optimizing the A-star algorithm using the Q-Learning algorithm and the double decay strategy; the A-star algorithm includes a total cost function;
[0026] The wiring path determination unit is used to solve the wiring path in a PCB simulation wiring environment by using the Q-Learning algorithm and the double attenuation strategy to assist the A-star algorithm to obtain the shortest path; the PCB simulation wiring environment is constructed according to the set starting node, target node and obstacle; the PCB wiring environment is a grid environment.
[0027] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0028] The present invention discloses a method and system for routing in a PCB environment using a reinforcement learning algorithm to assist an A-star algorithm, the method comprising optimizing the A-star algorithm using a Q-Learning algorithm and a dual attenuation strategy; the A-star algorithm selects path nodes using the calculation of a total cost function; in a PCB simulation routing environment, the A-star algorithm is assisted by the Q-Learning algorithm and the dual attenuation strategy to solve the routing path and obtain the shortest path. The present invention assists the routing process of the classic A-star algorithm through a reinforcement learning algorithm, and optimizes the performance through a dual attenuation strategy, thereby preventing the algorithm from falling into a local optimal solution during the routing process, accelerating the algorithm training time, and thus improving routing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0030] Figure 1 This is a flow chart of a method for the reinforcement learning algorithm of the present invention to assist the A-star algorithm in routing in a PCB environment;
[0031] Figure 2 Schematic diagram of the process of the Q-Learning algorithm in the PCB wiring environment of this embodiment;
[0032] Figure 3 Schematic diagram of simulating PCB wiring environment in this embodiment;
[0033] Figure 4 This is a schematic diagram of the result information of the classic A-star algorithm outputting the local optimal solution in this embodiment;
[0034] Figure 5 This is a schematic diagram of wiring result information completed by the Q-Learning algorithm assisting the A-star algorithm in this embodiment;
[0035] Figure 6 This is a schematic diagram of wiring result information completed by the Q-Learning algorithm and the double attenuation strategy combined with the auxiliary A-star algorithm in this embodiment. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] The purpose of the present invention is to provide a method and system for routing in a PCB environment using a reinforcement learning algorithm to assist the A-star algorithm, which can assist the routing process of the A-star algorithm through a double attenuation strategy, thereby preventing the routing process from falling into a local optimal solution and improving the routing efficiency of the PCB environment.
[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0039] like Figure 1-Figure 6As shown, the present invention provides a method for routing in a PCB environment using a reinforcement learning algorithm to assist an A-star algorithm, including:
[0040] Step 100: Optimizing the A-star algorithm using the Q-Learning algorithm and the double decay strategy; the A-star algorithm includes a total cost function;
[0041] Step 200: In a PCB simulation wiring environment, the Q-Learning algorithm and the double attenuation strategy are used to assist the A-star algorithm to solve the wiring path and obtain the shortest path; the PCB simulation wiring environment is constructed according to the set starting node, target node and obstacle; the PCB wiring environment is a grid environment.
[0042] By introducing the Q-Learning algorithm and the double attenuation strategy, the performance of the algorithm during wiring in the PCB environment is optimized; the A-star optimization algorithm is constructed by combining the Q-Learning algorithm with the multiple path output strategy. Among them, the Q-Learning algorithm is an artificial intelligence algorithm that outputs the shortest path through iterative training, and the training process can traverse all the paths that can be routed between the starting node and the target node; the double attenuation strategy prevents the wiring process from falling into the local optimal solution and improves the wiring efficiency by dynamically adjusting the values of the exploration factor and the learning rate in the Q-Learning algorithm.
[0043] The Q-Learning algorithm and the double decay strategy include:
[0044] During the wiring process of the Q-Learning algorithm, each different action corresponds to a reward value. The Q-value is updated based on the reward of each action in the training process and its corresponding next state, using the Q-learning update formula to update the Q-value. The double decay strategy is to dynamically reduce the values of the exploration factor and the learning rate as the number of training times increases during the operation of the Q-learning algorithm. The double decay strategy can prevent the algorithm from falling into a local optimal solution during the path finding process and accelerate the training process of the algorithm.
[0045] The process in the PCB simulation wiring environment specifically includes:
[0046] Build an experimental platform and set the starting node and target node in the obstacle environment to start the training process of the Q-Learning algorithm. First, create a Q-table to store the Q-value of each state and action. The size of the Q-table depends on the number of states and actions in the environment. Initially, all Q-values can be initialized to 0 or random values; at each time step, an action is selected according to the current state. The epsilon-greedy strategy is usually adopted, that is, an action is randomly selected with a probability of ε, and the action with the highest Q-value is selected with a probability of 1-ε; for each action:
[0047] Apply the selected action to the environment and observe the feedback from the environment, including rewards and the next state; use the Q-learning update formula to update the Q-value based on the observed rewards and the next state; repeat the steps of selecting actions, executing actions, and updating Q-value until the stopping condition is reached, such as reaching the maximum number of iterations or converging to a stable value; once learning is complete, the optimal path can be selected based on the trained Q-table. Starting from the starting state, select the action with the highest Q-value, transfer to the next state, and then repeat the process until the target state is reached.
[0048] In addition, the A-star optimization algorithm also includes:
[0049] The shortcomings of the path-finding method in the classic A-star algorithm are analyzed; the shortcomings include: since the classic A-star algorithm is a greedy search algorithm, it tends to move in a seemingly optimal direction. However, this local optimal decision may cause the algorithm to fail to find the global optimal solution, especially when the heuristic function is not completely accurate or misleading, there is a greater probability of undesirable routing results, which is an undesirable effect.
[0050] Based on the above technical solution, the following embodiments are provided.
[0051] In this embodiment, the main optimization idea is: the classic A-star algorithm is mainly used to solve the shortest path in static grids, such as for the automatic routing function in PCB. The selection of nodes when the A-star algorithm is running in the PCB routing environment is compared based on the values calculated by the total cost function, and the node with the smallest value is selected as the new landing point and finally connected into a new path. The local optimal decision of the A-star algorithm may cause the algorithm to be unable to find the global optimal solution, so the Q-Learning algorithm and the double attenuation strategy are introduced to assist the routing process of the A-star algorithm and optimize the performance of the improved algorithm in the PCB environment.
[0052] The Q-learning is a reinforcement learning algorithm used to solve problems in Markov decision processes (MDPs). It learns a strategy so that the agent can maximize long-term rewards in the environment. Q-learning is a value iteration-based method whose core idea is to learn the value function of each state-action pair, namely the Q-value function, to guide the agent to make decisions in the environment. The corresponding algorithm flowchart is as follows: Figure 2 As shown. The Q-value function refers to learning a Q-value function Q( s , a), it represents the expected value of the long-term reward obtained by taking action a in state s. The Q-value function defines the "benefit" of the agent taking each action in each state and is the basis for the agent to make decisions.
[0053] The double decay strategy utilizes the training process of reinforcement learning. During the operation of the Q-learning algorithm, as the number of training times increases, the exploration factor and the learning rate are dynamically reduced. Reducing the learning rate can improve the stability of the algorithm during the training process, especially when it is close to convergence. A smaller learning rate helps prevent the parameters from oscillating near the minimum value, making it easier for the model to converge to the optimal solution. Gradually reducing epsilon helps prevent the agent from falling into the local optimal solution too early. However, when the value is small, the agent is more inclined to use known information, but will not completely give up exploration. The ε-greedy strategy maintains a certain exploratory nature so that the agent can still occasionally try new actions to discover potential better paths, thus helping to prevent the agent from falling into the local optimal solution too early. The double decay strategy can prevent the algorithm from falling into the local optimal solution during the path-finding process and improve the efficiency of the algorithm.
[0054] Therefore, a specific experiment is conducted in a grid virtual map environment, but in order to better understand the solution provided by this application, it is necessary to understand the following:
[0055] Electronic Design Automation (EDA) is an upstream industry in the integrated circuit industry chain. EDA design software covers all processes of integrated circuit chip functional design, synthesis, verification, physical design (including layout, wiring, layout, design rule checking, etc.) and packaging testing. It is an essential and most important software tool for integrated circuit design and is known as the "mother of chips". EDA includes tools such as circuit design and simulation, printed circuit board (PCB) design, integrated circuit design, IP core design and programmable logic device design. PCB is a bridge that carries electronic components and connects circuits. It is known as the "mother of electronic products" and is widely used in communications, consumer electronics, automotive electronics, industrial control, medical equipment, national defense, aerospace and other fields.
[0056] The PCB board in the electronic device is to electrically interconnect the electronic components according to the design intent through physical wires, thereby realizing the specified functions. PCB wiring is a crucial part of electronic product design, and its goals can be summarized into two levels: according to the description of the circuit connection relationship, under the conditions of meeting the requirements of the design and process rules and meeting the electrical performance, 100% of the required electrical interconnection is completed within the limited area; and on the premise of completing the wiring, the PCB wiring template is also to further optimize the wiring results, so that the required PCB area is minimized and the functions and performance of the PCB are better. Among the PCB wiring algorithms, the most common and widely used algorithm is the A-star algorithm.
[0057] With the rapid development of integrated circuit technology and the increasing number of complex applications, integrated circuits are becoming more and more complex. In large-scale wiring environments, the classic A-star algorithm's automatic wiring algorithm has a prominent problem: since the classic A-star algorithm is a greedy search algorithm, it tends to move in a seemingly optimal direction. However, this local optimal decision may cause the algorithm to fail to find the global optimal solution, especially when the heuristic function is not completely accurate or misleading, there is a greater probability of unsatisfactory wiring results, which is an unsatisfactory effect.
[0058] The definitions related to the embodiment are as follows:
[0059] (1) PCB simulation wiring environment: The three major elements in PCB wiring include the starting node, the target node and the obstacle. The simulation environment of the example diagram of the present invention is as follows: Figure 3 As shown, the environment is built on the Pycharm platform. By setting the grid properties, the three elements mentioned above are set in these grids to distinguish between unrestricted spaces and obstacles, among which obstacles cannot be explored and cannot be passed during the routing process.
[0060] (2) A-star algorithm: The A-star algorithm combines the characteristics of the Dijkstra algorithm and the breadth-first search algorithm. It solves the shortest path by comparing the total cost function. The routing results of the classic A-star algorithm are as follows: Figure 4 As shown, the A-star algorithm can only output one shortest path.
[0061] (3) Application of Q-Learning algorithm in grid map: The Q-Learning algorithm uses the value iteration method to learn the value function of each state-action pair, namely the Q-value function, to guide the intelligent agent to make decisions in the environment and assist the A-star algorithm in the routing process of some short-distance lines with high requirements. The routing results are shown in Figure 2. Figure 5As shown, the Q-value function is updated during each training process, and the algorithm flow described above is continuously repeated in the grid map, and finally a wiring result in a specific environment can be obtained. The Q-value function calculation method of the Q-Learning algorithm in the present invention is:
[0062] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s′, a′)-Q(s, a)]
[0063] Where α is the learning rate, which determines the extent to which newly acquired information will overwrite old information. A value of 0 will stop the agent from learning, while a value of 1 will cause the agent to only consider the latest information. γ is the exploration factor, which indicates the impact of further rewards on the current action. s is the state, a is the action, r is the reward, and max a′ Q(s′, a′) means selecting the one with the highest reward value from all the combinations of states and actions that can be selected in the next step as the next exploration position. From this formula, we can get the updated reward value.
[0064] The specific implementation process is as follows:
[0065] This embodiment optimizes the performance of the algorithm after the local optimal decision of the classic A-star algorithm, which may cause the algorithm to be unable to find the global optimal solution, and the algorithm after artificial intelligence assistance. The application of the A-star algorithm in PCB automatic wiring planning tends to move in a seemingly optimal direction. However, this local optimal decision may cause the algorithm to be unable to find the global optimal solution, especially when the heuristic function is not completely accurate or misleading, there is a greater probability of undesirable wiring results. In order to solve this problem, the present invention introduces the Q-Learning algorithm and the double attenuation strategy to assist the wiring process of the A-star algorithm, so that the improved algorithm can output the shortest path information faster and more accurately, prevent the algorithm from falling into the local optimal solution during the path-finding process, and improve the efficiency of the algorithm.
[0066] The basic principle of the A-star algorithm is to select the next landing point by comparing the size of the total cost value corresponding to each node and loop this process to obtain the final path. The local optimal decision may cause the algorithm to be unable to find the global optimal solution. If you want the A-star algorithm to output better results, you need to use the characteristics of the Q-Learning algorithm that can traverse all results to assist the wiring process of the A-star algorithm. However, if the Q-Learning algorithm is simply introduced directly, it will lead to poor convergence speed and stability. This improvement effect is not ideal. Therefore, we have proposed a double decay strategy on this basis. By dynamically adjusting the exploration factor and learning rate values of the Q-Learning algorithm during operation, it is possible to prevent the algorithm from falling into the local optimal solution during the path-finding process and improve the efficiency of the algorithm. The present invention proposes to use the iterative training process of the Q-Learning algorithm in the reinforcement learning algorithm to traverse all results, and introduce a double decay strategy at the same time, so that the performance of the Q-Learning algorithm when wiring in the PCB environment is optimized.
[0067] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0068] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A method for routing in a PCB environment using a reinforcement learning algorithm to assist the A-star algorithm, characterized in that: include: The A-star algorithm is optimized by using the Q-Learning algorithm and the double decay strategy; the A-star algorithm includes a total cost function; In a PCB simulation wiring environment, the Q-Learning algorithm and the double attenuation strategy are used to assist the A-star algorithm to solve the wiring path and obtain the shortest path; the PCB simulation wiring environment is constructed according to the set starting node, target node and obstacle; the PCB wiring environment is a grid environment.
2. The method for routing in a PCB environment using a reinforcement learning algorithm assisted by an A-star algorithm according to claim 1, characterized in that: The total cost function is the sum of the actual cost function and the heuristic function.
3. The method for routing in a PCB environment using a reinforcement learning algorithm to assist an A-star algorithm according to claim 1, characterized in that: The use of the Q-Learning algorithm and the double attenuation strategy to assist the A-star algorithm in solving the wiring path specifically includes: In the PCB simulation wiring environment, the Q-Learning algorithm is first used to assist the A-star algorithm for wiring, and then the double attenuation strategy is used to optimize the wiring path to obtain the shortest path.
4. The method for routing in a PCB environment using a reinforcement learning algorithm to assist an A-star algorithm according to claim 1, characterized in that: The process of solving the wiring path is: using the total cost function to solve the wiring path nodes in the PCB simulation environment to obtain the shortest path.
5. The method for routing in a PCB environment using a reinforcement learning algorithm to assist an A-star algorithm according to claim 1, characterized in that: During the training process of the Q-Learning algorithm, the calculation formula for the action reward value is: Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s′,a′)-Q(s,a)] Among them, α is the learning rate, which is used to determine the extent to which newly acquired information will overwrite old information. If the value is 0, it means that the agent stops learning, and if the value is 1, it means that the agent only considers the latest information; γ is the exploration factor, which is used to indicate the impact of further rewards on the current action; s is the state, a is the action, r is the reward, and max a′ Q(s′, a′) means selecting the combination of all states and actions that can be selected in the next step with the highest reward value as the next exploration position, and then obtaining the updated reward value.
6. The method for routing in a PCB environment using a reinforcement learning algorithm to assist an A-star algorithm according to claim 3, characterized in that: The use of the Q-Learning algorithm to assist the A-star algorithm in routing specifically includes: Initialize Q-table: First, create a Q-table to store the Q-value of each state and action; the size of the Q-table depends on the number of states and actions in the environment; initially, initialize all Q-values to 0 or random values; Select action: At each time step, an action is selected based on the current state. The specific process is: using the epsilon-greedy strategy, randomly select an action with a probability of ε, and select the action with the highest Q-value with a probability of 1-ε; Execute actions: Apply the selected action to the environment, and get rewards and the next state based on the feedback from the environment; Update Q-value: Use the Q-learning update formula to update Q-value based on the reward and next state. Iteration: Repeat the steps of "select action", "execute action" and "update Q-value" until the stopping condition is reached; the stopping condition is reaching the maximum number of iterations or converging to a stable value; Path selection: Select the optimal path based on the trained Q-table. The specific process is: starting from the starting state, select the action with the highest Q-value, transfer to the next state, and then repeat the process until the target state is reached to obtain the output path.
7. The method for routing in a PCB environment using a reinforcement learning algorithm to assist an A-star algorithm according to claim 1, characterized in that: The double decay strategy specifically includes: during the operation of the Q-learning algorithm, as the number of training times increases, the values of the exploration factor and the learning rate are dynamically reduced.
8. A system for routing in a PCB environment using a reinforcement learning algorithm to assist the A-star algorithm, characterized in that: include: An algorithm optimization unit, used for optimizing the A-star algorithm using the Q-Learning algorithm and the double decay strategy; the A-star algorithm includes a total cost function; The wiring path determination unit is used to solve the wiring path in a PCB simulation wiring environment by using the Q-Learning algorithm and the double attenuation strategy to assist the A-star algorithm to obtain the shortest path; the PCB simulation wiring environment is constructed according to the set starting node, target node and obstacle; the PCB wiring environment is a grid environment.