Logic optimization command sequence combination method based on game reinforcement learning

Abstract

The invention discloses a logic optimization command sequence combination method based on game reinforcement learning, and belongs to the technical field of integrated circuit design automation. According to the method, two agents are introduced, an A2C algorithm is adopted between the two agents to construct a game reinforcement learning architecture, and the first agent takes delay optimization target award and global optimization target to train the Shapley value allocation thereof as an award function; the second agent takes the area optimization target award and the global optimization target to train the Shapley value allocation thereof as an award function; the distributed action network allows the intelligent agents to have different optimization tendencies, and the central evaluation network enables the intelligent agents to pay attention to the global condition and coordinate and optimize towards the overall target; in addition, the Shapley value is introduced to achieve fair distribution of the overall reward among the agents, and the problem that in the prior art, the optimal tradeoff solution of the two optimization targets of the area and the time delay cannot be found is solved.

Description

Technical Field

[0001] The invention relates to a logic optimization command sequence combination method based on game reinforcement learning, belonging to the technical field of integrated circuit design automation. Background Art

[0002] Logic optimization is a key link in digital integrated circuit design. By optimizing the number of logic gates and circuit structure of the circuit, it can not only reduce the circuit delay and optimize the timing constraints of the circuit, but also reduce the circuit area, internal resource occupation and circuit power consumption. In the process of integrated circuit logic optimization, the AIG structure plays a key role. Compared with other logical expressions, such as Boolean functions or Karnaugh maps, the compact structure of AIG occupies less memory space and can effectively represent complex logic circuits. In addition, the structure can also reduce the area and delay of logic circuits by identifying and eliminating repeated subgraphs and optimizing logic gate layout.

[0003] Generally speaking, methods for optimizing AIG structures include Resub, Rewrite, Balance, and Refactor. Resub can extract subcircuits and reuse the structure at the desired location; Rewrite can minimize the number of nodes by repeatedly updating and optimizing the AIG subgraph on the nodes; Balance can balance the load and delay of each path to meet the required constraints; Refactor can adjust and optimize the hierarchical structure and logic path of AIG. In logic optimization, the above methods are called repeatedly to form an optimization command sequence, and then the results are verified and evaluated.

[0004] With the development of Moore's Law, the scale of digital integrated circuits is getting larger and larger. The solution space required for optimizing command sequences is huge, and each combination of optimized command sequences results in different circuit structures and performances. An effective solution is to replace manual work with reinforcement learning based on dynamic programming (DRiLLS: Deep Reinforcement Learning for Logic Synthesis [C]. 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), 2020: 581-586.) to improve the efficiency of logic circuit design and optimization. Reinforcement learning is a machine learning method that aims to enable agents to learn how to make decisions to achieve certain goals through interaction with the environment. This dynamic programming idea can be well applied to the design optimization of digital integrated circuits. The circuit structure is taken as the state, the new optimization command is taken as the action of the agent, and the reward function is designed to guide the agent according to the area and delay requirements. In logic optimization, new commands are selected according to the current circuit structure, and the agent is trained by observing the results of the environment's feedback on its commands, so that the commands selected later continuously optimize the circuit logic to obtain the optimal command sequence combination.

[0005] However, there are some urgent problems to be solved when using reinforcement learning for integrated circuit logic optimization. One of the main problems is the handling of multiple objectives in the logic optimization process. In the process of logic optimization, it is hoped that the area and delay of the circuit can be reduced to their respective minimum values ​​at the same time. However, in fact, the area and delay of the circuit are both related and contradictory, which belongs to the general game problem. During optimization, it is not necessarily possible to minimize the circuit area and delay at the same time. Instead, the optimal trade-off solution between area and delay is found according to project requirements.

[0006] To this end, based on the pain points of the logic optimization process described in the prior art, the technical field urgently needs a method for exploring the best combination of logic optimization commands that can simultaneously achieve goals such as area and latency. Summary of the invention

[0007] In order to find the optimal trade-off solution between area and delay according to project requirements, the present invention provides a logic optimization command sequence combination method based on game reinforcement learning. The technical solution is as follows:

[0008] The logic optimization command sequence combination method of the present invention introduces a first agent with the goal of optimizing circuit delay and a second agent with the goal of optimizing circuit area. The A2C algorithm is used to construct a game reinforcement learning architecture between the first agent and the second agent to form an action network for delay optimization, an action network for area optimization, and an evaluation network responsible for global coordination.

[0009] The first agent is trained using the Shapley value distribution of the delay optimization target reward and the global optimization target as a reward function, and the second agent is trained using the Shapley value distribution of the area optimization target reward and the global optimization target as a reward function;

[0010] The logic optimization command sequence combination method comprises:

[0011] Step 1: Obtain the AIG state vector of the circuit netlist to be optimized from the logic synthesis environment through the first intelligent agent, and input the action network for the delay optimization target to obtain the probability value vector of each action being selected in the optimization command action space; select the optimization command action according to the probability value vector, and send it to the logic synthesis environment for execution, optimize the circuit netlist to be optimized, and further obtain the AIG state vector, delay value and area value of the optimized circuit netlist;

[0012] Step 2: Obtain the AIG state vector of the optimized circuit netlist from the logic synthesis environment through the second agent, and input the action network for the area optimization goal to obtain the probability value vector of each action being selected in the optimization command action space, select the optimization command action according to the obtained probability value vector, and send it to the logic synthesis environment for execution, optimize the circuit netlist obtained in the step 1 again, and further obtain the AIG state vector, delay value and area value of the optimized circuit netlist;

[0013] Step 3: Repeat steps 1 and 2 until twice the preset number of iterations N is met;

[0014] Step 4: Record all delay values ​​and area values ​​in the sequential game process, and sort them into non-dominated solutions to obtain the Pareto frontier of the logical optimization command sequence solution under the area and delay optimization objectives.

[0015] Optionally, during the training process, the loss value of the evaluation network responsible for global coordination is:

[0016]

[0017] Among them, V π (s i ) is the actual output of the evaluation network, discounted_episode_rewards i represents the discounted reward for the i-th iteration, which is calculated as:

[0018] discounted_episode_rewards0=episode_rewards0

[0019] discounted_episode_rewards i =discounted_episode_rewards i-1 *γ+episode_rewards i

[0020] Among them, episode_rewards0 represents the initial value of the global optimization target reward, γ is the discount factor, and episode_rewards i represents the global optimization target reward of the i-th iteration.

[0021] Optionally, during the training process, the loss value of the action network for the delay optimization objective is:

[0022] actor_loss=neg_log_prob*advantage

[0023]

[0024] advantage = Q π (s i ,a i ) 1 -V π (s i )

[0025] Where neg_log_prob represents the difference between the action distribution output by the action network for the delay optimization goal and the real label y after softmax normalization. i The cross entropy loss between ε and V is a small perturbation, advantage represents the advantage function, and V π (s i ) is the actual output of the evaluation network, Q π (s i ,a i ) 1 represents the state-action function, expressed as:

[0026] Q π (s i ,a i ) 1 =Shapley 1

[0027] Shapley 1 =(discounted_episode_rewards i +discounted_episode_rewards 1i -discounted_episode_rewards 2 i ) / 2

[0028] Among them, Shapley 1 represents the Shapley value distribution of the global optimization objective to the first agent, discounted_episode_rewards 1 i and discounted_episode_rewards 2 i ) represent the rewards obtained by the first agent and the first agent interacting with the environment in the i-th iteration respectively.

[0029] Optionally, during training, the loss value of the action network for the area optimization objective is:

[0030] actor_loss=neg_log_prob*advantage

[0031]

[0032] advantage = Q π (s i ,a i ) 2 -V π (s i )

[0033] Where neg_log_prob represents the softmax normalization of the action distribution output by the action network for the area optimization goal and the true label y i The cross entropy loss between ε and V is a small perturbation, advantage represents the advantage function, and V π (s i ) is the actual output of the evaluation network, Q π (s i ,a i ) 2 represents the state-action function, expressed as:

[0034] Q π (s i ,a i ) 2 =Shapley 2

[0035] Shapley 2 =(discounted_episode_rewards i+discounted_episode_rewards 2 i -discounted_episode_rewards 1 i ) / 2

[0036] Among them, Shapley 2 represents the Shapley value distribution of the global optimization objective to the second agent, discounted_episode_rewards 1 i and discounted_episode_rewards 2 i ) represent the rewards obtained by the first agent and the first agent interacting with the environment in the i-th iteration respectively.

[0037] Optionally, the AIG state space includes: the circuit's input, output, number of nodes, number of edge connections, depth, number of latches, number of AND gates, and number of NOT gates. Regularization is required for each input state.

[0038] Optionally, the actions in the optimization command action space include:

[0039] Rewrite, quick rewrite, replace, quick replace, refactor, quick refactor, balance.

[0040] Optionally, the logic synthesis environment is composed of Yosys and ABC open source tools.

[0041] The present invention provides a logic optimization command sequence combination system, which is used to implement the logic optimization command sequence combination method as described in any one of the above items, and the system includes: a logic synthesis environment, a first intelligent agent with the goal of optimizing circuit delay, and a second intelligent agent with the goal of optimizing circuit area, wherein the first intelligent agent and the second intelligent agent use an A2C algorithm to construct a game reinforcement learning architecture to form an action network for delay optimization, an action network for area optimization, and an evaluation network responsible for global coordination;

[0042] The first agent is trained using the Shapley value allocation of the delay optimization target reward and the global optimization target as a reward function, and the second agent is trained using the Shapley value allocation of the area optimization target reward and the global optimization target as a reward function.

[0043] The present invention provides a logic optimization command sequence combination device, comprising a memory and a processor;

[0044] The memory is used to store computer programs;

[0045] The processor is used to implement the logic optimization command sequence combination method as described in any one of the above items when executing the computer program.

[0046] The present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the logic optimization command sequence combination method as described in any one of the above items is implemented.

[0047] The beneficial effects of the present invention are:

[0048] The present invention uses the netlist to be optimized as input, utilizes the reinforcement learning algorithm to interact with the environment, selects actions in the action space composed of optimization commands such as reconstruction, rewriting, and balancing, and can realize the search for the optimal command sequence in the complex solution space. Two intelligent agents are introduced, and the A2C algorithm is used between each intelligent agent to construct a distributed action network and a centralized evaluation network architecture. The distributed action network can make selections using their own action networks to realize sequential games in sequence; while the centralized evaluation network can measure the value of each discrete action to the whole. In addition, the present invention introduces the Shapley value to realize the fair distribution of the overall reward among the intelligent agents, which solves the problem that the optimal trade-off solution for the two optimization objectives of area and delay cannot be found in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is the architecture diagram of the logic optimization command sequence combination method based on game reinforcement learning.

[0051] Figure 2 This is the architecture diagram of the distributed action network and the centralized evaluation network.

[0052] Figure 3 To evaluate the splicing of the network. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0054] Embodiment 1:

[0055] This embodiment provides a logic optimization command sequence combination method, which introduces a first agent with the goal of optimizing circuit delay and a second agent with the goal of optimizing circuit area. The A2C algorithm is used to construct a game reinforcement learning architecture between the first agent and the second agent to form an action network for delay optimization, an action network for area optimization, and an evaluation network responsible for global coordination.

[0056] The first agent is trained with the Shapley value distribution of the delay optimization target reward and the global optimization target as the reward function, and the second agent is trained with the Shapley value distribution of the area optimization target reward and the global optimization target as the reward function;

[0057] The logic optimization command sequence combination method of this embodiment includes:

[0058] Step 1: Obtain the AIG state vector of the circuit netlist to be optimized from the logic synthesis environment through the first intelligent agent, and input the action network for the delay optimization target to obtain the probability value vector of each action being selected in the optimization command action space; select the optimization command action according to the probability value vector, and send it to the logic synthesis environment for execution, optimize the circuit netlist to be optimized, and further obtain the AIG state vector, delay value and area value of the optimized circuit netlist;

[0059] Step 2: Obtain the AIG state vector of the optimized circuit netlist from the logic synthesis environment through the second intelligent agent, and input the action network for the area optimization goal to obtain the probability value vector of each action being selected in the optimization command action space, select the optimization command action according to the obtained probability value vector, and send it to the logic synthesis environment for execution, optimize the circuit netlist obtained in step 1 again, and further obtain the AIG state vector, delay value and area value of the optimized circuit netlist;

[0060] Step 3: Repeat steps 1 and 2 until twice the preset number of iterations N is met;

[0061] Step 4: Record all delay values ​​and area values ​​in the sequential game process, and sort them into non-dominated solutions to obtain the Pareto frontier of the logical optimization command sequence solution under the area and delay optimization objectives.

[0062] Furthermore, the solutions in the Pareto front can be selected according to the user's needs and according to the delay or area target tendency.

[0063] The logic synthesis environment in this embodiment can receive netlist input, execute an optimization command, and return the AIG state, area, and delay status after the optimization command is executed. It can be composed of Yosys and ABC open source tools.

[0064] The AIG state space consists of primary I / O, number of nodes, number of edges, depth, number of latches, number of ANDs, and number of NOTs. Regularization is required for each input state.

[0065] Optimize the command action space, which consists of rewrite, fast rewrite (rewrite-z), resub, fast resub-z, refactor, fast refactor (refactor-z), and balance.

[0066] The Shapley value allocation rule is a distribution method that achieves fair distribution based on the agent's own contribution to the global optimization goal.

[0067] Embodiment 2:

[0068] This embodiment provides a training process of the game reinforcement learning architecture constructed in the first embodiment. The game reinforcement learning architecture includes a decentralized action network and a centralized evaluation network, specifically including: an action network for delay optimization, an action network for area optimization, and an evaluation network responsible for global coordination, see Figure 2 .

[0069] The training steps of the open game reinforcement learning framework in this embodiment are as follows:

[0070] It includes two parts: independent training and splicing training.

[0071] First, two agents are designed, each with a set of action networks and evaluation networks.

[0072] The structure of the action network designed in this embodiment is: two fully connected layers, the number of neurons in each layer is 20; the structure of the evaluation network is: 1 hidden layer, the number of neurons is 15.

[0073] Independent training is as follows:

[0074] Step 1: Initialize the logic synthesis environment.

[0075] Step 2: The first agent Agent1 observes the current logic synthesis environment and obtains the AIG state vector at this time. Given the action network for delay optimization, the input is the AIG state vector, and the output is the probability value vector of each action selected in the action space. Agent1 based on the A2C algorithm will select actions according to the output probability distribution and send them to the logic synthesis environment for execution, optimize the netlist, and obtain the new AIG state vector, delay value and area value.

[0076] Step 3: Delay according to the delay value i and delay constraint delay constraint , calculate the delay optimization target reward, the calculation method is shown in formula (1):

[0077]

[0078] Among them, rewards_delay i represents the delay optimization target reward in the i-th iteration.

[0079] Step 4: Agent 1 repeats steps 2 to 3, setting the total number of iterations to N. At this time, step 3 feeds back a set of immediate reward sequences episode_rewards_delay N Using formula (2) and formula (3) to calculate the discount reward, we can get discounted_episode_rewards_delay i , where γ is the discount factor.

[0080] discounted_episode_rewards_delay0=episode_rewards_delay0 (2)

[0081]

[0082] Among them, episode_rewards_delay0 and discounted_episode_rewards_delay i They represent the initial value of the delay optimization target reward and the discounted value of the delay optimization target reward in the i-th iteration respectively.

[0083] Step 5: Initialize the logic synthesis environment.

[0084] Step 6: The second agent Agent2 observes the current logic synthesis environment and obtains the AIG state vector at this time. Given the action network for delay optimization, the input is the AIG state vector, and the output is the probability value vector of each action selected in the action space. Agent2 based on the A2C algorithm will select actions according to the output probability distribution and send them to the logic synthesis environment for execution, optimize the netlist, and obtain the new AIG state vector, delay value and area value.

[0085] Step 7: According to the area value area i and the initial value of the net surface area initial , calculate the area optimization target reward, the calculation method is shown in formula (4):

[0086]

[0087] rewards_area i represents the area optimization objective reward in the i-th iteration.

[0088] Step 8: Agent 2 repeats steps 6 to 7, setting the total number of iterations to N. At this time, step 7 feeds back a set of instant reward sequences episode_rewards_area N Using formula (5) and formula (6) to calculate the discount reward, we can get discounted_episode_rewards_area i Where γ is the discount factor.

[0089] discounted_episode_rewards_area0=episode_rewards_area0 (5)

[0090]

[0091] Among them, episode_rewards_area0 and discounted_episode_rewards_area i They represent the initial value of the area optimization objective reward and the discounted value of the area optimization objective reward in the i-th iteration respectively.

[0092] Step 9: Use formula (7) and formula (8) to calculate the loss function of Agent1 and Agent2 evaluation network. π (s i ) are the actual outputs of the evaluation network of Agent1 and Agent2 respectively. The Adam algorithm is used to optimize the evaluation network parameters with the goal of minimizing critic_loss.

[0093]

[0094] Step 10: Use formulas (9), (10) and (11) to calculate the loss function of the action network of Agent 1 and Agent 2. Formula (9) represents the difference between the action distribution output by the action network and the actual label y after softmax normalization. i The cross entropy loss between agents is ε, which is a small perturbation to prevent infinite values. Formula (10) uses the Q value of the state-action pair (the Q values ​​of Agent 1 and Agent 2 are discounted_episode_rewards_delay i and discounted_episode_rewards_area i) and the difference between the state values ​​of the respective evaluation network outputs as the advantage function; Formula (11) multiplies Formula (9) and Formula (10) to calculate the loss function of the action network. Since the total number of iterations is set to N, it is necessary to repeatedly execute Formula (9) to Formula (11) when calculating the loss function, and take the average actor_loss.

[0095]

[0096] advantage = Q π (s i ,a i )-V π (s i ) (10)

[0097] actor_loss=neg_log_prob*advantage (11)

[0098] Step 11: Repeat steps 1 to 10 M times to complete the independent training.

[0099] Next, it is splicing training.

[0100] Step 12: Figure 3 As shown, the action network of the first agent Agent1 and the second agent Agent2 remains unchanged, the evaluation network is spliced ​​to obtain a new shared evaluation network, and a fully connected layer is added to map the two inputs into a single input. Thus, the decentralized action network and the centralized evaluation network are obtained.

[0101] Step 13: Initialize the logic synthesis environment.

[0102] Step 14: The first agent Agent1 observes the current logic synthesis environment and obtains the current AIG state vector. Given the action network for delay optimization, the input is the AIG state vector, and the output is the probability value vector of each action selected in the action space. Agent1 based on the A2C algorithm will select actions according to the output probability distribution and send them to the logic synthesis environment for execution, optimize the netlist, change the logic synthesis environment, and obtain new AIG state vectors, delay values, and area values.

[0103] Step 15: The second agent Agent2 observes the logic synthesis environment at this time and obtains the AIG state vector at this time. Given the action network for area optimization, the input is the AIG state vector, and the output is the probability value vector of each action selected in the action space. Agent2 based on the A2C algorithm will select the action according to the output probability distribution. It is sent to the logic synthesis environment for execution, the netlist is optimized, the logic synthesis environment changes, and the new AIG state vector, delay value and area value are obtained.

[0104] Step 16: Calculate the global optimization target reward based on the delay value and area value, the global optimization target reward value episode_rewards i See Table 1.

[0105] Table 1 Global optimization target reward value

[0106]

[0107] Step 17: Agent 1 and Agent 2 repeat steps 14 to 16. When both Agent 1 and Agent 2 have completed their actions, it is recorded as two iterations. The total number of iterations is set to 2N. At this time, step 16 will feedback a set of immediate reward sequences respectively, and only the latter will be taken to form episode_rewards N Using formula (12) and formula (13) to calculate the discount reward, we can get discounted_episode_rewards i , where γ is the discount factor.

[0108] discounted_episode_rewards0=episode_rewards0 (12)

[0109]

[0110] Among them, episode_rewards0 and discounted_episode_rewards i They represent the initial value of the global optimization objective reward and the discounted value of the global optimization objective reward in the i-th iteration respectively.

[0111] Step 18: For the action network with delay optimization goal, it is necessary to consider the contribution to the global optimization goal. The contribution to the global optimization goal is calculated using the Shapley value allocation rule, as shown in formula (14). 1 i with discounted_episode_rewards 2i Separate the optimization commands executed by Agent 1 and Agent 2 in the i-th iteration and interact with the environment again, and obtain the rewards according to Table 1; then, calculate the Q value using formula (15) and substitute it into step 21 to obtain the actor_loss mean of the action network for the delay optimization goal. Finally, the Adam algorithm is used to optimize the action network parameters for the delay optimization goal with the goal of minimizing the actor_loss mean.

[0112] Shapley 1 =

[0113] (discounted_episode_rewards i +discounted_episode_rewards 1 i (14)

[0114] -discounted_episode_rewards 2 i ) / 2

[0115] Q π (s i ,a i ) 1 =Shapley 1 (15)

[0116] Among them, s i represents the state in the i-th iteration, a i Represents the action in the i-th iteration.

[0117] Step 19: For the action network for the area optimization goal, it is necessary to consider the contribution to the global optimization goal. The contribution to the global optimization goal is calculated using the Shapley value allocation rule, as shown in formula (16). Subsequently, the Q value is calculated using formula (17) and substituted into step 21 to obtain the actor_loss mean of the action network for the area optimization goal. Finally, the Adam algorithm is used to optimize the parameters of the action network for the area optimization goal with the goal of minimizing the actor_loss mean.

[0118] Shapley 2 =

[0119] (discounted_episode_rewards i +discounted_episode_rewards 2 i (16)

[0120] -discounted_episode_rewards 1 i ) / 2

[0121] Q π (s i ,a i ) 2 =Shapley 2 (17)

[0122] Step 20: Use formula (18) to calculate the loss function of the evaluation network. π (s_fusion i ) is the actual output obtained by concatenating the states of Agent1 and Agent2 and inputting them into the evaluation network. The Adam algorithm is used to optimize the evaluation network parameters with the goal of minimizing critic_loss.

[0123]

[0124] Step 21: Calculate the action network loss value. Formula (19) represents the difference between the action network output action distribution and the real label y after softmax normalization. i The cross entropy loss between the two, ε is a small perturbation to prevent infinite values; Formula (20) uses the difference between the Q value under the state-action pair and the state value (that is, the output value of the evaluation network under the current state combination) as the advantage function; Formula (21) multiplies Formula (19) and Formula (20) to calculate the loss function of the action network. Since the total number of iterations is set to N, it is necessary to repeatedly execute Formula (19) to Formula (21) when calculating the loss function, and take the average of actor_loss.

[0125]

[0126] advantage = Q π (s i ,a i )-V π (s i ) (20)

[0127] actor_loss=neg_log_prob*advantage (21)

[0128] Step 22: Repeat steps 13 to 21 M times to complete the splicing training.

[0129] Embodiment three:

[0130] This embodiment uses the logic optimization command sequence combination method based on game reinforcement learning described in the present invention to conduct experiments.

[0131] In this embodiment, adder.v, log2.v and multiplier.v provided by EPFL are selected as the source files of high-level description, and open source logic synthesis platforms such as abc and yosys are connected. Asap7.lib is selected as the process library file to convert, optimize and map the source files.

[0132] The implementation process of the logic optimization command sequence combination method based on game reinforcement learning in this embodiment is suitable for ASIC or FPGA synthesis environment.

[0133] The main body of the algorithm in this embodiment is written in Python, and the corresponding logic synthesis commands are completed based on the TCL script language. The optimization algorithm selects the optimization command according to the actual situation and writes it into the corresponding TCL script, which is sent to the logic synthesis platform for execution, and uses regular matching to read the delay and area data fed back by the platform to guide the next algorithm optimization direction.

[0134] In order to compare the performance of the method of the present invention, an existing method was selected as a control (the control method refers to "DRiLLS: Deep Reinforcement Learning for Logic Synthesis [C]. 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), 2020: 581-586."), and experiments were carried out on the method of the present invention and the control method respectively.

[0135] In this example, the discount factor γ is set to 0.99, the total number of training rounds M is 50, the number of iterations per training round N is 30, and the small perturbation ε is 1×10 -6 .

[0136] Table 2 Comparison of experimental results

[0137]

[0138] The data comparison in Table 2 clearly shows the performance difference between the logic optimization command sequence combination method based on game reinforcement learning of the present invention and the existing method. As can be seen from Table 2, the existing method obtains coordinated optimization of area and delay on the three inputs measured compared with the DRiLLS method. This result shows that the method of the present invention can effectively solve the contradiction between area and delay, and provides a more comprehensive and balanced solution for logic optimization.

[0139] In summary, the logic optimization command sequence combination method based on game reinforcement learning of the present invention has significant advantages and practical value in the field of logic optimization. By effectively balancing the two key optimization goals of area and delay, the method of the present invention provides a more efficient and accurate logic optimization solution for practical applications. At the same time, its excellent adaptability and robustness also make the method of the present invention an important reference and choice for future logic optimization work.

[0140] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A logic optimization command sequence combination method, characterized in that: The method introduces a first agent with the goal of optimizing circuit delay and a second agent with the goal of optimizing circuit area. The first agent and the second agent use the A2C algorithm to build a game reinforcement learning architecture to form an action network for delay optimization, an action network for area optimization, and an evaluation network responsible for global coordination. The first agent is trained using the Shapley value distribution of the delay optimization target reward and the global optimization target as a reward function, and the second agent is trained using the Shapley value distribution of the area optimization target reward and the global optimization target as a reward function; The logic optimization command sequence combination method comprises: Step 1: Obtain the AIG state vector of the circuit netlist to be optimized from the logic synthesis environment through the first intelligent agent, and input the action network for the delay optimization target to obtain the probability value vector of each action being selected in the optimization command action space; select the optimization command action according to the probability value vector, and send it to the logic synthesis environment for execution, optimize the circuit netlist to be optimized, and further obtain the AIG state vector, delay value and area value of the optimized circuit netlist; Step 2: Obtain the AIG state vector of the optimized circuit netlist from the logic synthesis environment through the second agent, and input the action network for the area optimization goal to obtain the probability value vector of each action being selected in the optimization command action space, select the optimization command action according to the obtained probability value vector, and send it to the logic synthesis environment for execution, optimize the circuit netlist obtained in the step 1 again, and further obtain the AIG state vector, delay value and area value of the optimized circuit netlist; Step 3: Repeat steps 1 and 2 until twice the preset number of iterations N is met; Step 4: Record all delay values ​​and area values ​​in the sequential game process, and sort them into non-dominated solutions to obtain the Pareto frontier of the logical optimization command sequence solution under the area and delay optimization objectives.

2. The logic optimization command sequence combination method according to claim 1, characterized in that: During the training process, the loss value of the evaluation network responsible for global coordination is: Among them, V π (s i ) is the actual output of the evaluation network, discounted_episode_rewards i represents the discounted reward of the global optimization target for the i-th iteration, which is calculated as: discounted_episode_rewards0=episode_rewards0 discounted_episode_rewards i =discounted_episode_rewards i-1 *γ+episode_rewards i Among them, episode_rewards0 represents the initial value of the global optimization target reward, γ is the discount factor, and episode_rewards i represents the global optimization target reward of the i-th iteration.

3. The logic optimization command sequence combination method according to claim 1, characterized in that: During the training process, the loss value of the action network for the delay optimization goal is: actor_loss=neg_log_prob*advantage advantage=Q π (s i ,a i ) 1 -V π (s i ) Where neg_log_prob represents the difference between the action distribution output by the action network for the delay optimization goal and the real label y after softmax normalization. i The cross entropy loss between ε and V is a small perturbation, advantage represents the advantage function, and V π (s i ) is the actual output of the evaluation network, Q π (s i ,a i ) 1 represents the state-action function, expressed as: Q π (s i ,a i ) 1 =shapley 1 shapley 1 =(discounted_episode_rewards i +discounted_episode_rewards 1 i -discounted_episode_rewards 2 i ) / 2 Among them, s i represents the state in the i-th iteration, a i represents the action in the i-th iteration, Shapley 1 represents the Shapley value distribution of the global optimization objective to the first agent, discounted_episode_rewards 1 i and discounted_episode_rewards 2 i Respectively represent separating the optimization command executed in the i-th iteration, and the first agent and the second agent respectively re-execute the optimization command and interact with the environment to obtain rewards.

4. The logic optimization command sequence combination method according to claim 1, characterized in that: During the training process, the loss value of the action network for the area optimization objective is: actor_loss=neg_log_prob*advantage advantage=Q π (s i ,a i ) 2 -V π (s i ) Where neg_log_prob represents the softmax normalization of the action distribution output by the action network for the area optimization goal and the true label y i The cross entropy loss between ε and V is a small perturbation, advantage represents the advantage function, and V π (s i ) is the actual output of the evaluation network, Q π (s i ,a i ) 2 represents the state-action function, expressed as: Q π (s i ,a i ) 2 =shapley 2 shapley 2 = (discounted_episode_rewards i +discounted_episode_rewards 2 i -discounted_episode_rewards 1 i ) / 2 Among them, s i represents the state in the i-th iteration, a i represents the action in the i-th iteration, Shapley 2 represents the Shapley value distribution of the global optimization objective to the second agent, discounted_episode_rewards 1 i and discounted_episode_rewards 2 i Respectively represent separating the optimization command executed in the i-th iteration, and the first agent and the second agent respectively re-execute the optimization command and interact with the environment to obtain rewards.

5. The logic optimization command sequence combination method according to claim 1, characterized in that: The AIG state space includes: the circuit's input, output, number of nodes, number of edge connections, depth, number of latches, number of AND gates, and number of NOT gates.

6. The logic optimization command sequence combination method according to claim 1, characterized in that: The actions in the optimization command action space include: rewriting, replacing, reconstructing, and balancing.

7. The logic optimization command sequence combination method according to claim 1, characterized in that: The logic synthesis environment is composed of Yosys and ABC open source tools.

8. A logic optimization command sequence combination system, characterized in that: The system is used to implement the logic optimization command sequence combination method according to any one of claims 1 to 7, wherein the system comprises: a logic synthesis environment, a first agent with the goal of optimizing circuit delay, and a second agent with the goal of optimizing circuit area, wherein the first agent and the second agent use an A2C algorithm to construct a game reinforcement learning architecture to form an action network for delay optimization, an action network for area optimization, and an evaluation network responsible for global coordination; The first agent is trained using the Shapley value allocation of the delay optimization target reward and the global optimization target as a reward function, and the second agent is trained using the Shapley value allocation of the area optimization target reward and the global optimization target as a reward function.

9. A logic optimization command sequence combination device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the logic optimization command sequence combination method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the logic optimization command sequence combination method according to any one of claims 1 to 7 is implemented.

Images