A logic optimization command sequence combination method based on game reinforcement learning

Abstract

The application discloses a logic optimization command sequence combination method based on game reinforcement learning and belongs to the technical field of integrated circuit design automation. Two agents are introduced, and a game reinforcement learning framework is constructed by adopting an A2C algorithm between the two agents. A first agent is trained by taking a delay optimization target reward and a global optimization target sharpe value distribution as a reward function, and a second agent is trained by taking an area optimization target reward and a global optimization target sharpe value distribution as a reward function. A decentralized action network allows the agents to have different optimization tendencies, and a centralized evaluation network allows the agents to focus on the global situation and coordinate optimization towards the overall target. In addition, the application introduces a sharpe value to realize fair distribution of the overall reward among the agents, and solves the problem that an optimal trade-off solution of the area and delay optimization targets cannot be found in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to a game-based reinforcement learning logic optimization command sequence combination method, belonging to the technical field of integrated circuit design automation. BACKGROUND

[0002] Logic optimization is a key link in the design of digital integrated circuits. By optimizing the number of logic gates and the structure of the circuit, not only can the delay of the circuit be reduced, the timing constraints of the circuit be optimized, but also the area of the circuit, the occupation of internal resources and the power consumption of the circuit can be reduced. In the process of integrated circuit logic optimization, AIG structure plays a key role. Compared with other logic expression methods, such as Boolean functions or Karnaugh maps, AIG has a more compact structure and occupies less memory space, and can effectively represent complex logic circuits. Moreover, this structure can also reduce the area and delay of the logic circuit by identifying and eliminating duplicate subgraphs, optimizing the layout of logic gates, etc.

[0003] Generally speaking, the methods for optimizing AIG structure include Resub, Rewrite, Balance, Refactor, etc. Resub can extract sub-circuits and repeatedly use the structure at the required location; Rewrite can minimize the number of nodes by repeatedly updating and optimizing the AIG subgraph on the node; Balance can balance the load and delay of each path to meet the required constraint conditions; Refactor can adjust and optimize the hierarchical structure and logic path of AIG. In logic optimization, the above methods are repeatedly called to form an optimization command sequence, and then the result is verified and evaluated.

[0004] With the development of Moore's law, the scale of digital integrated circuit is getting larger and larger, and the solution space required for optimizing the command sequence is huge, and each combination of optimization command sequence causes different structure and performance of the circuit. An effective solution is to replace artificial with reinforcement learning based on dynamic programming idea (DRiLLS: Deep Reinforcement Learning for Logic Synthesis[C]. 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), 2020: 581-586.) to improve the efficiency of logic circuit design and optimization. Reinforcement learning is a machine learning method that aims to let the agent learn how to make decisions to achieve a certain goal through interaction with the environment, and this dynamic programming idea can be well applied to the design and optimization of digital integrated circuits. The circuit structure is regarded as the state, and the new optimization command is regarded as the action of the agent, and the reward function is designed according to the area and delay requirements to guide the agent. In logic optimization, the current circuit structure is used to select new commands, and the agent is trained by observing the feedback results of the environment to the commands, so that the subsequent selected commands can continuously optimize the circuit logic to obtain the optimal command sequence combination mode.

[0005] However, there are some problems to be solved in the use of reinforcement learning for integrated circuit logic optimization at present, one of which is the problem of multi-objective processing in logic optimization. In the process of logic optimization, it is hoped that the area and delay of the circuit can be reduced to the minimum value at the same time, but in fact the area and delay of the circuit are related and contradictory, which belongs to general and game problem. In optimization, the circuit area and delay cannot be minimized at the same time, instead, the optimal trade-off solution of area and delay is found according to the project requirements.

[0006] Therefore, based on the pain points of the existing technology described in the logic optimization process, there is an urgent need in the technical field for a logic optimization command optimal combination exploration method that can realize the area and delay targets at the same time. SUMMARY

[0007] In order to realize the optimal trade-off solution of area and delay according to the project requirements, the present application provides a logic optimization command sequence combination method based on game reinforcement learning, and the technical solution is as follows:

[0008] The logic optimization command sequence combination method of the present application introduces a first agent for optimizing circuit delay and a second agent for optimizing circuit area, and the A2C algorithm is used to build a game reinforcement learning architecture between the first agent and the second agent, forming an action network for delay optimization target, an action network for area optimization target, and an evaluation network responsible for global planning.

[0009] The first agent is trained with a reward function of a delay optimization target reward and a Sharpe value distribution of the global optimization target to it, and the second agent is trained with a reward function of an area optimization target reward and a Sharpe value distribution of the global optimization target to it;

[0010] The logic optimization command sequence combination method comprises:

[0011] Step one: obtaining an AIG state vector of a to-be-optimized circuit netlist from a logic synthesis environment by the first agent, and inputting an action network facing a delay optimization target to obtain a probability value vector of each action selected in an optimization command action space; selecting optimization command actions according to the probability value vector and sending them into the logic synthesis environment for execution to optimize the to-be-optimized circuit netlist, and further obtaining an AIG state vector, a delay value and an area value of an optimized circuit netlist;

[0012] Step two: obtaining the AIG state vector of the optimized circuit netlist from the logic synthesis environment by the second agent, and inputting an action network facing an area optimization target to obtain a probability value vector of each action selected in an optimization command action space, selecting optimization command actions according to the obtained probability value vector and sending them into the logic synthesis environment for execution to optimize the circuit netlist obtained in the step one again, and further obtaining an AIG state vector, a delay value and an area value of an optimized circuit netlist;

[0013] Step three: repeating the step one and the step two until twice of a preset iteration number N is met;

[0014] Step four: recording all delay values and area values in a sequential game process, and sorting non-dominated solutions to obtain a Pareto frontier of logic optimization command sequence solutions under area and delay optimization targets.

[0015] Optionally, in the training process, a loss value of the evaluation network responsible for global planning is:

[0016]

[0017] wherein, V π (s i ) is an actual output of the evaluation network, and discounted_episode_rewards i represents a discounted reward of the i-th iteration, and the calculation method is:

[0018] discounted_episode_rewards0=episode_rewards0

[0019] discounted_episode_rewards i = discounted_episode_rewards i-1 * γ + episode_rewards i

[0020] wherein episode_rewards0represents an initial value of the global optimization target reward, γ is a discount factor, episode_rewards i represents the global optimization target reward of the i-th iteration.

[0021] Optionally, in the training process, the loss value of the action network oriented to the delay optimization target is:

[0022] actor_loss = neg_log_prob * advantage

[0023]

[0024] advantage = Q π (s i ,a i ) 1 - V π (s i )

[0025] wherein neg_log_prob represents the cross-entropy loss between the softmax normalized action distribution output by the action network oriented to the delay optimization target and the true label y i , ε is a small perturbation amount, advantage represents an advantage function, V π (s i ) is the actual output of the evaluation network, Q π (s i ,a i ) 1 represents a state-action function, and is represented as:

[0026] Q π (s i ,a i ) 1 = shapley 1

[0027] shapley 1 = (discounted_episode_rewards i + discounted_episode_rewards 1i -discounted_episode_rewards 2 i ) / 2

[0028] wherein shapley 1 represents the shapley value allocation of the global optimization objective to the first agent, discounted_episode_rewards 1 i and discounted_episode_rewards 2 i represent the rewards obtained by the first agent and the first agent and the environment in the ith iteration, respectively.

[0029] Optionally, in the training process, the loss value of the action network oriented to the area optimization objective is:

[0030] actor_loss = neg_log_prob * advantage

[0031]

[0032] advantage = Q π (s i ,a i ) 2 -V π (s i )

[0033] wherein neg_log_prob represents the cross-entropy loss between the softmax normalized action distribution output by the action network oriented to the area optimization objective and the true label y i , ε is a small perturbation amount, advantage represents the advantage function, V π (s i ) is the actual output of the evaluation network, Q π (s i ,a i ) 2 represents the state-action function, and is represented as:

[0034] Q π (s i ,a i ) 2 = shapley 2

[0035] shapley 2 = (discounted_episode_rewards i+discounted_episode_rewards 2 i -discounted_episode_rewards 1 i ) / 2

[0036] wherein shapley 2 represents the shapley value allocation of the global optimization target to the second agent, and discounted_episode_rewards 1 i and discounted_episode_rewards 2 i respectively represent the rewards obtained by the first agent and the first agent and the environment in the ith iteration.

[0037] Optionally, the AIG state space includes: input, output, node number, edge connection number, depth, latch number, AND gate number, and NOT gate number of the circuit. Regularization needs to be performed every time the input state is input.

[0038] Optionally, the actions in the optimization command action space include:

[0039] rewriting, fast rewriting, replacing, fast replacing, restructuring, fast restructuring, and balancing.

[0040] Optionally, the logic synthesis environment is composed of Yosys and ABC open source tools.

[0041] The application provides a logic optimization command sequence combination system for implementing the logic optimization command sequence combination method according to any one of the above, and the system includes: a logic synthesis environment, a first agent with an optimization circuit delay as a target, and a second agent with an optimization circuit area as a target, a game reinforcement learning architecture is constructed between the first agent and the second agent by using an A2C algorithm, forming an action network for a delay optimization target, an action network for an area optimization target, and an evaluation network responsible for global planning.

[0042] The first agent is trained by taking the delay optimization target reward and the shapley value allocation of the global optimization target to the first agent as a reward function, and the second agent is trained by taking the area optimization target reward and the shapley value allocation of the global optimization target to the second agent as a reward function.

[0043] The application provides a logic optimization command sequence combination device, including a memory and a processor.

[0044] The memory is used for storing a computer program.

[0045] The processor is configured to implement the logical optimization command sequence combination method according to any one of the preceding embodiments when executing the computer program.

[0046] The application provides a computer readable storage medium, wherein a computer program is stored on the storage medium, and the computer program is configured to implement the logical optimization command sequence combination method according to any one of the preceding embodiments when executed by a processor.

[0047] The application has the following advantages:

[0048] The application takes a netlist to be optimized as input, interacts with an environment by using a reinforcement learning algorithm, selects an action in an action space composed of optimization commands such as reconstruction, rewriting and balancing, and can find an optimal command sequence in a complex solution space. Two agents are introduced, an A2C algorithm is used between the agents, a decentralized action network and a centralized evaluation network architecture are constructed. The decentralized action network can select by using respective action networks and realize sequential game playing; and the centralized evaluation network can measure the value of respective independent actions to the whole. In addition, the application introduces a Shapley value to realize fair distribution of the overall reward among the agents, and solves the problem that an optimal trade-off solution of two optimization targets of area and delay cannot be found in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0050] Figure 1 The figure is a logical optimization command sequence combination method architecture diagram based on game reinforcement learning.

[0051] Figure 2 The figure is an architecture diagram of a decentralized action network and a centralized evaluation network.

[0052] Figure 3 The figure is an evaluation network splicing. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the application clearer, the following will further describe the embodiments of the application in combination with the drawings.

[0054] Embodiment one:

[0055] The embodiment provides a logic optimization command sequence combination method, which introduces a first agent for optimizing circuit delay and a second agent for optimizing circuit area, adopts an A2C algorithm to construct a game reinforcement learning architecture between the first agent and the second agent, forms an action network for a delay optimization target, an action network for an area optimization target, and an evaluation network responsible for global overall planning;

[0056] The first agent is trained by taking a delay optimization target reward and a global optimization target Sharpe value distribution as a reward function, and the second agent is trained by taking an area optimization target reward and a global optimization target Sharpe value distribution as a reward function.

[0057] The logic optimization command sequence combination method of the embodiment comprises the following steps:

[0058] Step one: obtaining an AIG state vector of a to-be-optimized circuit netlist from a logic synthesis environment by the first agent, inputting the AIG state vector into the action network for the delay optimization target, obtaining a probability value vector of each action selected in an optimization command action space, selecting the optimization command action according to the probability value vector, and inputting the optimization command action into the logic synthesis environment for execution, so as to optimize the to-be-optimized circuit netlist and further obtain an AIG state vector, a delay value and an area value of the optimized circuit netlist.

[0059] Step two: obtaining the AIG state vector of the optimized circuit netlist from the logic synthesis environment by the second agent, inputting the AIG state vector into the action network for the area optimization target, obtaining a probability value vector of each action selected in the optimization command action space, selecting the optimization command action according to the obtained probability value vector, inputting the optimization command action into the logic synthesis environment for execution, optimizing the circuit netlist obtained in step one again, and further obtaining an AIG state vector, a delay value and an area value of the optimized circuit netlist.

[0060] Step three: repeating steps one and two until twice a preset iteration number N is satisfied.

[0061] Step four: recording all delay values and area values in the sequential game process, and performing non-dominated solution sorting on the delay values and the area values, to obtain a Pareto frontier of a logic optimization command sequence solution under the area and delay optimization targets.

[0062] Further, the solution in the Pareto frontier can be selected according to a delay or area target tendency according to a user's needs.

[0063] The logic synthesis environment in the embodiment can receive a netlist input, execute an optimization command, return an AIG state and area and delay information after the optimization command is executed, and be composed of Yosys and ABC open source tools.

[0064] The AIG state space is composed of primary I / O, nodes, edges, levels, latches, ANDs, and NOTs. Regularization is needed each time the state is input.

[0065] The optimization command action space is composed of rewrite, rewrite-z, resub, resub-z, refactor, refactor-z, and balance.

[0066] The Shapley value allocation rule is a distribution method that achieves fair distribution according to the contribution of the agent itself to the global optimization target.

[0067] Embodiment Two:

[0068] The embodiment provides a training process of the game reinforcement learning architecture constructed in Embodiment One, and the game reinforcement learning architecture includes a decentralized action network and a centralized evaluation network, and specifically includes an action network for a delay optimization target, an action network for an area optimization target, and an evaluation network responsible for global planning, as shown in Figure 2 .

[0069] The training steps of the game reinforcement learning architecture are as follows:

[0070] The training includes independent training and splicing training.

[0071] First, two agents are designed, each of which has a set of action networks and evaluation networks.

[0072] The structure of the action network designed in the embodiment is two fully connected layers, and the number of neurons in each layer is 20; the structure of the evaluation network is one hidden layer, and the number of neurons is 15.

[0073] The independent training is as follows:

[0074] Step One: Initialize the logic synthesis environment.

[0075] Step Two: The first agent Agent 1 observes the current logic synthesis environment and obtains the AIG state vector at this time. Given the action network for the delay optimization target, the input is the AIG state vector, and the output is a probability value vector of each action selected in the action space. Agent 1 based on the A2C algorithm will select actions according to the output probability distribution and send them to the logic synthesis environment for execution, optimize the netlist, and obtain a new AIG state vector, delay value, and area value.

[0076] Step three: Calculate the delay optimization target reward according to the delay value delay i and the delay constraint delay constraint , the calculation method is shown in formula (1):

[0077]

[0078] wherein rewards_delay i represents the delay optimization target reward in the ith iteration.

[0079] Step four: Agent 1 repeats steps two to three, and sets the total number of iterations as N, at this time step three feeds back a sequence of immediate rewards episode_rewards_delay N . Using formula (2) and formula (3) to calculate the discounted reward, the discounted_episode_rewards_delay i can be obtained, wherein γ is the discount factor.

[0080] discounted_episode_rewards_delay0=episode_rewards_delay0 (2)

[0081]

[0082] wherein episode_rewards_delay0 and discounted_episode_rewards_delay i respectively represent the initial value of the delay optimization target reward and the discounted value of the delay optimization target reward in the ith iteration.

[0083] Step five: Initialize the logic synthesis environment.

[0084] Step six: The second intelligent agent Agent 2 observes the current logic synthesis environment and obtains the AIG state vector at this time. Given the action network oriented to the delay optimization target, the input is the AIG state vector, and the output is the probability value vector of each action selected in the action space. Based on the A2C algorithm, Agent 2 will select actions according to the output probability distribution and send them into the logic synthesis environment for execution to optimize the netlist and obtain a new AIG state vector, delay value and area value.

[0085] Step seven: Calculate the area optimization target reward according to the area value area i and the netlist area initial value area initial , the calculation method is shown in formula (4):

[0086]

[0087] rewards_area i represents the area optimization target reward in the i-th iteration.

[0088] Step eight: Agent 2 repeats steps six to seven, sets the total number of iterations as N, and at this time step seven feeds back a set of immediate reward sequence episode_rewards_area N . Using formula (5) and formula (6) to calculate the discounted reward, the discounted_episode_rewards_area i can be obtained, where γ is the discount factor.

[0089] discounted_episode_rewards_area0=episode_rewards_area0 (5)

[0090]

[0091] where episode_rewards_area0 and discounted_episode_rewards_area i respectively represent the initial value of the area optimization target reward and the discounted value of the area optimization target reward in the i-th iteration.

[0092] Step nine: using formula (7) and formula (8) to calculate the loss function of the evaluation network of Agent 1 and Agent 2. Where V π (s i ) are the actual outputs of the evaluation network of Agent 1 and Agent 2 respectively. Using the Adam algorithm, the evaluation network parameters are optimized with the goal of minimizing critic_loss.

[0093]

[0094] Step ten: using formula (9), (10) and (11) to calculate the loss function of the action network of Agent 1 and Agent 2. Where formula (9) represents the calculation of the cross-entropy loss between the action distribution output by the action network after softmax normalization and the true label y i , ε is a small perturbation to prevent infinite values; formula (10) uses the Q value under the state-action pair (Agent 1 and Agent 2's Q values are respectively discounted_episode_rewards_delay i and discounted_episode_rewards_area iThe difference between the value of the output state of each evaluation network and the value of the output state is used as the advantage function; Formula (11) multiplies Formula (9) and Formula (10) to calculate the loss function of the action network. Since the total number of iterations is set to N, Formula (9) to Formula (11) need to be executed repeatedly when calculating the loss function, and the mean value of actor_loss is taken.

[0095]

[0096] advantage = Q π (s i ,a i )-V π (s i (10)

[0097] actor_loss=neg_log_prob*advantage (11)

[0098] Step 11: Repeat steps 1 through 10 M times to complete the independent training.

[0099] Next, we will practice assembling the parts.

[0100] Step 12: As Figure 3 As shown, keeping the action networks of Agent1 and Agent2 unchanged, the evaluation networks are concatenated to obtain a new shared evaluation network, and then a fully connected layer is added to map the two inputs to a single input. Thus, the distributed action network and centralized evaluation network are obtained.

[0101] Step 13: Initialize the logic synthesis environment.

[0102] Step Fourteen: Agent 1 observes the current logic synthesis environment and obtains the current AIG state vector. Given an action network for delay optimization, with the AIG state vector as input and the probability vector of each action selected in the action space as output, Agent 1, based on the A2C algorithm, selects actions according to the output probability distribution and sends them to the logic synthesis environment for execution. This optimizes the netlist, changes the logic synthesis environment, and yields new AIG state vectors, delay values, and area values.

[0103] Step fifteen: The second agent Agent2 observes the logical synthesis environment at this time, and obtains the AIG state vector at this time. Given the action network oriented to the area optimization target, the input is the AIG state vector, and the output is the probability value vector of each action selected in the action space. Agent2 based on the A2C algorithm will select actions according to the probability distribution of the output. And send it into the logical synthesis environment to execute, optimize the netlist, and the logical synthesis environment changes, get the new AIG state vector, delay value and area value.

[0104] Step sixteen: According to the delay value and the area value, the global optimization target reward is calculated, and the global optimization target reward value is episode_rewards i See Table 1.

[0105] Table 1 Global optimization target reward value

[0106]

[0107] Step seventeen: Agent1 and Agent2 repeat steps fourteen to sixteen, when Agent1 and Agent2 actions are executed, it is recorded as two iterations, and the total number of iterations is set to 2N, at this time step sixteen respectively feedback a group of immediate reward sequence, only take the latter to constitute episode_rewards N . Using formula (12) and formula (13) to calculate the discounted reward, we can get discounted_episode_rewards i , where γ is the discount factor.

[0108] discounted_episode_rewards0=episode_rewards0 (12)

[0109]

[0110] Where, episode_rewards0and discounted_episode_rewards i respectively represent the initial value of the global optimization target reward and the discounted value of the global optimization target reward in the ith iteration.

[0111] Step eighteen: For the action network oriented to the delay optimization target, the contribution to the global optimization target needs to be considered. The contribution to the global optimization target is calculated by using the Shapley value allocation rule, see formula (14). Where, discounted_episode_rewards 1 i and discounted_episode_rewards 2i are the rewards obtained from the environment according to Table 1 by separating the optimization commands performed by Agent 1 and Agent 2 in the i-th iteration and re-interacting with the environment; then, the Q value is calculated using formula (15) and substituted into step twenty-one to obtain the actor_loss mean of the action network facing the delay optimization target. Finally, the Adam algorithm is used to minimize the actor_loss mean to optimize the parameters of the action network facing the delay optimization target.

[0112] shapley 1 =

[0113] (discounted_episode_rewards i + discounted_episode_rewards 1 i (14)

[0114] -discounted_episode_rewards 2 i ) / 2

[0115] Q π (s i ,a i ) 1 = shapley 1 (15)

[0116] where s i represents the state in the i-th iteration, and a i represents the action in the i-th iteration.

[0117] Step nineteen: for the action network facing the area optimization target, the contribution to the global optimization target needs to be considered. The contribution to the global optimization target is calculated using the Shapley value allocation rule, as shown in formula (16). Then, the Q value is calculated using formula (17) and substituted into step twenty-one to obtain the actor_loss mean of the action network facing the area optimization target. Finally, the Adam algorithm is used to minimize the actor_loss mean to optimize the parameters of the action network facing the area target.

[0118] shapley 2 =

[0119] (discounted_episode_rewards i + discounted_episode_rewards 2 i (16)

[0120] - discounted_episode_rewards 1 i ) / 2

[0121] Q π (s i ,a i ) 2 =shapley 2 (17)

[0122] Step twenty: calculate the loss function of the evaluation network by formula (18). Wherein, V π (s_fusion i ) is the actual output obtained by inputting the state of Agent 1 and Agent 2 into the evaluation network. Using the Adam algorithm, the evaluation network parameters are optimized with the goal of minimizing critic_loss.

[0123]

[0124] Step twenty-one: calculate the action network loss value. Wherein, formula (19) represents the cross-entropy loss between the action distribution output by the action network after softmax normalization and the true label y i , and ε is a small perturbation to prevent infinite values; formula (20) uses the difference between the Q value under the state-action pair and the state value (i.e. the output value of the evaluation network under the current state combination) as the advantage function; formula (21) multiplies formula (19) and formula (20) to calculate the loss function of the action network. Since the total number of iterations is set to N, formula (19) to formula (21) need to be repeatedly executed when calculating the loss function, and the average of actor_loss is taken.

[0125]

[0126] advantage = Q π (s i ,a i )-V π (s i ) (20)

[0127] actor_loss = neg_log_prob * advantage (21)

[0128] Step twenty-two: repeat steps thirteen to twenty-one for M times, and the splicing training is completed.

[0129] Example three:

[0130] The embodiment applies the logic optimization command sequence combination method based on game reinforcement learning to carry out experiments.

[0131] In the embodiment, the adder.v, log2.v and multiplier.v provided by the EPFL are selected as source files of high-level description, the open source logic synthesis platform such as abc and yosys is accessed, the asap7.lib is selected as a process library file, and the conversion, optimization and mapping of the source file are carried out.

[0132] The implementation process of the logic optimization command sequence combination method based on game reinforcement learning in the embodiment is applicable to an ASIC or FPGA synthesis environment.

[0133] The algorithm main body in the embodiment is written by using the Python language, and the corresponding logic synthesis command is completed based on the TCL script language. The optimization algorithm selects the optimization command according to the actual situation and writes the corresponding TCL script, sends it into the logic synthesis platform for execution, and reads the delay and area data fed back by the platform by using the regular matching, to guide the following algorithm optimization direction.

[0134] In order to compare the performance of the method, an existing method is selected as a control (the control method is described in reference to “DRiLLS: Deep Reinforcement Learning for Logic Synthesis [C]. 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), 2020: 581-586.”), and experiments are carried out on the method and the control method.

[0135] In the embodiment, the discount factor γ is set to 0.99, the total number of training rounds M is 50, the iteration number N of each training round is 30, and the small perturbation amount ε is 1x10 -6 .

[0136] Table 2: Comparison of experimental results

[0137]

[0138] The data in Table 2 clearly shows the performance difference between the logic optimization command sequence combination method based on game reinforcement learning and the existing method. As can be seen from Table 2, compared with the DRiLLS method, the existing method obtains the coordinated optimization of area and delay on the three inputs. This result shows that the method can effectively solve the contradiction between area and delay, and provides a more comprehensive and balanced solution for logic optimization.

[0139] In summary, the logic optimization command sequence combination method based on game reinforcement learning of the present application has significant advantages and practical value in the field of logic optimization. By effectively balancing the two key optimization targets of area and delay, the method provides a more efficient and accurate logic optimization solution for practical applications. At the same time, its excellent adaptability and robustness also make the method an important reference and choice for future logic optimization work.

[0140] Some steps in the embodiments of the present application can be implemented by software, and the corresponding software programs can be stored in a readable storage medium, such as an optical disc or a hard disk, etc.

[0141] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of combining a sequence of logic optimization commands, the method comprising: The method introduces a first agent aiming at optimizing circuit delay and a second agent aiming at optimizing circuit area, and a game reinforcement learning architecture is constructed between the first agent and the second agent by using an A2C algorithm, forming an action network aiming at a delay optimization target, an action network aiming at an area optimization target, and an evaluation network responsible for global planning; The first agent is trained by using a reward function of a delay optimization target reward and a global optimization target Sharpe value allocation to the first agent, and the second agent is trained by using a reward function of an area optimization target reward and a global optimization target Sharpe value allocation to the second agent. The logic optimization command sequence combination method comprises: Step one: obtaining an AIG state vector of a to-be-optimized circuit netlist from a logic synthesis environment by the first agent, and inputting the AIG state vector into the action network aiming at the delay optimization target to obtain a probability value vector of each action selected in an optimization command action space; selecting the optimization command action according to the probability value vector, and sending the optimization command action into the logic synthesis environment for execution to optimize the to-be-optimized circuit netlist, and further obtaining an AIG state vector, a delay value and an area value of an optimized circuit netlist; Step two: obtaining the AIG state vector of the optimized circuit netlist from the logic synthesis environment by the second agent, and inputting the AIG state vector into the action network aiming at the area optimization target to obtain a probability value vector of each action selected in the optimization command action space; selecting the optimization command action according to the obtained probability value vector, and sending the optimization command action into the logic synthesis environment for execution to optimize the circuit netlist obtained in the step one, and further obtaining an AIG state vector, a delay value and an area value of an optimized circuit netlist; Step three: repeating the step one and the step two, and setting a total iteration number as 2N until the first agent and the second agent are executed to meet the preset iteration number 2N; Step four: recording all delay values and area values in a sequential game process, and performing non-dominated solution sorting on all delay values and area values to obtain a Pareto frontier of a logic optimization command sequence solution under area and delay optimization targets.

2. The method of claim 1, wherein, In the training process, a loss value of the evaluation network responsible for global planning is: where V π (s i ) is the actual output of the evaluation network, discounted_episode_rewards i represents the global optimization target discounted reward of the i-th iteration, and the calculation method is as follows: discounted_episode_rewards0=episode_rewards0 discounted_episode_rewards i = discounted_episode_rewards i-1 * gamma + episode_rewards i wherein episode_rewards0represents the initial value of the global optimization target reward, γ is a discount factor, and episode_rewards i represents the global optimization target reward of the i-th iteration.

3. The method of claim 1, wherein, In the training process, a loss value of the action network aiming at the delay optimization target is: actor_loss=neg_log_prob*advantage advantage=Q π (s i ,a i ) 1 -V π (s i ) wherein neg_log_prob represents a cross-entropy loss between a softmax-normalized action distribution of an action network output for the latency-oriented optimization target and a true label y i , ε is a tiny perturbation amount, advantage represents an advantage function, V π (s i ) is an actual output of the evaluation network, Q π (s i ,a i ) 1 represents a state-action function, and is represented as: Q π (s i ,a i ) 1 =shapley 1 shapley 1 = (discounted_episode_rewards i +discounted_episode_rewards 1 i - discounted_episode_rewards 2 i ) / 2 where s i represents the state in the i-th iteration, a i represents the action in the i-th iteration, shapley 1 represents the global optimization target's Shapley value assignment to the first agent, discounted_episode_rewards 1 i and discounted_episode_rewards 2 i respectively represent separating out the optimization command performed in the i-th iteration, the rewards obtained by the first agent and the second agent respectively re-performing the optimization command and interacting with the environment.

4. The method of claim 1, wherein, In the training process, a loss value of the action network aiming at the area optimization target is: actor_loss=neg_log_prob*advantage advantage=Q π (s i ,a i ) 2 -V π (s i ) wherein neg_log_prob represents a cross-entropy loss between a softmax-normalized action distribution of an action network output for the area-oriented optimization target and a true label y i , ε is a tiny perturbation amount, advantage represents an advantage function, V π (s i ) is an actual output of the evaluation network, Q π (s i ,a i ) 2 represents a state-action function, and is represented as: Q π (s i ,a i ) 2 =shapley 2 shapley 2 = (discounted_episode_rewards i +discounted_episode_rewards 2 i - discounted_episode_rewards 1 i ) / 2 where s i represents the state in the i-th iteration, a i represents the action in the i-th iteration, shapley 2 represents the global optimization target's Shapley value assignment to the second agent, discounted_episode_rewards 1 i and discounted_episode_rewards 2 i respectively represent separating out the optimization command performed in the i-th iteration, the rewards obtained by the first and second agents respectively re-executing the optimization command and interacting with the environment.

5. The method of claim 1, wherein, The AIG state space comprises: circuit input, circuit output, node number, edge connection number, depth, latch number, AND gate number and NOT gate number.

6. The method of claim 1, wherein, The actions in the optimization command action space comprise: rewriting, replacing, reconstructing and balancing.

7. The method of claim 1, wherein, The logic synthesis environment is composed of Yosys and ABC open source tools.

8. A logic optimization command sequence assembly system, characterized by, To realize the logic optimization command sequence combination method as claimed in any one of claims 1-7, the system comprises a logic synthesis environment, a first agent aiming at optimizing circuit delay and a second agent aiming at optimizing circuit area, a game reinforcement learning architecture is built between the first agent and the second agent by using A2C algorithm, forming an action network for delay optimization target, an action network for area optimization target, and an evaluation network responsible for global planning; The first agent is trained by using a reward function of delay optimization target reward and global optimization target Sharpe value allocation to the first agent, and the second agent is trained by using a reward function of area optimization target reward and global optimization target Sharpe value allocation to the second agent.

9. A logic optimization command sequence assembly apparatus, characterized by, comprising a memory and a processor; The memory is used to store a computer program. The processor is used to realize the logic optimization command sequence combination method as claimed in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the logic optimization command sequence combination method as claimed in any one of claims 1-7 is realized.

Images