A reward correction-based reinforcement learning logic synthesis sequence optimization space exploration method and system

Abstract

The present application relates to a kind of logic synthesis sequence optimization space exploration method and system based on reward correction reinforcement learning, belong to the technical field of electronic design automation, logic synthesis process is embedded into the reinforcement learning algorithm based on reward correction, logic synthesis process is modeled as the environment of reinforcement learning, realize automatic exploration logic synthesis sequence optimization space.The reinforcement learning algorithm is improved for logic synthesis scene, carries out feature extraction and action coding, improves the precision of circuit state representation, expands search space.The present application increases the exploration ability in early training by adding reward correction, obtains adaptive reward function by agent training, reduces the performance decline caused by the limitation of prior knowledge on circuit optimization direction, obtains more optimal logic synthesis area result under the condition of meeting delay constraint.

Description

TECHNICAL FIELD

[0001] The present application relates to a reward correction-based reinforcement learning logic synthesis sequence optimization space exploration method and system, belonging to the technical field of electronic design automation. BACKGROUND

[0002] Electronic Design Automation (EDA) is a set of computer-aided tools and methods for supporting the design and optimization of complex electronic systems. The goal of EDA technology is to accelerate the design, verification, and optimization of integrated circuits (ICs), system-on-chip (SoC), and programmable logic devices (such as FPGA) through efficient automation processes. It covers all aspects from design input, functional verification, logic synthesis to routing, physical implementation, and performance testing, and is an indispensable technical support for modern semiconductor design.

[0003] As chip technology continues to evolve towards nanoscale, the size and complexity of circuit design grow exponentially. At present, design projects often involve hundreds of millions or even billions of transistors, which puts higher requirements on EDA tools. In this context, how to improve design efficiency, shorten development cycle and optimize performance has become an important research direction in the field of electronic design.

[0004] In the EDA process, logic synthesis is one of the key steps. The main task of logic synthesis is to convert high-level abstract design descriptions (such as RTL code written in hardware description languages such as Verilog or VHDL) into gate-level netlists. In this process, not only the design function is implemented, but also several key performance indicators are optimized, such as: area (reducing the number of logic gates and resource occupation), delay (reducing signal propagation delay in the circuit), power consumption (reducing power consumption to improve energy efficiency). Logic synthesis is a multi-objective optimization problem that needs to find the best balance between area, delay, and power consumption. Traditional logic synthesis tools mostly use heuristic algorithms or rule-driven optimization processes, applying predefined circuit transformation scripts to optimize the design. However, as the circuit size and design constraints continue to increase, the optimization difficulty is also gradually rising.

[0005] In traditional logic synthesis process, the optimization effect depends on the selected circuit optimization scripts and their execution order. Common optimization operations include logic merging, redundancy elimination, timing adjustment, gate-level optimization, etc. These operations need to be selected according to the specific circuit characteristics, and the trade-off between area, delay and power consumption, etc. However, as the design complexity increases, the combination and order of circuit optimization scripts show exponential growth, making the exploration of the optimization space extremely difficult. In addition, traditional heuristic algorithms can only search within a limited local solution space, and are prone to fall into local optimal solution, which cannot guarantee the discovery of global optimal solution. The quality of the optimization result depends largely on the manual adjustment of parameters and strategies by experienced engineers, which is not only time-consuming but also difficult to adapt to the needs of large-scale design.

[0006] In order to cope with the above challenges, in recent years, reinforcement learning (RL) has been introduced into the field of logic synthesis as a new research direction. Reinforcement learning is a machine learning method suitable for sequential decision-making problems, especially in scenarios where the state and action space is huge and needs to be optimized gradually.

[0007] In logic synthesis, reinforcement learning can dynamically explore different combinations and orders of optimization scripts, learn the optimized strategy, and gradually improve the result quality. By introducing reinforcement learning, the system no longer relies on human experience to select optimization scripts, but automatically searches for the optimal solution in the design space through the model, thereby solving the problem that traditional heuristic methods are prone to fall into local optimal solution. SUMMARY

[0008] In view of the deficiencies of the prior art, the present application provides a logic synthesis sequence optimization space exploration method and system based on reward correction reinforcement learning. The present application embeds the rules of logic synthesis circuit optimization into the reinforcement learning algorithm based on reward correction, models the logic synthesis process as a reinforcement learning environment, and realizes automatic exploration of the logic synthesis sequence optimization space. The present application improves the accuracy of circuit state representation and expands the search space by improving the reinforcement learning algorithm for the logic synthesis scenario, feature extraction and action coding. In the process of automatically generating sequences by traditional algorithm exploration, too much human prior knowledge intervention limits the circuit state exploration process. The present application increases the exploration ability in the early training stage by adding reward correction, obtains an adaptive reward function from the agent training, reduces the performance decline caused by the limitation of prior knowledge on the circuit optimization direction, and obtains a better logic synthesis area result under the condition of meeting the delay constraint.

[0009] Term explanation:

[0010] 1、AIG circuit: A graph structure used to represent and optimize logic circuits, widely used in logic synthesis and verification in the field of electronic design automation. AIG represents Boolean logic functions in a concise and efficient way, consisting of AND gates and inverters. Its core feature is to build logic networks using only two-input AND gates, improving the simplicity of the structure.

[0011] 2、Agent: Agent refers to an autonomous system that can perceive, make decisions, and take actions in an environment to achieve a certain goal. In the fields of artificial intelligence (AI) and reinforcement learning (RL), Agent is a core concept representing the subject that performs a task or interacts with the environment.

[0012] 3、Result quality: Result quality (QoR) is a comprehensive indicator used to measure the quality of the final outcome in the fields of electronic design automation (EDA), machine learning, and other engineering fields. In different application scenarios, the specific meaning and evaluation dimensions represented by QoR are different, and in this invention, it represents the trade-off between area and delay.

[0013] 4、Circuit performance parameters: Also known as circuit design indicators or circuit performance values. For example, the area, delay, power consumption, etc. of the circuit.

[0014] 5、LUT netlist: LUT netlist is a form used to represent circuit structure in electronic design automation (EDA) and field programmable gate array (FPGA) design. LUT (Look-Up Table) is a basic unit used to implement logic functions, and in FPGA, logic circuits are usually mapped to a set of LUTs and their interconnections. LUT netlist describes these LUTs and their connections to build complex logic circuits.

[0015] 6、Graph convolutional network: Graph convolutional network (GCN) is a deep learning model used to process graph-structured data. Its main feature is to effectively learn representations from the information of nodes and their neighborhoods.

[0016] The technical solutions of the invention are as follows:

[0017] A logic synthesis sequence optimization space exploration method based on reward correction reinforcement learning, the implementation method is to take the logic synthesis process as the environment of the reinforcement learning algorithm based on reward correction, and use the reinforcement learning iterative action generation method to select and optimize the script, and combine into the final optimization script sequence. The method can reduce the influence of prior knowledge, and find an area optimal solution meeting the delay constraint with higher efficiency, including the following steps:

[0018] Step 1: Obtain the AIG circuit characteristics, and set the characteristics as the input of the reinforcement learning agent;

[0019] Step 2: Train the latent space of the optimization script, and set the latent space as the action space of the reinforcement learning agent;

[0020] Step 3: Build a double-agent model, including a policy agent and a reward agent, set the policy agent to generate the optimization script of each iteration, and the reward agent to generate the correction reward value of each iteration;

[0021] Step 4: Policy agent iteration, select the optimization script to be executed according to the current circuit characteristics, and interact with the environment to obtain the next time circuit characteristics and the environment reward of the current iteration;

[0022] Step 5: Reward agent iteration, combine the current circuit characteristics obtained in step 4 and the selected action as the state of the reward agent, select the correction reward value according to the state, and feed back to the policy agent, integrate the correction reward and the environment reward to form the total reward;

[0023] Step 6: Trajectory temporary storage, set a trajectory of the policy agent as the current circuit characteristics, the current executed optimization script, the current total reward, and the next time circuit characteristics, which are obtained from steps 4 and 5;

[0024] Set a trajectory of the reward agent as the current circuit characteristics, the correction reward value of the current reward, the current environment reward, and the next time circuit characteristics, which are obtained from steps 4 and 5;

[0025] Step 7: Agent update, judge whether the set number of trajectory data is collected, if the set number of trajectory data is collected, update the network parameters of the agent, and clear the trajectory data of the policy agent, otherwise do not update, continue to collect trajectory data;

[0026] Step 8: Judge whether the stop condition is reached, if the iteration number reaches the set upper limit, save the current model, output the optimization script sequence, otherwise, repeat steps 4-8;

[0027] Step 9: For the circuit used for training, the optimized script sequence output in step 8 is input into the logic synthesis tool to obtain the area and delay of the optimized circuit, and for the circuit used for testing, the model saved in step 8 is loaded to generate an optimized sequence for the current test circuit, which is input into the logic synthesis tool to obtain the area and delay of the optimized circuit.

[0028] Preferably, in step 1, the AIG circuit features include inherent features of the AIG circuit and graph features of the AIG circuit, the inherent features of the AIG circuit include: the number of AND gates, the number of connection edges, the length of the critical path, the number of nodes after mapping to the LUT netlist, the number of logic stages after mapping, the inherent features generate a vector representation of the inherent features through a fully connected layer, the number of layers of the fully connected layer is at least two, and further preferably, the number of layers is adjusted between two and four according to the complexity of the inherent features.

[0029] Further, the graph features of the AIG circuit are obtained by a graph convolution network, the graph convolution network includes convolution layers and pooling layers, the number of layers of the convolution layers is at least three, and further, the number of layers is adjusted between three and five according to the complexity of the AIG circuit, the local information of each node is integrated into global features through the convolution layers to form the graph features; the inherent features and the graph features are spliced into a complete circuit feature vector, which is integrated through at least one fully connected layer to obtain an integrated circuit feature vector, which is standardized and used as an input of the reinforcement learning algorithm to describe the circuit state.

[0030] Preferably, in step 2, the latent space of the training optimization script is realized by a self-encoder, the self-encoder includes an encoder and a decoder, both the encoder and the decoder include two to four fully connected layers, the total number of layers of the fully connected layers is adjusted between four and eight based on the complexity and running speed of the AIG circuit, wherein the encoder generates the latent space for agent training, and the decoder decodes the original optimization script for environment interaction.

[0031] Further, in constructing the latent action space, the initial optimization script set is first encoded by the encoder to form an encoded latent action space, the vectors in the latent action space are encoded representations of the initial optimization script, and then the vectors in the latent action space are restored to the original optimization script by the decoder to ensure lossless reconstruction of information.

[0032] Preferably, the strategy agent and the reward agent in step 3 are two collaborative training agents constructed based on reinforcement learning algorithms, wherein the reward agent is implemented using a Deep Deterministic Policy Gradient (DDPG) algorithm, and the policy network and the value network of the reward agent each contain at least two fully connected layers; the strategy agent is implemented using a Proximal Policy Optimization (PPO) algorithm, and the policy network and the value network of the strategy agent each contain at least two fully connected layers.

[0033] Preferably, in step 4, a probability distribution of the potential action space is generated according to the current circuit state, a potential action is selected according to the probability distribution, the probability distribution of the potential action space is the probability of each potential action being selected, the potential action is decoded by the decoder constructed in step 2 to obtain an optimization script that can be recognized by a logic synthesis tool, and then the script is input into the logic synthesis tool to obtain a new AIG circuit, and the process returns to step 1 to construct an AIG circuit feature vector, to obtain a circuit feature at the next time and an environment reward of the current iteration.

[0034] Preferably, the total reward in step 5 is:

[0035]

[0036] In the formula, Area init is the area of the initial AIG circuit, Delay constraint is the delay constraint on the circuit, Area t and Delay t are the area and delay at the current time, respectively, and α and β are weight factors of the corresponding terms, is the reward correction value at the current time;

[0037] According to the state at the next time, the action at the next time is selected, and then the state at the next time defined in the reward agent and the environment reward of the current iteration are obtained.

[0038] Further, the network parameters of the updated agent in step 7 refer to the weights and biases of each layer of fully connected layers of the policy network and the value network of the strategy agent, and the policy network and the value network of the reward agent.

[0039] Further, the strategy agent and the reward agent adopt different update strategies. The strategy agent adopts a same-strategy mode, updates its parameters using the trajectory collected from the current AIG circuit, and the reward agent adopts a different-strategy mode, uses an experience replay buffer, and updates its parameters using the historical trajectory in the buffer.

[0040] Further, in step 7, the number of groups is set to 20 groups, if 20 groups of trajectory data are collected, the network parameters of the agent are updated, and the trajectory data of the strategy agent are cleared, otherwise no update is made, and the trajectory data continues to be collected.

[0041] A reward correction-based reinforcement learning logic synthesis sequence optimization space exploration system comprises:

[0042] The feature extraction module is configured to obtain a circuit feature vector, obtain graph features of each node and its adjacent nodes by using a graph convolution method, obtain inherent features of the circuit by reading the initial circuit, and combine the two parts of features as final feature output.

[0043] The action space generation module is configured to obtain a latent space for training by encoding an initial optimization script, decode the latent space to obtain an optimization script that can be recognized by a logic synthesis tool, and connect a reinforcement learning agent and a training environment.

[0044] The model construction module is configured to save and update network parameters of two reinforcement learning agents.

[0045] The strategy generation module is configured to generate a correction reward for each iteration by using network parameters of a reward agent, and generate an optimization script for each iteration by using network parameters of a strategy agent.

[0046] The beneficial effects of the present application are:

[0047] 1. The present application proposes a reward correction-based reinforcement learning logic synthesis sequence optimization space exploration method and system, which embeds the rules of logic synthesis circuit optimization into the reinforcement learning algorithm, uses an improved reinforcement learning algorithm to adaptively adjust the reward function to increase the exploration intensity, optimizes the exploration process, and balances exploration and utilization in the optimization process.

[0048] 2. The present application proposes a reward correction-based reinforcement learning logic synthesis sequence optimization space exploration method and system, which converts an AIG circuit into an adjacency matrix, combines logic synthesis with a graph convolution network, and improves the accuracy of state representation.

[0049] 3. The present application proposes a reward correction-based reinforcement learning logic synthesis sequence optimization space exploration method and system, which combines optimization scripts in the logic synthesis process with an autoencoder, simplifies the action space, and solves the problem of agent training not converging caused by complex action space.

[0050] 4, The application provides a reward correction-based reinforcement learning logic synthesis sequence optimization space exploration method and system, which improves the traditional reinforcement learning algorithm by combining logic synthesis field knowledge, so that the improved reinforcement learning method is more suitable for the logic synthesis process, and adaptive optimization adjustment of different circuits is realized.

[0051] 5, The application provides a reward correction-based reinforcement learning logic synthesis sequence optimization space exploration method and system, which combines more optimization scripts, and realizes more efficient and rapid optimization compared with human experts. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 Fig. 1 is a flowchart of the reward correction-based reinforcement learning logic synthesis sequence optimization space exploration method and system of the application;

[0053] Figure 2 Fig. 2 is a feature extraction model network structure schematic diagram of embodiment 1;

[0054] Figure 3 Fig. 3 is an autoencoder network structure schematic diagram of embodiment 1;

[0055] Figure 4 Fig. 4 is an agent structure schematic diagram of embodiment 1;

[0056] Figure 5 Fig. 5 is a key step model training flowchart of embodiment 1;

[0057] Figure 6 Fig. 6 is a scatter plot of exploration result distribution of embodiment 1. DETAILED DESCRIPTION

[0058] The application will be further described below through embodiments and in combination with the drawings, but is not limited thereto.

[0059] Embodiment 1:

[0060] Embodiment 1 of the application uses an improved reinforcement learning algorithm to optimize the design of a case in the logic synthesis process. Table 1 lists the initial circuit indicators and their values.

[0061] Table 1: Initial circuit indicators and their values

[0062]

[0063] This embodiment accelerates the design space exploration by temporarily storing multiple intermediate processes, evaluates the results of each synthesis process and saves the optimal result, i.e. the minimum area that meets the delay constraint, then further explores according to the synthesis result of the previous moment until the predetermined number of training rounds is reached, realizes automatic exploration of the optimization space, and meets the area optimization of the delay constraint.

[0064] In this embodiment, the AIG circuit input system uses the open-source AIG circuit file from EPFL, the synthesis tool uses BerkeleyABC, and the process library uses the ASAP7nm standard library. However, the input AIG circuit file, synthesis tool, and process library are not limited to the above examples.

[0065] Figure 1 This is a flow chart of a logic synthesis sequence optimization space exploration method and system based on reward-corrected reinforcement learning of the present invention. Figure 1 As shown, at step 101, an AIG circuit is selected as an input file, and the selected AIG circuit is input into step 102 to obtain circuit features, which are then input into the reinforcement learning agent. In the AIG circuit, the circuit is represented only by AND gates and NOT gates, and the connections between the circuits are represented by solid and dashed lines. A solid line represents a direct connection between two AND gates, and a dashed line represents a NOT gate inserted between two AND gates. In different embodiments, the size of the AIG circuit can vary, with differences reflected in the number of AND gates, NOT gates, and connecting lines between gate units in the AIG circuit. To obtain circuit features in step 102, a graph convolutional network algorithm is used to convert the circuit gate units and connecting line information into an adjacency matrix. This matrix is ​​then input into a convolutional layer for data dimensionality reduction, resulting in a simplified feature vector that can represent the graph features. This vector is used to represent the circuit state and provide data support for reinforcement learning training.

[0066] Then, at step 103, the strategy agent and the reward agent interact with the logic synthesis environment to obtain the circuit state information, the executed actions, the environment rewards and the modified rewards. In one implementation scenario, the optimization objectives of the aforementioned strategy agent and the reward agent are different. The strategy agent interacts with the logic synthesis environment, and its goal is to select a better optimization script based on the current state. Therefore, it is necessary to establish a reinforcement learning model for a discrete action space; while the reward agent does not directly interact with the logic synthesis environment. Its goal is to generate a modified reward value based on the current state and the optimization script, which can be a discrete or continuous action space. In this embodiment, a reinforcement learning model for a continuous action space is established based on deep deterministic policy gradients. The two models include multiple layers of fully connected layers, and the output dimensions of the final layers of the fully connected layers of each network are the dimensions of the desired state and the dimensions of the action, respectively.

[0067] In this embodiment, the state information of the circuit is obtained by combining inherent features and graph features, the executed action is represented as the action estimate of the policy network of the policy agent, the environmental reward is calculated from the current state information, and the corrected reward is represented as the reward estimate of the policy network of the reward agent, ultimately forming a trajectory.

[0068] After obtaining the above-mentioned trajectory, at step 104, the model parameters are trained based on the obtained trajectory. In this embodiment, the trajectory is set to a round length. This parameter is manually set based on the circuit characteristics and can be increased or decreased as appropriate. If the circuit scale is small, the circuit state set will be relatively small, and therefore the number of reachable states will be smaller, so fewer trajectories can be used to train the model. If the circuit scale is large, the number of possible states during training will also increase, so the number of trajectories needs to be appropriately increased to ensure sufficient model training.

[0069] The specific implementation steps are as follows:

[0070] Step 1: Construct AIG circuit features. In order to more accurately represent the state of a circuit during the optimization process, the required circuit features include: the inherent features of the AIG circuit, such as the number of AND gates, the number of connecting edges, the length of the critical path, the number of nodes after mapping to the LUT netlist, and the number of logic levels after mapping. The inherent features are generated into a vector representation of the inherent features through the fully connected layer, such as Figure 2 As shown in , the number of fully connected layers is two. The graph features of the AIG circuit are obtained by the graph convolutional network, as shown in Figure 2 As shown in the figure, the graph convolutional network consists of three convolutional layers and one pooling layer. The specific number of layers increases according to the complexity of the circuit. The node degree of the circuit and the node degree features of its adjacent nodes are aggregated through the convolutional layer to form an intermediate layer network that can represent local features. The final graph feature vector is convolved again to form a fully connected layer together with the inherent feature vector to form a circuit feature vector, which represents the current circuit state. This state is the state input of reinforcement learning.

[0071] Figure 2 This is a schematic diagram of the feature extraction model network structure, as shown in Figure 2 As shown in , the feature extraction model includes a graph convolution model 201, an inherent feature model 202 and a feature combination model 203. The graph convolution model 201, the inherent feature model 202 and the feature combination model 203 are only set for this embodiment. Corresponding to different embodiments or different scenarios, the number of network layers and dimensions of the model can be dynamically adjusted. In this embodiment, the graph convolution model 201 is serially connected through three layers of convolution layers, and then passes through a pooling layer to obtain a simplified graph feature vector. The input of the inherent feature model 202 is the inherent features of the circuit extracted by the logic synthesis tool, such as the number of AND gates in the circuit. The data is dimensioned through two layers of fully connected layers to accurately represent the inherent state. Then, the feature combination model 203 cascades the graph features and inherent features obtained by the graph convolution model 201 and the inherent feature model 202, and passes through a fully connected layer together to reduce the dimension of the data again, so as to accurately represent the circuit state with a simplified feature vector.

[0072] Step 2: Training the latent action space. During the logic synthesis process, there are a large number of optimization scripts and corresponding parameters, forming a huge action set. Table 2 lists some examples of Berkeley ABC scripts.

[0073] Table 2 Some script examples

[0074]

[0075] The latent space for training the optimization script is implemented by an autoencoder, which includes an encoder and a decoder. Both the encoder and the decoder include two fully connected layers. Based on the complexity and running speed of the AIG circuit, the total number of fully connected layers is adjusted between four and eight layers. The encoder generates the latent space for intelligent agent training, and the decoder decodes the original optimization script for environmental interaction.

[0076] When constructing the latent action space, the initial set of optimization scripts (such as rewrite-lz and resub-lz) is first encoded by an encoder to form an encoded latent action space. The vectors in the latent action space are the encoded representations of the initial optimization scripts. Subsequently, the vectors in the latent action space are restored to the original optimization scripts through a decoder to ensure lossless reconstruction of the information. Through this process, the autoencoder captures the structural features of the optimization script set, providing a more compact and easy-to-explore latent action space for reinforcement learning models.

[0077] Figure 3 is a schematic diagram of the autoencoder network structure, as shown in Figure 3 As shown in , the autoencoder 300 includes an encoder 302 and a decoder 304. The encoder 300 is only set for this embodiment. Corresponding to different embodiments or different scenarios, the number of network layers and dimensions of the encoder 302 and the decoder 304 can be dynamically adjusted. In this embodiment, the encoder 302 is serially connected by two encoding layers (fully connected layers), and the decoder 304 is serially connected by two decoding layers (fully connected layers). The parameterized optimization script 301 is encoded and mapped to the potential action space 303 by the encoder 302 as the action space of the intelligent agent. During the training of the intelligent agent, the action in the potential action space 303 is generated, and then decoded by the decoder 304 to obtain the parameterized optimization script 305. The result is a lossless decoding of the parameterized optimization script 301, which is used to map to the optimization script that can be recognized by the logic synthesis tool after the intelligent agent selects the potential action, and feed it back to the logic synthesis tool for execution to obtain the state and reward information used to update the parameters. By constructing the autoencoder 300, the exploration scope of the logic synthesis optimization script is expanded, while the training burden of reinforcement learning in complex action spaces is reduced. On the basis of improving the exploration capability, the convergence speed of training is taken into account.

[0078] Step 3: Build the policy agent and reward agent. The policy agent is based on the Proximal Policy Optimization (PPO) algorithm and consists of a policy network and a value network, each with two fully connected layers. The policy network's fully connected layers are then activated by a Softmax function to generate an action distribution. The policy network is set to select actions in the latent space, and the value network is used to evaluate the state value of the current policy. The reward agent is based on the Deep Deterministic Policy Gradient (DDPG) algorithm and consists of a policy network and a value network, each with two fully connected layers. The policy network's fully connected layers are then activated by a Tanh function to generate a modified reward, which is fed back to the policy agent. The value network evaluates the effectiveness of the modified reward.

[0079] Figure 4 is a schematic diagram of the intelligent agent structure, as shown in Figure 4 As shown, the agent 400 includes a strategy agent 402 and a reward agent 403, wherein the strategy agent 402 includes a strategy network 404 and a value network 405, and the reward agent 403 includes a strategy network 406 and a value network 407. The agent 400 is only set for this embodiment. Corresponding to different embodiments or different scenarios, the agent composition and the number of network layers can be dynamically adjusted. In this embodiment, the strategy network 404 is composed of two fully connected layers, and then generates the distribution of actions through the Softmax activation function. The optimal action is selected according to the probability distribution. The action is the action in the potential action space 303, and the parameterized optimization script 305 is obtained by decoding through the decoder 304. The value network 405 is composed of two fully connected layers. After the data is upgraded and reduced in dimension, a numerical value is obtained. The numerical value represents the state value of the current state. The strategy network 406 is also composed of two fully connected layers. After the Tanh activation function, a modified reward is generated, which is used to adaptively adjust the reward function according to different embodiments. The value network 407 is composed of two fully connected layers, which evaluates the effect of the corrected reward generated by the reward agent 403 and helps update the network parameters of the agent 403 in the direction of reducing the prediction error.

[0080] Step 4: Policy Agent Iteration. Based on the current circuit state, a probability distribution of the potential action space is generated. Potential actions are selected based on this probability distribution. The probability distribution of the potential action space represents the probability of each potential action being selected. Furthermore, the decoder constructed in Step 2 is used to decode the potential actions into an optimization script that can be recognized by the logic synthesis tool. This script is then input into the logic synthesis tool to generate a new AIG circuit. The process then returns to Step 1 to construct the AIG circuit feature vector, obtaining the circuit features for the next moment and the environmental reward for the current iteration.

[0081] Step 5: Reward agent iteration. According to the current time circuit characteristics obtained in step 4 and the selected action combined as the state of the reward agent, the modified reward value is selected according to the state, and the total reward is fed back to the policy agent to form the total reward, which is:

[0082]

[0083] In the formula, Area init is the area of the initial AIG circuit, Delay constraint is the delay constraint for the circuit, Area t and Delay t are the area and delay at the current time, and a and b are the weight factors of the corresponding items, is the reward modification value at the current time.

[0084] According to the next time state, the next time action is selected, and then the next time state defined in the reward agent and the environment reward of the current iteration are obtained.

[0085] Step 6: Trajectory temporary storage. The policy agent is constructed based on the same policy proximal policy optimization algorithm, and each trajectory temporarily stored is obtained in step 4, including the current circuit characteristics, the currently executed optimization script, the current total reward, the next time circuit characteristics, and the same policy method only uses the trajectory of the current round for parameter updating. After the trajectory of the current round is used, the trajectory needs to be cleared; the reward agent is constructed based on the different policy deep deterministic policy gradient method, and each trajectory temporarily stored is obtained in step 5, including the current circuit characteristics, the current reward modification item, the current environment reward, and the next time circuit characteristics. The different policy method has an experience replay buffer that saves the trajectory of each iteration without covering the trajectory.

[0086] Step 7: Agent update. It is judged whether 20 sets of trajectory data are collected. If 20 sets of trajectory data are collected, the network parameters of the agent are updated, and the trajectory of the policy agent is cleared. Otherwise, no update is made, and the trajectory data continues to be collected. The collected data is the trajectory data defined in step 6.

[0087] Updating the network parameters of the agent means updating the weights and biases of each layer of fully connected layer nodes of the policy network and value network of the policy agent, and the policy network and value network of the reward agent.

[0088] The policy agent and the reward agent use different update strategies. The policy agent uses the same policy method, uses a set of trajectories collected from the current AIG circuit to update its parameters, and the reward agent uses the different policy method, uses the experience replay buffer to collect historical trajectories in the buffer to update its parameters.

[0089] Step 8: Determine whether the stop condition is reached. If the number of iterations reaches the set upper limit, save the current model, output the logical synthesis optimization script sequence, otherwise, repeat steps 4-8.

[0090] Step 9: For the circuit used for training, input the optimization script sequence output in step 8 into the logic synthesis tool to obtain the area and delay of the optimized circuit, for the circuit used for testing, load the model saved in step 8, generate an optimization sequence for the current test circuit, input into the logic synthesis tool, and obtain the area and delay of the optimized circuit.

[0091] Figure 5 The flow chart for training the key step model is as shown in Figure 5 The feature extraction model 200 is applied to obtain the circuit state, the autoencoder 300 is applied to interact with the logic synthesis tool to obtain the next time state and environmental reward, and the correction reward. In this embodiment, the policy agent, the reward agent, and the trajectory buffer are initialized, then the circuit state is obtained through the feature extraction model, and then the action is selected according to the circuit state. At this time, the action is in the latent space, so the next time state and the environmental reward and the correction reward obtained by interacting with the logic synthesis tool need to be combined with the autoencoder 300 to decode the optimization script that can be recognized by the logic synthesis tool. Then, the current trajectory is saved to the buffer of each agent respectively, and the update condition is judged. If the update condition is met, the network parameters of the policy agent and the reward agent are updated, and then the stop condition is judged. If the update condition is not met, the stop condition is directly judged. If the stop condition is met, the logical synthesis optimization sequence is output and the model is saved before exiting, otherwise, the circuit state is obtained to continue iteration.

[0092] Figure 6 The scatter plot of the exploration result distribution of this embodiment is shown in Figure 6 The gray data points represent the circuit state at each time obtained during the exploration process, the blue data points represent the Pareto boundary in the exploration process of this embodiment, the red solid points and multiplication signs represent the optimal performance of the circuit obtained by other methods, the red plus sign represents the performance of the initial circuit, and the red dashed line represents the delay constraint of this embodiment. The black triangle is the circuit performance result obtained by this embodiment which satisfies the delay constraint and has the optimal area. From Figure 6 It can be seen that the Pareto curve obtained by using the improved reinforcement learning algorithm to explore the logical synthesis sequence optimization space satisfies the trade-off relationship between area and delay. When one kind of performance is optimized, the other kind of performance is often reduced. Compared with other methods, the method used in this embodiment has better area under the same delay constraint, which proves the effectiveness of the method in exploring the logical synthesis sequence optimization space.

[0093] Embodiment 2

[0094] A logic synthesis sequence optimization space exploration system based on reward correction reinforcement learning, comprising:

[0095] The feature extraction module is configured to obtain a circuit feature vector, obtain the graph features of each node and its adjacent nodes by using a graph convolution method, obtain the inherent features of the circuit by reading the initial circuit, and combine the two parts of features as the final feature output;

[0096] The action space generation module is configured to obtain a latent space for training by encoding the initial optimization script, decode the latent space to obtain an optimization script that can be recognized by the logic synthesis tool, and connect the reinforcement learning agent and the training environment;

[0097] The model construction module is configured to save and update the network parameters of the two reinforcement learning agents;

[0098] The strategy generation module is configured to generate a correction reward for each iteration by the network parameters of the reward agent, and generate an optimization script for each iteration by the network parameters of the strategy agent.

[0099] The above-described embodiments are only preferred embodiments of the present application, and Embodiment 1 is described in more detail and in more detail, but it should not be construed as limiting the scope of the patent. For Embodiment 2, since it is basically similar to the method embodiment, the description is relatively simple, and the implementation of the related modules can be referred to the part of the description of Method Embodiment 1. It should be noted that for ordinary skilled in the art, some improvements and replacements without departing from the principles of the present application should be considered as the protection scope of the present application. Therefore, the protection scope of the present application patent should be subject to the protection scope of the claims.

Claims

1. A logic synthesis sequence optimization space exploration method based on reward correction reinforcement learning, characterized in that, The steps include the following: Step 1: Obtain AIG circuit features, including inherent features of the AIG circuit and graph features of the AIG circuit, the inherent features of the AIG circuit being features directly embodied in the AIG file for each circuit, and the graph features being obtained by a graph convolution network, the inherent features and the graph features being jointly integrated into a circuit feature vector through at least one fully connected layer, and the feature being set as an input of a reinforcement learning agent; Step 2: Train a latent space of an optimization script, and set the latent space as an action space of the reinforcement learning agent; Step 3: Build a double-agent model, including a policy agent and a reward agent, set the policy agent to generate an optimization script for each iteration, and set the reward agent to generate a modified reward value for each iteration; Step 4: Iteration of the policy agent, select an optimization script to be executed according to the current circuit feature, and interact with the environment to obtain a next-time circuit feature and an environment reward of the current iteration; Step 5: Iteration of the reward agent, combine the current-time circuit feature and the selected action obtained in step 4 as a state of the reward agent, select a modified reward value according to the state, and feed back to the policy agent, integrate the modified reward and the environment reward to form a total reward; The total reward is: （1） where Area init is the area of the initial AIG circuit, Delay constraint is the delay constraint on the circuit, Area t and Delay t are the area and delay at the current time, and α and β are the weight factors of the corresponding terms, is the reward correction value at the current time. Select a next-time action according to the next-time state, and then obtain the next-time state defined in the reward agent and the environment reward of the current iteration; Step 6: Trajectory temporary storage, set a trajectory of the policy agent as a current circuit feature, a currently executed optimization script, a current total reward, and a next-time circuit feature, which are obtained from steps 4 and 5; Set a trajectory of the reward agent as a current circuit feature, a modified reward value of a current reward, a current environment reward, and a next-time circuit feature, which are obtained from steps 4 and 5; Step 7: Agent update, determine whether a set number of trajectory data are collected, if the set number of trajectory data are collected, update the network parameters of the agent, and clear the trajectory data of the policy agent, otherwise, do not update, and continue to collect trajectory data; Step 8: Determine whether a stop condition is reached, if the number of iterations reaches a set upper limit, save the current model, and output an optimization script sequence, otherwise, repeat steps 4-8; Step 9: For a circuit used for training, input the optimization script sequence output in step 8 into a logic synthesis tool to obtain an area and a delay of an optimized circuit, and for a circuit used for testing, load the model saved in step 8, generate an optimization sequence for the current test circuit, input into the logic synthesis tool, and obtain an area and a delay of an optimized circuit.

2. The reward-modification-based reinforcement learning-based logical synthesis sequence optimization space exploration method according to claim 1, characterized in that, In the step 1, the AIG circuit features include inherent features of the AIG circuit and graph features of the AIG circuit, the inherent features of the AIG circuit include: the number of AND gates, the number of connection edges, the length of the critical path, the number of nodes after mapping into a LUT netlist, and the number of logic levels after mapping, the inherent features are represented by a vector generated by a fully connected layer, the number of layers of the fully connected layer is at least two, and is adjusted between two and four. The graph feature of the AIG circuit is obtained by a graph convolution network, the graph convolution network comprises convolution layers and pooling layers, the number of convolution layers is at least three and is adjusted between three and five, local information of each node is integrated into global features through step-by-step aggregation and updating of the convolution layers, and the graph feature is formed; the integrated circuit feature vector is obtained by the intrinsic features and the graph features through at least one fully connected layer, and after standardization, the integrated circuit feature vector is used as an input of a reinforcement learning algorithm to describe a circuit state. 3.The reward-modification-based reinforcement learning-based logical synthesis sequence optimization space exploration method according to claim 1, wherein, In the step 2, a latent space of the training optimization script is implemented by a self-encoder, the self-encoder comprises an encoder and a decoder, the encoder and the decoder each comprise two to four fully connected layers, wherein the encoder generates a latent space for agent training, and the decoder decodes the original optimization script for environment interaction; In constructing the latent action space, the initial optimization script set is first encoded by the encoder to form an encoded latent action space, and the vectors in the latent action space are encoded representations of the initial optimization script, and then the vectors in the latent action space are restored to the original optimization script by the decoder.

4. The reward-modification-based reinforcement learning-based logical synthesis sequence optimization space exploration method according to claim 1, characterized in that, The strategy agent and the reward agent in the step 3 are two collaborative training agents constructed based on a reinforcement learning algorithm, wherein the reward agent is implemented by using a deep deterministic policy gradient algorithm, and the policy network and the value network of the reward agent each comprise at least two fully connected layers; the strategy agent is implemented by using a proximal policy optimization algorithm, and the policy network and the value network of the strategy agent each comprise at least two fully connected layers. 5.The reward-modification-based reinforcement learning-based logical synthesis sequence optimization space exploration method according to claim 1, wherein, In the step 4, a probability distribution of the latent action space is generated according to a current circuit state, a latent action is selected according to the probability distribution, the probability distribution of the latent action space is a probability of each latent action being selected, the latent action is decoded by using the decoder constructed in the step 2 to obtain an optimization script recognized by a logic synthesis tool, and then the script is input to the logic synthesis tool to obtain a new AIG circuit, and the AIG circuit feature vector is constructed again in the step 1 to obtain a circuit feature at a next moment and an environment reward of a current iteration.

6. The reward-modification-based reinforcement learning-based logical synthesis sequence optimization space exploration method according to claim 1, characterized in that, In the step 7, the network parameters of the agents are updated, and the weights and biases of each fully connected layer node of the policy network and the value network of the strategy agent and the policy network and the value network of the reward agent are updated.

7. The reward-modification-based reinforcement learning-based logical synthesis sequence optimization space exploration method according to claim 6, characterized in that, The strategy agent and the reward agent adopt different updating strategies, the strategy agent adopts a same-strategy mode, the parameters of the strategy agent are updated by using trajectories collected from the current AIG circuit, and the reward agent adopts a different-strategy mode, the parameters of the reward agent are updated by using historical trajectories collected from an experience replay buffer.

8. The reward-modification-based reinforcement learning-based logical synthesis sequence optimization space exploration method according to claim 1, characterized in that, In the step 7, the number of groups is set to 20 groups, if 20 groups of trajectory data are collected, the network parameters of the agents are updated, and the trajectory data of the strategy agent is cleared, otherwise, no updating is performed, and trajectory data is continuously collected.

9. A reward-modification-based reinforcement learning logic synthesis sequence optimization space exploration system, characterized by, The system executes the steps in the reward correction-based reinforcement learning-based logic synthesis sequence optimization space exploration method of claim 1, and the system comprises: The feature extraction module is configured to obtain a circuit feature vector, obtain a graph feature of each node and its adjacent node by using a graph convolution method, obtain an inherent feature of the circuit by reading an initial circuit, the inherent feature being a feature of each circuit having direct data embodiment in an AIG file, and combine the two features as final feature output; The action space generation module is configured to obtain a latent space for training by encoding an initial optimization script, decode the latent space to obtain an optimization script recognizable by a logic synthesis tool, and connect the reinforcement learning agent and the training environment; The model construction module is configured to save and update the network parameters of the two reinforcement learning agents. The strategy generation module is configured to generate a correction reward of each iteration by the network parameters of the reward agent, and generate an optimization script of each iteration by the network parameters of the strategy agent.

Images