Logical comprehensive sequence optimization space exploration method and system for reinforcement learning based on reward correction

Abstract

The invention relates to a logic comprehensive sequence optimization space exploration method and system for reinforcement learning based on reward correction, and belongs to the technical field of electronic design automation. A logic comprehensive process is embedded into a reinforcement learning algorithm based on reward correction, and the logic comprehensive process is modeled as a reinforcement learning environment; and the logic comprehensive sequence optimization space is automatically explored. A reinforcement learning algorithm is improved for a logic comprehensive scene, feature extraction and action coding are carried out, the circuit state expression precision is improved, and the search space is expanded. According to the method, by adding award correction, the exploration ability in the early stage of training is improved, a self-adaptive award function is obtained through intelligent agent training, performance reduction caused by limitation of priori knowledge on the circuit optimization direction is reduced, and a better logic comprehensive area result is obtained under the condition that delay constraints are met.

Description

Technical Field

[0001] The invention relates to a logic synthesis sequence optimization space exploration method and system based on reward-corrected reinforcement learning, belonging to the technical field of electronic design automation. Background Art

[0002] Electronic Design Automation (EDA) is a set of computer-aided tools and methods used to support the design and optimization of complex electronic systems. The goal of EDA technology is to accelerate the design, verification and optimization of integrated circuits (ICs), system-on-chips (SoCs) and programmable logic devices (such as FPGAs) through efficient automated processes. It covers all aspects from design input, functional verification, logic synthesis to wiring, physical implementation and performance testing, and is an indispensable technical support for modern semiconductor design.

[0003] As chip technology continues to evolve toward the nanoscale, the scale and complexity of circuit design are growing exponentially. Current design projects often involve hundreds of millions or even billions of transistors, which places higher demands on EDA tools. In this context, how to improve design efficiency, shorten development cycles, and optimize performance has become an important research direction in the field of electronic design.

[0004] In the EDA process, logic synthesis is one of the key links. The main task of logic synthesis is to convert the design description at a high level of abstraction (such as RTL code written in hardware description languages ​​such as Verilog or VHDL) into a gate-level netlist. In this process, not only the design function needs to be realized, but also several key performance indicators need to be optimized, such as: area (reducing the number of required logic gates and resource occupation), delay (reducing the propagation delay of the signal in the circuit), and power consumption (reducing power consumption to improve energy efficiency). Logic synthesis is a multi-objective optimization problem that requires finding the best balance between area, delay and power consumption. Traditional logic synthesis tools mostly use heuristic algorithms or rule-driven optimization processes to optimize the design by applying predefined circuit transformation scripts. However, with the continuous increase in circuit scale and design constraints, the difficulty of optimization is also gradually increasing.

[0005] In the traditional logic synthesis process, the optimization effect depends on the selected circuit optimization script and its execution order. Common optimization operations include logic merging, redundancy elimination, timing adjustment, gate-level optimization, etc. These operations need to be selected according to the specific circuit characteristics and weighed between objectives such as area, delay, and power consumption. However, as the design complexity increases, the combination and sequence of circuit optimization scripts grow exponentially, making the exploration of the optimization space extremely difficult. In addition, traditional heuristic algorithms can often only search within a limited local solution space, easily falling into local optimal solutions, and cannot guarantee the discovery of global optimal solutions. The quality of the optimization results depends largely on experienced engineers manually adjusting parameters and strategies, which is not only time-consuming but also difficult to adapt to the needs of large-scale design.

[0006] In order to meet the above challenges, reinforcement learning (RL) has been introduced into the field of logic synthesis as a new research direction in recent years. Reinforcement learning is a machine learning method that is suitable for sequential decision-making problems, especially in scenarios where the state and action space is huge and needs to be gradually optimized.

[0007] In logic synthesis, reinforcement learning can learn optimization strategies and gradually improve the quality of results by dynamically exploring the combination and order of different optimization scripts. By introducing reinforcement learning, the system no longer relies on manual experience to select optimization scripts, but automatically searches for the optimal solution in the design space through the model, thus solving the problem that traditional heuristic methods are prone to falling into local optimality. Summary of the invention

[0008] In view of the deficiencies of the prior art, the present invention provides a method and system for exploring the space of logic synthesis sequence optimization based on reward-corrected reinforcement learning. The present invention embeds the rules of logic synthesis circuit optimization into the reinforcement learning algorithm based on reward correction, models the logic synthesis process as a reinforcement learning environment, and realizes automatic exploration of the logic synthesis sequence optimization space. The present invention improves the reinforcement learning algorithm for the logic synthesis scenario, performs feature extraction and action encoding, improves the accuracy of circuit state representation, and expands the search space. In the process of traditional algorithm exploration and automatic sequence generation, too much human prior knowledge is added to limit the exploration process of circuit states. The present invention increases the exploration ability in the early stage of training by adding reward correction, obtains an adaptive reward function by intelligent agent training, reduces the performance degradation caused by the limitation of prior knowledge on the direction of circuit optimization, and obtains better logic synthesis area results under the condition of meeting delay constraints.

[0009] Terminology explanation:

[0010] 1. AIG circuit: It is a graph structure used to represent and optimize logic circuits. It is widely used in logic synthesis and verification in the field of electronic design automation. AIG represents Boolean logic functions in a concise and efficient way. It consists of AND gates and inverters. Its core feature is that it only uses two-input AND gates to build a logic network, which improves the simplicity of the structure.

[0011] 2. Agent: An agent is an autonomous system that can perceive, make decisions, and take actions in the environment to achieve a certain goal. In fields such as artificial intelligence (AI) and reinforcement learning (RL), the agent is a core concept, representing the subject that performs a task or interacts with the environment.

[0012] 3. Quality of Results: Quality of Results (QoR) is a comprehensive indicator used to measure the quality of final results in electronic design automation (EDA), machine learning and other engineering fields. In different application scenarios, the specific meaning and evaluation dimension of QoR are different. In this invention, it represents the trade-off between area and delay.

[0013] 4. Circuit performance parameters: also known as circuit design indicators or circuit performance values. For example, circuit area, delay, power consumption, etc.

[0014] 5. LUT Netlist: LUT Netlist is a form of circuit structure used in electronic design automation (EDA) and field programmable gate array (FPGA) design. LUT (Look-Up Table) is a basic unit for implementing logic functions. In FPGA, logic circuits are usually mapped to a set of LUTs and their interconnections. LUT Netlist constructs complex logic circuits by describing these LUTs and the connections between them.

[0015] 6. Graph Convolutional Networks: Graph Convolutional Networks (GCN) is a deep learning model for processing graph structured data. Its main feature is that it can effectively learn representations from information about nodes and their neighborhoods.

[0016] The technical solution of the present invention is as follows:

[0017] A logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning is implemented by using the logic synthesis process as the environment of the reward-corrected reinforcement learning algorithm, using the reinforcement learning iterative action generation method to select the optimization script and combine it into the final optimization script sequence. This method can reduce the influence of prior knowledge and find the optimal solution of the area that meets the delay constraint with higher efficiency, including the following steps:

[0018] Step 1: Obtain the AIG circuit feature and set it as the reinforcement learning agent input;

[0019] Step 2: Train the latent space of the optimization script and set the latent space as the action space of the reinforcement learning agent;

[0020] Step 3: Build a dual-agent model, including a strategy agent and a reward agent. Set the strategy agent to generate an optimization script for each iteration, and the reward agent to generate a revised reward value for each iteration.

[0021] Step 4: The strategy agent iterates, selects the currently executed optimization script according to the circuit characteristics at the current moment, and interacts with the environment to obtain the circuit characteristics at the next moment and the environmental reward of the current iteration;

[0022] Step 5: Reward agent iteration, combining the current circuit characteristics obtained in step 4 with the selected action as the state of the reward agent, selecting a revised reward value based on the state, feeding it back to the strategy agent, and integrating the revised reward with the environment reward to form a total reward;

[0023] Step 6: Temporary storage of trajectories. Set a trajectory of the strategy agent to the current circuit characteristics, the currently executed optimization script, the current total reward, and the circuit characteristics at the next moment, obtained from steps 4 and 5.

[0024] Set a trajectory of the reward agent to be the current circuit characteristics, the corrected reward value of the current reward, the current environment reward, and the circuit characteristics at the next moment, obtained by steps 4 and 5;

[0025] Step 7: Agent update, determine whether the set number of trajectory data sets are collected. If the set number of trajectory data sets are collected, update the agent's network parameters and clear the strategy agent's trajectory data. Otherwise, do not update and continue to collect trajectory data.

[0026] Step 8: Determine whether the stop condition is reached. If the number of iterations reaches the set upper limit, save the current model and output the optimization script sequence. Otherwise, repeat steps 4-8.

[0027] Step 9: For the circuit used for training, input the optimization script sequence output in step 8 into the logic synthesis tool to obtain the area and delay of the optimized circuit. For the circuit used for testing, load the model saved in step 8, generate the optimization sequence for the current test circuit, input it into the logic synthesis tool, and obtain the area and delay of the optimized circuit.

[0028] Preferably, in step 1, the AIG circuit features include the inherent features of the AIG circuit and the graph features of the AIG circuit. The inherent features of the AIG circuit include: the number of AND gates, the number of connecting edges, the critical path length, the number of nodes after mapping to the LUT netlist, and the number of logic levels after mapping. The inherent features generate a vector representation of the inherent features through a fully connected layer, and the number of fully connected layers is at least two. Further preferably, the inherent features are set according to the complexity of the inherent features and adjusted between two and four layers.

[0029] Furthermore, the graph features of the AIG circuit are obtained by a graph convolutional network, which includes convolutional layers and pooling layers. The number of convolutional layers is at least three. Further, the number of layers is adjusted between three and five according to the complexity of the AIG circuit. The local information of each node is integrated into a global feature through gradual aggregation and updating through the convolutional layers to form a graph feature. The inherent features and the graph features are spliced ​​into a complete circuit feature vector, and the integrated circuit feature vector is obtained through at least one fully connected layer. After standardization, it is used as the input of the reinforcement learning algorithm to describe the circuit state.

[0030] Preferably, in step 2, the latent space of the training optimization script is implemented by an autoencoder, which includes an encoder and a decoder, and both the encoder and the decoder include two to four fully connected layers. Based on the complexity and running speed of the AIG circuit, the total number of fully connected layers is adjusted between four and eight layers, wherein the encoder generates a latent space for intelligent agent training, and the decoder decodes the original optimization script for environmental interaction.

[0031] Furthermore, when constructing the latent action space, the initial set of optimization scripts is first encoded by an encoder to form an encoded latent action space. The vectors in the latent action space are the encoded representations of the initial optimization scripts. Subsequently, the vectors in the latent action space are restored to the original optimization scripts through a decoder to ensure lossless reconstruction of the information.

[0032] Preferably, the policy agent and reward agent in step 3 are two collaboratively trained agents constructed based on a reinforcement learning algorithm, wherein the reward agent is implemented using a deep deterministic policy gradient (DDPG) algorithm, and the policy network and value network of the reward agent each contain at least two fully connected layers; the policy agent is implemented using a proximal policy optimization (PPO) algorithm, and the policy network and value network of the policy agent each contain at least two fully connected layers.

[0033] Preferably, in step 4, a probability distribution of a potential action space is generated according to the current circuit state, and a potential action is selected according to the probability distribution. The probability distribution of the potential action space is the probability of each potential action being selected. The decoder constructed in step 2 is used to decode the potential action to obtain an optimization script that can be recognized by a logic synthesis tool. The script is then input into the logic synthesis tool to obtain a new AIG circuit, and the process returns to step 1 to construct an AIG circuit feature vector to obtain the circuit features at the next moment and the environmental reward of the current iteration.

[0034] Preferably, the total reward in step 5 is:

[0035]

[0036] Where Area init is the area of ​​the initial AIG circuit, Delay constraint To constrain the delay of the circuit, Area t and Delay t are the area and delay at the current moment, α and β are the weight factors of the corresponding items, The reward correction value at the current moment;

[0037] The next moment action is selected according to the next moment state, and then the next moment state defined in the reward agent and the environmental reward of the current iteration are obtained.

[0038] Furthermore, the network parameters of the updated agent in step 7 specifically refer to the policy network and value network of the policy agent, and the weights and biases of each fully connected layer node of the policy network and value network of the reward agent.

[0039] Furthermore, the policy agent and the reward agent adopt different update strategies. The policy agent adopts a same-strategy approach and uses the trajectory collected by the current AIG circuit to update its parameters, while the reward agent adopts a different-strategy approach and uses the experience replay buffer to collect the historical trajectories in the buffer to update its parameters.

[0040] Furthermore, in step 7, the number of groups is set to 20. If 20 groups of trajectory data are collected, the network parameters of the agent are updated and the trajectory data of the strategy agent is cleared. Otherwise, no update is performed and trajectory data collection continues. The collected data is the trajectory data defined in step 6.

[0041] A logic synthesis sequence optimization space exploration system based on reward-corrected reinforcement learning, including:

[0042] The feature extraction module is configured to obtain a circuit feature vector, obtain the graph features of each node and its adjacent nodes by using a graph convolution method, obtain the inherent features of the circuit by reading the initial circuit, and combine the two parts of the features as the final feature output;

[0043] An action space generation module is configured to obtain a latent space for training by encoding an initial optimization script, decode the latent space to obtain an optimization script that can be recognized by a logic synthesis tool, and connect a reinforcement learning agent and a training environment;

[0044] The model building module is configured as two reinforcement learning agents to save and update the agent network parameters;

[0045] The strategy generation module is configured to generate a modified reward for each iteration through the network parameters of the reward agent, and to generate an optimization script for each iteration through the network parameters of the strategy agent.

[0046] The beneficial effects of the present invention are:

[0047] 1. The present invention proposes a logic synthesis sequence optimization space exploration method and system based on reward-corrected reinforcement learning, which embeds the rules of logic synthesis circuit optimization into the reinforcement learning algorithm, and uses the improved reinforcement learning algorithm to adaptively adjust the reward function to increase the exploration intensity, optimize the exploration process, and take into account both exploration and utilization in the optimization process.

[0048] 2. The present invention proposes a logic synthesis sequence optimization space exploration method and system based on reward-corrected reinforcement learning, which converts the AIG circuit into an adjacency matrix, combines logic synthesis with a graph convolutional network, and improves the accuracy of state representation.

[0049] 3. The present invention proposes a logic synthesis sequence optimization space exploration method and system based on reward-corrected reinforcement learning, which combines the optimization script in the logic synthesis process with the autoencoder, simplifies the action space, and solves the problem of non-convergence of intelligent agent training caused by complex action space.

[0050] 4. The present invention proposes a logic synthesis sequence optimization space exploration method and system based on reward-corrected reinforcement learning, and improves the traditional reinforcement learning algorithm in combination with logic synthesis field knowledge, so that the improved reinforcement learning method is more suitable for the logic synthesis process, and realizes adaptive optimization adjustment of different circuits.

[0051] 5. The present invention proposes a logic synthesis sequence optimization space exploration method and system based on reward-corrected reinforcement learning, which combines more optimization scripts and achieves more efficient and faster optimization compared to human experts. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of a logic synthesis sequence optimization space exploration method and system based on reward-corrected reinforcement learning of the present invention;

[0053] Figure 2 This is a schematic diagram of the network structure of the feature extraction model of Example 1;

[0054] Figure 3 This is a schematic diagram of the autoencoder network structure of Example 1;

[0055] Figure 4 This is a schematic diagram of the intelligent agent structure of Example 1;

[0056] Figure 5 This is a flowchart of the key step model training in Example 1;

[0057] Figure 6 The scatter plot is a distribution diagram of the exploration results of Example 1. DETAILED DESCRIPTION

[0058] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0059] Embodiment 1:

[0060] Embodiment 1 of the present invention is to optimize the design of a case in the logic synthesis process using an improved reinforcement learning algorithm. Table 1 lists the initial circuit indicators and their values.

[0061] Table 1 Initial circuit indicators and their values

[0062]

[0063] This embodiment accelerates the exploration of the design space by temporarily storing multiple intermediate processes, evaluating the results of each synthesis process and saving the optimal result, that is, an exploration that satisfies the delay constraint and has the smallest area, and then further exploration is performed based on the synthesis result of the previous moment until a predetermined number of training rounds is reached, thereby realizing automatic exploration of the optimization space and area optimization that satisfies the demonstration constraint.

[0064] The AIG circuit input by the system of this embodiment uses the open source AIG circuit file of EPFL, the synthesis tool uses BerkeleyABC, and the process library uses the ASAP7nm standard library. However, the input AIG circuit file, synthesis tool and process library are not limited to the above examples.

[0065] Figure 1 The figure is a flow chart of a logic synthesis sequence optimization space exploration method and system based on reward-corrected reinforcement learning of the present invention. Figure 1 As shown, at step 101, an AIG circuit is selected as an input file, and the selected AIG circuit is input into step 102 to obtain circuit features, and the circuit features are input into the reinforcement learning agent. In the AIG circuit, the circuit is represented only by an AND gate and a NOT gate, and the connection relationship between the circuits is represented by solid lines and dotted lines, wherein the solid line represents the direct connection between two AND gates, and the dotted line represents the insertion of a NOT gate between two AND gates. In different embodiments, the size of the AIG circuit may be different, and the difference is reflected in the number of AND gates, the number of NOT gates, and the number of connecting lines between gate units in the AIG circuit. The graph convolutional network algorithm is used to obtain the circuit features at step 102, and the circuit conversion gate unit and connecting line information are converted into an adjacency matrix, and the matrix is ​​input into the convolution layer for data dimensionality reduction to obtain a simplified feature vector that can represent the graph features, which is used to represent the circuit state and provide data support for reinforcement learning training.

[0066] Next, at step 103, the strategy agent and the reward agent interact with the logic synthesis environment to obtain circuit state information, executed actions, environmental rewards, and modified rewards. In an implementation scenario, the optimization objectives of the aforementioned strategy agent and reward agent are different. The strategy agent interacts with the logic synthesis environment, and its goal is to select a better optimization script based on the current state, so it is necessary to establish a reinforcement learning model for discrete action space; while the reward agent does not directly interact with the logic synthesis environment, and its goal is to generate a modified reward value based on the current state and the optimization script, which can be a discrete or continuous action space. In this embodiment, a reinforcement learning model for continuous action space is established based on deep deterministic policy gradients. The two models include multiple layers of fully connected layers, and the output dimensions of the final layers of the fully connected layers of each network are the dimensions of the desired state and the dimensions of the action, respectively.

[0067] In this embodiment, the state information of the circuit is obtained by combining inherent features and graph features, the executed action is represented as the action estimate of the policy network of the policy agent, the environmental reward is calculated from the current state information, and the corrected reward is represented as the reward estimate of the policy network of the reward agent, ultimately forming a trajectory.

[0068] After obtaining the above trajectory, at step 104, the model parameters are trained based on the obtained trajectory. In this embodiment, the trajectory is set to a round length, and this parameter is manually set according to the circuit characteristics and can be appropriately increased or decreased. If the circuit scale is small, the probability of the circuit state set will be relatively small, so the reachable state will be less, and the model can be trained with fewer trajectories; if the circuit scale is large, the number of states that may appear during the training process will also increase, so it is necessary to appropriately increase the number of trajectories to ensure that the model is fully trained.

[0069] The specific implementation steps are as follows:

[0070] Step 1: Construct AIG circuit features. In order to more accurately represent the state of a circuit during the optimization process, the required circuit features include: the inherent features of the AIG circuit, such as the number of AND gates, the number of connecting edges, the length of the critical path, the number of nodes after mapping to the LUT netlist, and the number of logical levels after mapping. The inherent features are generated through the fully connected layer to form a vector representation of the inherent features, such as Figure 2 As shown in , the number of fully connected layers is two. The graph features of the AIG circuit are obtained by the graph convolutional network, as shown in Figure 2 As shown in the figure, the graph convolutional network includes three convolutional layers and one pooling layer. The specific number of layers increases according to the complexity of the circuit. The node degree of the circuit and the node degree features of its adjacent nodes are aggregated through the convolutional layer to form an intermediate layer network that can represent local features. The final graph feature vector is convolved again and passes through a fully connected layer together with the inherent feature vector to form a circuit feature vector, which represents the current circuit state. This state is the state input of reinforcement learning.

[0071] Figure 2 This is a schematic diagram of the network structure of the feature extraction model, such as Figure 2 As shown in , the feature extraction model includes a graph convolution model 201, an inherent feature model 202 and a feature combination model 203. The graph convolution model 201, the inherent feature model 202 and the feature combination model 203 are only set for this embodiment. Corresponding to different embodiments or different scenarios, the number of network layers and dimensions of the model can be dynamically adjusted. In this embodiment, the graph convolution model 201 is connected in series through three layers of convolution layers, and then passes through a pooling layer to obtain a simplified graph feature vector. The input of the inherent feature model 202 is the inherent features of the circuit extracted by the logic synthesis tool, such as the number of AND gates of the circuit, etc. The data is dimensioned through two layers of fully connected layers to accurately represent the inherent state. Then, the feature combination model 203 cascades the graph features and inherent features obtained by the graph convolution model 201 and the inherent feature model 202, and passes through a fully connected layer together to reduce the dimension of the data again, so as to accurately represent the circuit state with a simplified feature vector.

[0072] Step 2: Training the potential action space. There are a large number of optimization scripts and corresponding parameters in the logic synthesis process, forming a huge action set. Table 2 lists some script examples of Berkeley ABC.

[0073] Table 2 Some script examples

[0074]

[0075] The latent space of the training optimization script is implemented by an autoencoder, which includes an encoder and a decoder. The encoder and decoder both include two fully connected layers. Based on the complexity and running speed of the AIG circuit, the total number of fully connected layers is adjusted between four and eight layers. The encoder generates the latent space for intelligent agent training, and the decoder decodes the original optimization script for environmental interaction.

[0076] When constructing the latent action space, the initial set of optimization scripts (such as rewrite-lz, resub-lz, etc.) is first encoded by the encoder to form an encoded latent action space. The vectors in the latent action space are the encoded representations of the initial optimization scripts. Subsequently, the vectors in the latent action space are restored to the original optimization scripts through the decoder to ensure lossless reconstruction of the information. Through this process, the autoencoder captures the structural features in the optimization script set, providing a more compact and easy-to-explore latent action space for the reinforcement learning model.

[0077] Figure 3 is a schematic diagram of the autoencoder network structure, as shown in Figure 3 As shown in , the autoencoder 300 includes an encoder 302 and a decoder 304. The encoder 300 is only set for this embodiment. Corresponding to different embodiments or different scenarios, the number of network layers and dimensions of the encoder 302 and the decoder 304 can be dynamically adjusted. In this embodiment, the encoder 302 is serially connected by two encoding layers (fully connected layers), and the decoder 304 is serially connected by two decoding layers (fully connected layers). Encode the parameterized optimization script 301, map it to the potential action space 303 through the encoder 302, as the action space of the agent, generate the action in the potential action space 303 during the training of the agent, and then decode it through the decoder 304 to obtain the parameterized optimization script 304. The result is a lossless decoding of the parameterized optimization script 301, which is used to map to the optimization script that the logic synthesis tool can recognize after the agent selects the potential action, and feed it back to the logic synthesis tool for operation to obtain the state and reward information for updating the parameters. By constructing the autoencoder 300, the exploration scope of the logic synthesis optimization script is expanded, while the training burden of reinforcement learning in complex action spaces is reduced. On the basis of improving the exploration capability, the convergence speed of training is taken into account.

[0078] Step 3: Construct the policy agent and reward agent. The policy agent is based on the proximal policy optimization (PPO) algorithm, including a policy network and a value network, each with two fully connected layers. The policy network fully connected layer is also activated by the Softmax function to generate the distribution of actions. The policy network is set to select actions in the latent space, and the value network is used to evaluate the state value of the current policy. The reward agent is based on the deep deterministic policy gradient (DDPG) algorithm, including a policy network and a value network, each with two fully connected layers. The policy network fully connected layer is activated by the Tanh function to generate a modified reward, which is fed back to the policy agent, and the value network evaluates the effect of the modified reward.

[0079] Figure 4 is a schematic diagram of the intelligent agent structure, such as Figure 4 As shown, the agent 400 includes a strategy agent 402 and a reward agent 403, wherein the strategy agent 402 includes a strategy network 404 and a value network 405, and the reward agent 403 includes a strategy network 406 and a value network 407. The agent 400 is only set for this embodiment, and the agent composition and the number of network layers can be dynamically adjusted for different embodiments or different scenarios. In this embodiment, the strategy network 404 is composed of two layers of fully connected layers, and then the distribution of actions is generated through the Softmax activation function, and the optimal action is selected according to the probability distribution. The action is the action in the potential action space 303, and the parameterized optimization script 305 is obtained by decoding the decoder 304. The value network 405 is composed of two layers of fully connected layers, and a numerical value is obtained after the data is upgraded and reduced in dimension, which represents the state value of the current state. The strategy network 406 is also composed of two layers of fully connected layers, and a modified reward is generated through the Tanh activation function, which is used to adaptively adjust the reward function according to different embodiments. The value network 407 is composed of two fully connected layers, which evaluates the effect of the modified reward generated by the reward agent 403 and helps the network parameters of the agent 403 to be updated in the direction of reducing the prediction error.

[0080] Step 4: Iteration of the policy agent. Generate the probability distribution of the potential action space according to the current circuit state, select the potential action according to the probability distribution, and the probability distribution of the potential action space is the probability of each potential action being selected. Further, use the decoder constructed in step 2 to decode the potential action to obtain an optimization script that can be recognized by the logic synthesis tool, and then input the script into the logic synthesis tool to obtain a new AIG circuit, return to step 1 to construct the AIG circuit feature vector, and obtain the circuit features at the next moment and the environmental reward of the current iteration.

[0081] Step 5: Reward agent iteration. The current circuit characteristics obtained in step 4 and the selected action are combined as the state of the reward agent. According to the state, the modified reward value is selected and fed back to the strategy agent to form the total reward. The total reward is:

[0082]

[0083] Where Area init is the area of ​​the initial AIG circuit, Delay constraint To constrain the delay of the circuit, Area t and Delay t are the area and delay at the current moment, α and β are the weight factors of the corresponding items, The reward correction value at the current moment.

[0084] The next moment action is selected according to the next moment state, and then the next moment state defined in the reward agent and the environmental reward of the current iteration are obtained.

[0085] Step 6: Temporary storage of trajectories. The policy agent is constructed based on the proximal policy optimization algorithm of the same strategy. Each temporarily stored trajectory is obtained from step 4, including the current circuit characteristics, the currently executed optimization script, the current total reward, and the circuit characteristics of the next moment. The same strategy method only uses the trajectory of the current round for parameter update. After the trajectory of the current round is used up, the trajectory needs to be cleared; the reward agent is constructed based on the deep deterministic policy gradient method of the different strategy. Each temporarily stored trajectory is obtained from step 5, including the current circuit characteristics, the correction term of the current reward, the current environment reward, and the circuit characteristics of the next moment. The different strategy method has an experience replay buffer to save the trajectory of each iteration without overwriting the trajectory.

[0086] Step 7: Agent update. Determine whether 20 sets of trajectory data have been collected. If 20 sets of trajectory data have been collected, update the agent's network parameters and clear the strategy agent's trajectory. Otherwise, do not update and continue to collect trajectory data. The collected data is the trajectory data defined in step 6.

[0087] Update the network parameters of the agent, specifically the policy network and value network of the policy agent, and the weights and biases of each fully connected layer node of the policy network and value network of the reward agent.

[0088] The policy agent and the reward agent adopt different update strategies. The policy agent adopts the same strategy approach and uses a set of trajectories collected by the current AIG circuit to update its parameters. The reward agent adopts the different strategy approach and uses the experience replay buffer to collect the historical trajectories in the buffer to update its parameters.

[0089] Step 8: Determine whether the stop condition is reached. If the number of iterations reaches the set upper limit, save the current model and output the logic synthesis optimization script sequence, otherwise, repeat steps 4-8.

[0090] Step 9: For the circuit used for training, input the optimization script sequence output in step 8 into the logic synthesis tool to obtain the area and delay of the optimized circuit. For the circuit used for testing, load the model saved in step 8, generate the optimization sequence for the current test circuit, input it into the logic synthesis tool, and obtain the area and delay of the optimized circuit.

[0091] Figure 5 The model training flow chart for key steps is as follows: Figure 5 As shown in , the feature extraction model 200 is used to obtain the circuit state, and the autoencoder 300 is used to interact with the logic synthesis tool to obtain the next moment state and environmental rewards, and the correction reward. In this embodiment, the strategy agent, the reward agent and the trajectory buffer are initialized, and then the circuit state is obtained through the feature extraction model, and then the action is selected according to the circuit state. The action at this time is the action in the latent space. Therefore, the interaction with the logic synthesis tool to obtain the next moment state and environmental rewards, and the correction reward needs to be combined with the autoencoder 300, and the optimization script that can be recognized by the logic synthesis tool is decoded, and then the current trajectory is saved to the buffer of each agent, and the update condition is judged. If the update condition is met, the network parameters of the strategy agent and the reward agent are updated, and then the stop condition is judged. If the update condition is not met, the stop condition is directly judged. If the stop condition is met, the logic synthesis optimization sequence is output and the model is saved and then exited, otherwise it returns to obtain the circuit state and continue to iterate.

[0092] Figure 6 This is a scatter plot of the exploration results of this embodiment. Figure 6 As shown in , the gray data points represent the circuit states at various moments obtained during the exploration process, the blue data points represent the Pareto frontier during the exploration process of this embodiment, the red solid points and multiplication signs represent the optimal performance of the circuit obtained by other methods, the red plus sign represents the performance of the initial circuit, the red dotted line represents the delay constraint of this embodiment, and the black triangle is the circuit performance result obtained by the exploration of this embodiment that meets the delay constraint and has the optimal area. Figure 6 It can be seen that the Pareto curve obtained by using the improved reinforcement learning algorithm to explore the space of logic synthesis sequence optimization satisfies the trade-off relationship between area and delay. When optimizing one performance, it often leads to a decrease in the other performance. Compared with other methods, it can be seen that the method used in this embodiment has a better area under the same delay constraint, which proves the effectiveness of this method in exploring the space of logic synthesis sequence optimization.

[0093] Example 2

[0094] A logic synthesis sequence optimization space exploration system based on reward-corrected reinforcement learning, including:

[0095] The feature extraction module is configured to obtain a circuit feature vector, obtain the graph features of each node and its adjacent nodes by using a graph convolution method, obtain the inherent features of the circuit by reading the initial circuit, and combine the two parts of the features as the final feature output;

[0096] An action space generation module is configured to obtain a latent space for training by encoding an initial optimization script, decode the latent space to obtain an optimization script that can be recognized by a logic synthesis tool, and connect a reinforcement learning agent and a training environment;

[0097] The model building module is configured as two reinforcement learning agents to save and update the agent network parameters;

[0098] The strategy generation module is configured to generate a modified reward for each iteration through the network parameters of the reward agent, and to generate an optimization script for each iteration through the network parameters of the strategy agent.

[0099] The above-described embodiments are only preferred implementation modes of the present application. The description of Embodiment 1 is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. For Embodiment 2, since it is basically similar to the method embodiment, the description is relatively simple, and the implementation of the relevant modules can refer to the partial description of Method Embodiment 1. It should be pointed out that for ordinary technicians in the technical field, several improvements and replacements without departing from the principle of the present invention should also be regarded as the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the scope of protection of the claims.

Claims

1. A logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning, characterized in that: The steps include: Step 1: Obtain the AIG circuit feature and set it as the reinforcement learning agent input; Step 2: Train the latent space of the optimization script and set the latent space as the action space of the reinforcement learning agent; Step 3: Build a dual-agent model, including a strategy agent and a reward agent. Set the strategy agent to generate an optimization script for each iteration, and the reward agent to generate a revised reward value for each iteration. Step 4: The strategy agent iterates, selects the currently executed optimization script according to the circuit characteristics at the current moment, and interacts with the environment to obtain the circuit characteristics at the next moment and the environmental reward of the current iteration; Step 5: Reward agent iteration, combining the current circuit characteristics obtained in step 4 with the selected action as the state of the reward agent, selecting a revised reward value based on the state, feeding it back to the strategy agent, and integrating the revised reward with the environment reward to form a total reward; Step 6: Temporary storage of trajectories. Set a trajectory of the strategy agent to the current circuit characteristics, the currently executed optimization script, the current total reward, and the circuit characteristics at the next moment, obtained from steps 4 and 5. Set a trajectory of the reward agent to be the current circuit characteristics, the corrected reward value of the current reward, the current environment reward, and the circuit characteristics at the next moment, obtained by steps 4 and 5; Step 7: Agent update, determine whether the set number of trajectory data sets are collected. If the set number of trajectory data sets are collected, update the agent's network parameters and clear the strategy agent's trajectory data. Otherwise, do not update and continue to collect trajectory data. Step 8: Determine whether the stop condition is reached. If the number of iterations reaches the set upper limit, save the current model and output the optimization script sequence. Otherwise, repeat steps 4-8. Step 9: For the circuit used for training, input the optimization script sequence output in step 8 into the logic synthesis tool to obtain the area and delay of the optimized circuit. For the circuit used for testing, load the model saved in step 8, generate the optimization sequence for the current test circuit, input it into the logic synthesis tool, and obtain the area and delay of the optimized circuit.

2. The logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning according to claim 1 is characterized in that: In step 1, the AIG circuit features include the inherent features of the AIG circuit and the graph features of the AIG circuit. The inherent features of the AIG circuit include: the number of AND gates, the number of connection edges, the critical path length, the number of nodes after being mapped to the LUT netlist, and the number of logical levels after mapping. The inherent features are generated into vector representations of the inherent features through a fully connected layer. The number of fully connected layers is at least two, and more preferably, the number is adjusted between two and four layers. The graph features of the AIG circuit are obtained by a graph convolutional network, which includes convolutional layers and pooling layers. The number of convolutional layers is at least three. Further, the convolutional layers are adjusted between three and five layers. The local information of each node is integrated into a global feature through gradual aggregation and updating through the convolutional layers to form graph features. The inherent features and the graph features are jointly obtained through at least one fully connected layer to obtain an integrated circuit feature vector, which is used as the input of the reinforcement learning algorithm after standardization to describe the circuit state.

3. The logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning according to claim 1 is characterized in that: In step 2, the latent space of the training optimization script is implemented by an autoencoder, which includes an encoder and a decoder, and the encoder and the decoder both include two to four fully connected layers, wherein the encoder generates a latent space for agent training, and the decoder decodes the original optimization script for environment interaction; When constructing the latent action space, the initial set of optimization scripts is first encoded by the encoder to form an encoded latent action space. The vectors in the latent action space are the encoded representations of the initial optimization scripts. Subsequently, the vectors in the latent action space are restored to the original optimization scripts through the decoder.

4. The logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning according to claim 1 is characterized in that: The policy agent and reward agent in step 3 are two collaboratively trained agents built based on a reinforcement learning algorithm, wherein the reward agent is implemented using a deep deterministic policy gradient algorithm, and the policy network and value network of the reward agent each contain at least two fully connected layers; the policy agent is implemented using a proximal policy optimization algorithm, and the policy network and value network of the policy agent each contain at least two fully connected layers.

5. The logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning according to claim 1 is characterized in that: In step 4, the probability distribution of the potential action space is generated according to the current circuit state, and the potential action is selected according to the probability distribution. The probability distribution of the potential action space is the probability of each potential action being selected. The decoder constructed in step 2 is used to decode the potential action to obtain an optimization script recognized by the logic synthesis tool, and then the script is input into the logic synthesis tool to obtain a new AIG circuit. Return to step 1 to construct the AIG circuit feature vector to obtain the circuit features at the next moment and the environmental reward of the current iteration.

6. The logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning according to claim 1 is characterized in that: The total reward in step 5 is: Where Area init is the area of ​​the initial AIG circuit, Delay constraint To constrain the delay of the circuit, Area t and Delay t are the area and delay at the current moment, α and β are the weight factors of the corresponding items, The reward correction value at the current moment; The next moment action is selected according to the next moment state, and then the next moment state defined in the reward agent and the environmental reward of the current iteration are obtained.

7. The logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning according to claim 1 is characterized in that: The network parameters of the updated agent in step 7 specifically refer to the policy network and value network of the policy agent, and the weights and biases of each fully connected layer node of the policy network and value network of the reward agent.

8. The logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning according to claim 7 is characterized in that: The strategy agent and the reward agent adopt different update strategies. The strategy agent adopts the same strategy method and uses the trajectory collected by the current AIG circuit to update its parameters. The reward agent adopts the different strategy method and uses the experience replay buffer to collect the historical trajectory in the buffer to update its parameters.

9. The logic synthesis sequence optimization space exploration method based on reward-corrected reinforcement learning according to claim 1 is characterized in that: In step 7, the number of groups is set to 20. If 20 sets of trajectory data are collected, the network parameters of the agent are updated and the trajectory data of the strategy agent is cleared. Otherwise, no update is performed and trajectory data collection continues.

10. A logic synthesis sequence optimization space exploration system based on reward-corrected reinforcement learning, characterized in that: include: The feature extraction module is configured to obtain a circuit feature vector, obtain the graph features of each node and its adjacent nodes by using a graph convolution method, obtain the inherent features of the circuit by reading the initial circuit, and combine the two parts of the features as the final feature output; An action space generation module is configured to obtain a latent space for training by encoding an initial optimization script, decode the latent space to obtain an optimization script that can be recognized by a logic synthesis tool, and connect a reinforcement learning agent and a training environment; The model building module is configured as two reinforcement learning agents to save and update the agent network parameters; The strategy generation module is configured to generate a modified reward for each iteration through the network parameters of the reward agent, and to generate an optimization script for each iteration through the network parameters of the strategy agent.

Images