Motion prediction method, motion training method, program product, electronic device, and chip system

Abstract

The invention provides an action prediction and training method, a program product, electronic equipment and a chip system, and relates to the technical field of chip design, and the method can expand the exploration space of an operator sequence and improve the optimization effect. The method is executed based on reinforcement learning, and the reinforcement learning comprises a strategy network. The execution object of the action is a logic circuit or other objects; the method comprises the following steps: determining a tth state feature vector; wherein the tth state feature vector is used for representing the environment state after the (t-1) th action is executed; t is an integer variable and is greater than or equal to 1; inputting the t-th state feature vector into a strategy network, and at least outputting a first parameter; based on the first parameter, determining a tth operator to obtain a tth action; wherein the tth action is used for executing the tth operator on the execution object; the tth operator and the (t-1) th operator corresponding to the (t-1) th action are operators of the same type or different types.

Description

Technical Field

[0001] The present application relates to the field of chip design technology, and in particular to an action prediction, training method, program product, electronic device and chip system. Background Art

[0002] Chip design can generally be divided into front-end design and back-end design. Logic synthesis is an important step in front-end design. Generally, synthesis tools such as Electronic Design Automation (EDA) are used to convert the logic circuit described in the hardware description language into a netlist based on the target technology library containing basic circuit units provided by the chip manufacturer. Generally, the register transfer level (RTL) code is converted into a gate-level netlist.

[0003] In a logic synthesis scheme based on heuristics and iterations, for different circuits, an operator sequence designed in advance based on expert experience is dynamically generated, and each operator in the operator sequence is executed in sequence on the logic circuit to be converted. Its technical goal is to optimize the performance, power consumption, and area (PPA) of the final netlist.

[0004] However, the operator sequence designed in advance in this scheme is spliced ​​in groups, and multiple operators in the same group are of the same type, which limits the exploration space and affects the optimization effect. Summary of the invention

[0005] The present application provides an action prediction, training method, program product, electronic device and chip system, which can realize mixed arrangement of operators in a sequence and further improve the optimization effect of logic synthesis.

[0006] In the first aspect, the embodiment of the present application provides an action prediction method, which is based on reinforcement learning execution, and reinforcement learning includes a strategy network; the execution object of the action is a logic circuit or other object; the method includes: determining the tth state feature vector; wherein the tth state feature vector is used to characterize the state of the environment after executing the t-1th action; t is an integer variable and t≥1; inputting the tth state feature vector into the strategy network, and outputting at least a first parameter; based on the first parameter, determining the tth operator, and obtaining the tth action; wherein the tth action is used to execute the tth operator on the execution object; the tth operator and the t-1th operator corresponding to the t-1th action are operators of the same type or operators of different types. The t-1th operator and the tth operator are equivalent to any two adjacent operators in the action sequence, so that any two adjacent operators in the action sequence can be operators of the same type or operators of different types. Compared with the operator arrangement method in the related art, more operator arrangement and combination methods can be obtained, which expands the exploration space and is conducive to improving the optimization performance, such as improving the PPA of chip design in the logic synthesis stage.

[0007] In a feasible implementation, the t-th state feature vector is input into the policy network, and at least the first parameter is output, including: inputting the t-th state feature vector into the policy network, and outputting the first parameter and the second parameter; after at least the first parameter is output, the method further includes: determining the t-th group of parameters corresponding to the t-th operator based on the second parameter; the t-th action is used to execute the t-th operator on the execution object according to the t-th group of parameters. While optimizing the operator sequence, the operator parameters are also optimized, and a better combination of the operator sequence and the operator parameters is explored, so as to give play to the regulatory role of the operator parameters in the optimization process, thereby improving PPA.

[0008] In a feasible implementation, after obtaining the t-th action, the method further includes: obtaining an action sequence, the action sequence including information of operators corresponding to at least one action; the action sequence is a variable length sequence. The fixed length of the action sequence limits the optimization effect, while the variable length can break this limitation and is conducive to exploring a sequence with better optimization effect.

[0009] In a feasible implementation, the method is applied to the logic synthesis stage in chip design; the action sequence includes information of at least two types of operators among logic optimization operators, mapping operators or post-mapping operators.

[0010] In a feasible implementation, after determining the t-th operator based on the first parameter, the method further includes: when the t-1th operator corresponding to the t-1th action is a logic optimization operator and the t-th operator is a post-mapping operator, inserting a mapping operator after the logic optimization operator and before the post-mapping operator. The insertion of the mapping operator enables the mixed sorted operator sequence to be compatible with processes such as logic optimization, mapping, and post-mapping.

[0011] In a feasible implementation, after determining the t-th operator based on the first parameter, the method further includes: when the t-1th operator corresponding to the t-1th action is a mapping operator or a post-mapping operator, and the t-th operator is a logic optimization operator or a mapping operator, inserting a fallback operator after the mapping operator or the post-mapping operator and before the logic optimization operator or the mapping operator. The insertion of the fallback operator enables the mixed sorted operator sequence to be compatible with processes such as logic optimization, mapping, and post-mapping.

[0012] In a feasible implementation, determining the tth state feature vector includes: performing a mixed representation of the environment state based at least on the state of the execution object after executing the t-1th action and the graph structure information corresponding to the execution object to obtain the tth state feature vector.

[0013] In a feasible implementation, determining the tth state feature vector includes: obtaining the tth first feature vector based on the state of the execution object after executing the t-1th action; obtaining the second feature vector based on the graph structure information corresponding to the execution object; obtaining the tth third feature vector based on the historical action sequence; wherein the historical action sequence includes operator information and parameter information corresponding to at least one action executed before executing the tth action; obtaining the tth state feature vector based on the tth first feature vector, the second feature vector and the tth third feature vector. The mixed representation of the environmental state can fully extract the characteristics of the current environment and quantify the impact of the expression action on the environment.

[0014] In a feasible implementation, based on the state of the execution object after executing the t-1th action, the tth first feature vector is obtained, including: after executing the t-1th action, determining the state vector of the execution object in the current state; inputting the state vector into a preset multi-layer perceptron, and outputting the first feature vector through the multi-layer perceptron; based on the graph structure information corresponding to the execution object, the second feature vector is obtained, including: determining the graph structure information corresponding to the execution object, inputting the graph structure information into a preset graph neural network, and outputting the second feature vector through the graph neural network; based on the historical action sequence, the tth third feature vector is obtained, including: inputting the historical action sequence into a preset first encoder to obtain the tth third feature vector; wherein the first encoder is an encoder based on the attention mechanism.

[0015] In a feasible implementation, the execution object is a logic circuit; determining the state vector of the execution object in the current state includes: determining the state vector of the execution object in the current state based on at least one of a first group of information, a second group of information, and a third group of information; wherein the first group of information includes at least one of the number of nodes, the logic depth, and the number of edges of the logic circuit after executing the t-1th action; the second group of information includes at least one of the number of LUTs and the logic depth of the LUT in a lookup table LUT circuit; wherein the LUT circuit is a circuit containing the LUT obtained by executing a TM operator on the logic circuit; and the third group of information is the length of the historical action sequence.

[0016] In a feasible implementation, the policy network includes a first actor and a second actor; the first actor and the second actor both include trainable parameters; the t-th state feature vector is input into the policy network, and the first parameter and the second parameter are output, including: the t-th state feature vector is input into the first actor, and the first parameter is output; the first parameter is used to represent the probability distribution of each operator as the t-th operator; the t-th state feature vector is input into the second actor, and the second parameter is output; the second parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter is the expected value of the normal distribution, and the second sub-parameter is the standard deviation of the normal distribution; based on the first parameter, the t-th operator is determined, including: according to the first parameter, classified distribution sampling is performed to determine the t-th operator; based on the second parameter, the t-th group of parameters corresponding to the t-th operator is determined, including: according to the first sub-parameter and the second sub-parameter, normal distribution sampling is performed to determine the t-th group of parameters corresponding to the t-th operator.

[0017] In a feasible implementation, the method further includes: inputting a historical action trajectory into a preset exploration budget model, and outputting a first coefficient and a second coefficient; wherein the historical action trajectory includes the environment state, operator information, parameter information, and reward function value corresponding to at least one action that has been executed before the tth action; based on the first coefficient and the second coefficient, determining the loss function value corresponding to the exploration budget model; and optimizing the trainable parameters in the policy network according to the loss function value. The introduction of the exploration budget mechanism can reduce the exploration of invalid sequences, allowing the model to explore in a more promising direction and improve efficiency.

[0018] In a feasible implementation, an exploration budget model includes a second encoder based on an attention mechanism and an exploration budget network; inputting a historical action trajectory into a preset exploration budget model, and outputting a first coefficient and a second coefficient, including: inputting the historical action trajectory into a second encoder, and outputting a first difference vector to a third difference vector of the t-th action relative to the t-1-th action; the first difference vector is used to characterize the difference information of the t-th operator relative to the t-1-th operator; the second difference vector is used to characterize the difference information of the t-th group of parameters relative to the t-1-th group of parameters; the third difference vector is used to characterize the difference information of the reward function value of the environment for the t-th action feedback relative to the reward function value of the environment for the t-1-th action feedback; the difference vector is input into the exploration budget network, and the first coefficient and the second coefficient are output.

[0019] In a feasible implementation, based on the first coefficient and the second coefficient, the loss function value corresponding to the exploration budget model is determined, including: determining the entropy of the normal distribution corresponding to the t-th state feature vector; determining the entropy of the classification distribution corresponding to the t-th state feature vector; based on the first coefficient, the second coefficient, the entropy of the normal distribution and the entropy of the classification distribution, obtaining the loss function value corresponding to the exploration budget model.

[0020] In a second aspect, an embodiment of the present application provides a training method, which is based on reinforcement learning, and reinforcement learning includes a policy network and a value network; the value network includes a value function, and the policy network includes trainable parameters; the method includes: determining the tth state feature vector; wherein the tth state feature vector is used to characterize the state of the environment after executing the t-1th action; t is an integer variable and t≥1; optimizing the trainable parameters in the policy network at least according to the value of the value function; wherein the value of the value function is the output obtained by inputting the tth state feature vector into the value network; inputting the tth state feature vector into the policy network, and outputting at least a first parameter; based on the first parameter, determining the tth operator to obtain the tth action; wherein the tth action is used to execute the tth operator on the execution object; the tth operator and the t-1th operator corresponding to the t-1th action are operators of the same type or operators of different types, and the execution object of the action is a logic circuit or other object.

[0021] In a feasible implementation, the method further includes: inputting a historical action trajectory into a preset exploration budget model, and outputting a first coefficient and a second coefficient; wherein the historical action trajectory includes the environment state, operator information, parameter information, and reward function value corresponding to at least one action that has been executed before the tth action; based on the first coefficient and the second coefficient, determining the loss function value corresponding to the exploration budget model; and optimizing the trainable parameters in the strategy network according to the loss function value.

[0022] In a feasible implementation, based on the first coefficient and the second coefficient, the loss function value corresponding to the exploration budget model is determined, including: determining the entropy of the categorical distribution obeyed by the tth operator; determining the entropy of the normal distribution obeyed by the tth group of parameters; based on the first coefficient, the second coefficient, the entropy of the normal distribution and the entropy of the categorical distribution, obtaining the loss function value corresponding to the exploration budget model.

[0023] An embodiment of the present application further provides a computer program product, which includes a program. When the program is executed by an electronic device, the electronic device implements the method as described in any one of the first or second aspects above.

[0024] In a third aspect, the technical solution of the present application also provides an electronic device for sending data through a data transmission interface, the electronic device comprising: one or more processors; a memory; at least one application; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute a method as described in any one of the first or second aspects above.

[0025] In a fourth aspect, the technical solution of the present application also provides a storage medium, including computer instructions, which, when executed on an electronic device, enables the electronic device to execute a method as described in any one of the first or second aspects above.

[0026] In a fifth aspect, the technical solution of the present application also provides a chip system, comprising: a communication interface for inputting and / or outputting data; a processor for executing a computer executable program so that a device equipped with the chip system executes a method as described in any one of the first or second aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of operator sequence sorting rules in the related art;

[0028] Figure 2 This is a schematic diagram of an application scenario in which the method provided in an embodiment of the present application is applied to chip design;

[0029] Figure 3 It is a schematic diagram of operator sorting in an action sequence in one embodiment of the method provided by the present application;

[0030] Figure 4 is a schematic diagram of the system architecture of an embodiment of the method provided by the present application;

[0031] Figure 5 It is a flowchart of an embodiment of the method provided by the present application;

[0032] Figure 6It is a schematic diagram of the operating principle based on reinforcement learning in one embodiment of the method provided by the present application;

[0033] Figure 7 is a schematic diagram of a hybrid characterization module in one embodiment of the method provided by the present application;

[0034] Figure 8 is a schematic diagram of a policy network module in an embodiment of the method provided by the present application;

[0035] Fig. 9 is a schematic diagram of a system architecture in which an exploration budget model is added in one embodiment of the method provided by the present application;

[0036] Fig.10 is a schematic diagram of a structure of an exploration budget model in an embodiment of the method provided by the present application;

[0037] Fig.11 is a flowchart of an embodiment of the method provided by the present application in the training phase;

[0038] Fig.12 is a schematic diagram of the system architecture of an embodiment of the method provided by the present application;

[0039] Figure 13 (a)-(d) are example diagrams of insertion operators;

[0040] Fig.14 is a diagram showing an example of a hierarchical structure of a Budget tuner in one embodiment;

[0041] Fig.15 An example diagram of an operator and parameter space in an embodiment;

[0042] Fig.16 This is an example diagram of the comparative effect data of different algorithms under 1000 samples;

[0043] Fig.17 This is an example of the EPFL optimal area / level refresh result data;

[0044] Fig.18 Schematic diagram of the hardware structure of the electronic device. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0046] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0047] Chip design is divided into front-end design and back-end design. Front-end design generally includes specification formulation, RTL (Register Transfer Level) design, simulation verification, logic synthesis (Logic Synthesis), static timing analysis (Static Timing Analysis, STA), formal verification and other stages. Front-end design is the process from determining design requirements to outputting the netlist; back-end design includes design for testability (Design For Test, DFT), physical implementation, clock tree synthesis (Clock Tree Synthesis, CTS), routing (Place & Route), parasitic parameter extraction, layout physical verification and other stages. The physical layout is handed over to the chip foundry in the GDS II file format, and the actual circuit is made on the silicon wafer, and then packaged and tested to get the chip.

[0048] Logic synthesis is one of the core steps of chip design. Its goal is to convert abstract logic descriptions into actual physical circuits. Generally, it converts high-level hardware descriptions into gate-level circuit netlists required for physical implementation. For example, RTL-level circuits described in Verilog or VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) are converted into gate-level netlists for placement, routing, and verification at the chip level. Logic synthesis connects the front-end and back-end of chip design and is crucial to the final power consumption, latency, and area (PPA) of the chip.

[0049] At present, EDA tools play a vital role in the field of electronic design. EDA tools are usually required for translation and optimization in the logic synthesis stage. EDA tools use computers and related technologies to assist in the design and verification of electronic chips and systems, accelerating the development process of electronic products. With the development of technology, modern electronic systems have become more and more complex. An industrial chip often contains millions or even tens of millions of logic gates. Traditional EDA tools face challenges in dealing with increasingly complex designs and growing scales.

[0050] The logic synthesis stage can be divided into three stages: translation, logic optimization (LO) and mapping. Each stage contains different operators. These operators start from different design principles and optimize the circuit structure under the premise of satisfying circuit equivalence.

[0051] The design of the operator itself is important, but how to schedule the operation order of the operator and select the operator parameters also determines the optimization effect of the logic synthesis process. In the traditional solution of related technology, the selection of operator sequence and parameters is optimized through manual configuration. Due to the limitation of subjective cognition, it is difficult to find the sequence and parameter configuration that can achieve the expected result.

[0052] In recent years, with the vigorous development of artificial intelligence (AI), AI has achieved great success in many fields. In order to overcome EDA design challenges, electronic design practitioners are also actively trying to combine AI technology with knowledge in this technical field to achieve better chip modeling and larger-scale parameter optimization. In response to the operator sequence parameter search problem faced by logic synthesis, various fields are trying to use data-driven AI technology for optimization.

[0053] To facilitate understanding of the solutions proposed in the embodiments of the present application, the relevant technologies are briefly described:

[0054] Related technology 1:

[0055] A heuristic and iterative logic synthesis flow, the goal is to optimize the PPA of the final netlist. For different circuits, it can dynamically generate operator sequences designed in advance based on expert experience, using two-layer nested iteration and deterministic heuristic methods to help the algorithm jump out of the local minimum, reduce the running time after convergence, and generate adaptive sequences that adapt to different circuits. By adjusting the selection and parameters of operators and iteration rounds, a trade-off can be made between running time and synthesis indicators.

[0056] The inventors found in the research process that the related technology is based on expert experience, and multiple groups of operator sequences are designed in advance, and then the multiple groups of operator sequences are spliced ​​into the final optimized sequence. Among them, a group of operator sequences includes multiple operators, and the order of multiple operators within a group of operator sequences is fixed and cannot be adjusted. Instead, the splicing and ordering between groups are performed in groups. For example, Figure 1As shown, based on expert experience, we can obtain area-oriented logic optimization group (i.e., Area-oriented logic opt shown in the figure), delay-oriented logic optimization group (Delay-oriented logic opt), area-oriented process mapping group (Area-oriented tech.mapping), delay-oriented process mapping group (Delay-oriented tech.mapping), area-oriented postmapping group (Area-oriented postmapping), delay-oriented postmapping group (Delay-oriented postmapping), etc., where, taking the area-oriented logic optimization group as an example, the group includes multiple logic optimization operators (i.e., LO operators shown in the figure), but does not include other types of operators such as process mapping; taking the delay-oriented postmapping group as an example, the delay-oriented postmapping group includes multiple post-mapping operators (PM operators), but does not include other types of operators, that is, the sequence within each group only contains operators of the same type, and the order between operators within the group cannot be adjusted. In addition, there are also fixed order restrictions between groups, for example, the logic optimization group should be before the process mapping group, and the process mapping group should be before the post-mapping group.

[0057] In this way, the sequence tuning mechanism adopted by the related technology 1 limits the exploration space of the operator sequence, making it impossible to explore the operator sequence that may achieve better performance; in addition, the scheme cannot optimize the operator parameters in the sequence, and the operator parameters are also one of the important factors affecting the optimization effect. Failure to optimize the parameters will also limit the optimization effect.

[0058] Related technology 2:

[0059] Bayesian Optimization (BSBO) is an efficient data-driven black-box optimization method that uses collected data to build a surrogate model and uses the surrogate model to guide the optimization process. Specifically, BSBO uses the Embedding Align Cell (EAC) to model sequence parameters and uses this to build a neural network as a surrogate model, using historical data to estimate the quality of results (QoR) of the logic synthesis process. With the help of the surrogate model, the algorithm balances the exploration and exploitation of QoR through acquisition functions. In each iteration, the algorithm seeks the optimal solution of these acquisition functions to select a batch of sequences (including operators and corresponding hyperparameters) for user evaluation.

[0060] However, the inventors found that the sequence recommended by BSBO is of fixed length and cannot be adjusted dynamically. Limiting the operator sequence to a fixed length also limits the exploration space to a certain extent. In addition, BSBO fails to utilize the intermediate state information of the netlist, and the sample utilization rate is low.

[0061] It should be noted that in BSBO, although the sequence contains operators and corresponding hyperparameters, in the context of machine learning, hyperparameters are parameters whose values ​​are set before starting the learning process, rather than parameter data obtained through training. In other words, hyperparameters are pre-set and will not be optimized during the training process.

[0062] The present application embodiment provides an action prediction method and a training method, see Figure 2 As shown, exemplarily, the method can be applied in chip design application scenarios, and specifically can be applied in the logic synthesis stage in front-end design. Figure 2 In the system specification, System Specification means system specification, architecture Design means architecture design, high-level Synthesis means high-level synthesis, Logic Design (RTL) means logic design at RTL level; Logic Synthesis means logic synthesis; Physical Synthesis means physical synthesis, also called automatic layout generation; Physical Verification means physical verification; Fabrication Packing and Testing means manufacturing, packaging and testing.

[0063] Following the application scenario of chip design, those skilled in the art can also apply the method to other application scenarios that require prediction of operator sequences. For example, it is not ruled out that the method can be applied to actual application scenarios such as production process optimization and personnel structure optimization.

[0064] In the application scenario of chip design logic synthesis, the execution object of the action is a logic circuit. In other application scenarios, it can be other execution objects, such as a topological structure diagram used to describe the production process flow. The following is an example of an application scenario of logic synthesis and an execution object of a logic circuit.

[0065] The method provided in the embodiment of the present application proposes an action sequence prediction scheme based on reinforcement learning. The intelligent agent predicts the next operator and the parameters corresponding to the next operator. The operator and the corresponding operator parameters constitute the next action, and at least one action constitutes an action sequence. After multiple rounds of algorithm iterations, the optimal sequence and parameter configuration can be found, and the sequence and parameter configuration obtained by the optimization are applied to the logic synthesis process, and the synthesized netlist is handed over to the back-end layout and wiring for further EDA design. This method can weigh the tuning time overhead and optimization effect, optimize the logic synthesis effect to a greater extent within a limited sample space, effectively reduce the chip's latency and area, and optimize performance.

[0066] As mentioned above, logic synthesis can be divided into three stages: translation, logic optimization, and mapping. The method provided in the embodiment of the present application can be applied after the translation stage, that is, to tune the sequence and parameters of various operators used in the logic optimization, mapping and other stages, with the logic circuit to be processed as the execution object, and through interaction with the environment, the action sequence is predicted to obtain a chip with better PPA.

[0067] In some embodiments of the present application, mapping is further divided into technology mapping (TM, hereinafter referred to as mapping) and post mapping (PM), which is equivalent to dividing the stage after translation in logic synthesis into three parts: logic optimization (LO), mapping (TM), and post mapping (PM).

[0068] Logic optimization: According to the imposed timing and area constraints, the translation results are logically reorganized and optimized according to a certain process. For example, logic optimization is performed on logic circuits (such as NAND graphs) to reduce the area, power consumption or delay of the circuit. Common optimization techniques include logic merging, logic decomposition, constant propagation and redundant logic removal.

[0069] Mapping can be understood as searching for qualified cells from the target process library to form the logic netlist of the actual circuit according to the imposed timing and area constraints. Among them, the goal of technology mapping (i.e., process mapping) is to select the logic gates that best suit the target technology library to achieve the requirements of performance, power consumption, and area, and replace the logic gates in the logic netlist with gates in a specific target technology library for subsequent physical layout and routing.

[0070] Post-mapping, under the premise of meeting the design constraints, further improves and optimizes the performance and resource utilization of the gate-level netlist. By applying appropriate optimization techniques and algorithms, the efficiency, reliability and maintainability of the circuit can be improved.

[0071] Each link includes different operators. The logic optimization link includes one or more LO operators, the mapping link includes one or more TM operators, and the post-mapping link includes one or more PM operators. For example, LO type operators include rewrite, st, blut, b, dsdb, if-g, if-y, dc2, dch, etc.; TM type operators include if, if2, lf, mf, jf, kf, etc.; PM type operators include mfs, speedup, satlut, etc.

[0072] In addition, in other embodiments, logic synthesis may also include high-order RTL optimization and post-optimization stages. Therefore, in the embodiments of the present application, the operators involved in the arrangement may also include high-order RTL optimization operators and / or post-optimization operators. The following is mainly illustrated by taking LO operator, TM operator and PM operator as examples, and the processing method of adding other operators can be obtained accordingly.

[0073] In another part of the embodiments, the post-translation phase in logic synthesis can still be divided into two phases: logic optimization and mapping. Both the TM operator and the PM operator are regarded as operators in the mapping phase, and both belong to the mapping type.

[0074] It should be noted that, in the embodiments of the present application, an operator is an abbreviation for a series of specified operations performed on an execution object. An operator can be understood as a functional unit or computing unit that contains a specified program or command.

[0075] The method provided in this application proposes improvements in at least the following aspects in different embodiments:

[0076] In some embodiments, the present application proposes to perform all-in-one modeling for logic synthesis, that is, mixed modeling of multiple types (more than two) of operators, allowing different types of operators to be connected, supporting the mixed appearance of operators in the three stages of LO, TM, and PM, and is not limited to the fixed pattern sequence of LO→TM→PM. All-in-one modeling is also reflected in that when modeling, there is no need to set different strategies for different types of operators in the strategy network, but prediction can be made through the same strategy.

[0077] In the same embodiment or some other embodiments, not only the operator is predicted, but also the operator parameters are predicted, which solves the problem that the operator parameters cannot be optimized; and the operator parameters can be dynamically predicted and adjusted, are not hyperparameters, and can be continuously optimized during the training process.

[0078] In the same embodiment or some other embodiments, the present application also proposes a hybrid representation of the logic synthesis environment state, introducing the intermediate state of the circuit into the training process, and hybridly representing the current environment state through the circuit state, the graph structure of the circuit diagram, or in combination with the historical action sequence.

[0079] In the same embodiment or some other embodiments, the present application also proposes a budget exploration module to dynamically adjust the exploration budget of operator actions and parameter actions.

[0080] In the same embodiment or some other embodiments, the action sequence is a variable-length sequence, and the length of the action sequence may be the number of operators in the action. Reinforcement learning will evaluate in real time whether to continue predicting the next action based on the feedback of the environment and the value network on the action. Therefore, the number of operators in the predicted action sequence is dynamically variable and there is no fixed length limit. This also expands the exploration space in the dimension of sequence length, providing more options for obtaining chips with better performance.

[0081] Specifically, Figure 3 As shown, through the policy control in the intelligent agent, the predicted action sequence supports mixed sorting of operators of different types or the same type in units of single operators, realizing mixed splicing between operators, rather than being limited to fixed-order splicing between groups. In the obtained action sequence based on the operator all-in-one hybrid modeling mechanism proposed in this application, any two adjacent operators can be operators of the same type or operators of different types, while in the related art, two adjacent operators in the same group are operators of the same type, and the related art cannot achieve that any two adjacent operators can be of the same type or different types.

[0082] In addition, based on the mechanism proposed in this application, operators do not have to be arranged in a fixed order of LO→TM→PM, so there are more possibilities for permutations and combinations of operators, which greatly increases the exploration space.

[0083] For example, Figure 3 In the example, the operator in the first action Action 1 is the LO operator, and the operator in the second action Action 2 can also be the LO operator, and the LO operator can be followed by the TM operator (Action 3). In this way, taking a single operator as a unit, any two adjacent operators can be of the same type or of different types, and mixed sorting of operators of different types is supported. Moreover, in the related art, the LO operator, the TM operator, and the PM operator are generally arranged in a fixed order of LO→TM→PM, but in the embodiment of the present application, the operators can disrupt this fixed order. For example, in Figure 3In the example, the TM operator in the t-th action (Action t) can be arranged after the PM operator in the t-2-th action (Action t-2), but a Strash (Action t-1) operator needs to be inserted in between. The insertion of operators will be explained in detail in the subsequent content.

[0084] The action prediction method provided by the embodiment of the present application is described in detail below.

[0085] See also Figure 4 As shown, the method provided in the embodiment of the present application is based on the Actor-Critic mechanism, predicts the next action through the Actor (strategy network), takes the logic circuit (for example, the logic circuit includes the AIG circuit and / or the LUT circuit) as the execution object, and after executing the predicted next action, at least the current state of the logic circuit is used as the environment state to obtain the state feature vector, and the state feature vector is input into the strategy network Actor and the value network Critic, and the reward function value Reward of the environment (Env) for the current action feedback is obtained. The Actor adjusts the trainable parameters in the Actor according to the reward function value and the value V of the value function output by the Critic, which is equivalent to adjusting the strategy. According to the adjusted strategy, the next action Action is predicted, and this iteration is performed until the logic circuit meets the expected indicators or performance, and the prediction is stopped.

[0086] Among them, Figure 6 As shown, Actor is equivalent to the RL agent in reinforcement learning. The agent will predict the next action based on the evaluation (observation) fed back by the logic synthesis reinforcement learning environment (Logic synthesis RL Env) and the strategy of the agent. In some embodiments, the next action includes operators and parameters. In other embodiments, the next action may only include operators. After determining the next action, the execution object, such as a logic circuit, is shown in the example, where the logic circuit can be an AIG circuit and / or a LUT circuit. According to the predicted operator parameters, the corresponding operator is executed on the logic circuit to obtain the reward score reward (i.e., the reward function value) fed back by the environment. According to the reward score, the agent optimizes the adjustable parameters in the strategy so that the next action can obtain a higher reward score.

[0087] In some embodiments, a hybrid representation method is used to represent the current environment state, that is, the state feature vector used to represent the current environment state is obtained by fusing or mixing the state feature corresponding to the logic circuit, the feature corresponding to the graph structure of the circuit in the specified format in the logic circuit, and the feature corresponding to the historical action sequence. This will be explained in detail in the subsequent content.

[0088] Exemplarily, the specific product form obtained based on the method proposed in the embodiment of the present application can be an application or a plug-in of the application, for example, it can be an EDA tool (application) or a functional plug-in for installation in an EDA tool.

[0089] It should be noted that the specific product implemented based on the method provided in the embodiment of the present application can be a product corresponding to the training stage or a product corresponding to the application stage. The training stage is the stage of training the reinforcement learning model; the application stage is the stage of applying the trained reinforcement learning model to the actual application scenario for action prediction.

[0090] In some embodiments, during the application stage, based on the method proposed in the embodiments of the present application, the product supplier can provide users with related products after training, predict the action sequence of the actual execution object, and there is no need to optimize the trainable parameters in the policy network Actor during the prediction process. In some other embodiments, an automatic adjustment mechanism for trainable parameters can be set, that is, during the application stage, without manual intervention, the parameters can be automatically adjusted according to the feedback of the environment on the action. In this case, the actual product provided by the product supplier can be based on the training. In the actual predicted application scenario, further optimize its own parameters, so as to achieve better performance in actual applications. That is, the product produced based on the method proposed in the embodiments of the present application can continue to adjust the parameters or not during the application stage.

[0091] Specifically, the embodiment of the present application provides an action prediction method that can be performed based on reinforcement learning. In the training stage, reinforcement learning includes a policy network Actor and a value network Critic; wherein the value network includes a value function, the policy network includes trainable parameters, and the Actor performs policy control (optimizes trainable parameters) based at least on the evaluation of the Critic (the value of the value function). In the application stage, the reinforcement learning model includes at least a trained policy network Actor, that is, in the application stage, the Critic may or may not be set in the specific product provided by the embodiment of the present application.

[0092] like Figure 5 As shown, the action prediction method proposed in the embodiment of the present application may include the following process:

[0093] S101, determine the tth state feature vector.

[0094] S102, input the tth state feature vector into the policy network, and output at least a first parameter.

[0095] S103, based on the first parameter, determine the tth operator and obtain the tth action.

[0096] In S101, the t-th state feature vector is a feature vector obtained after performing a mixed representation on the current environment state after executing the t-1-th action.

[0097] like Figure 4 As shown, in some embodiments, the t-th state feature vector can be obtained based only on the state of the logic circuit, that is, the environment state can only include the state of the logic circuit. In some other embodiments, the environment state can at least include the state of the logic circuit and the graph structure information corresponding to the logic circuit. Then, determining the t-th state feature vector is to perform a mixed representation of the environment state based on at least the state of the execution object after executing the t-1th action and the graph structure information corresponding to the execution object to obtain the t-th state feature vector.

[0098] Specifically, in some embodiments, the environment state can be represented by a mixed representation of the state of the logic circuit, the graph structure information corresponding to the logic circuit, and the historical action sequence. For example, the mixed representation can be implemented in the following manner:

[0099] Based on the state of the execution object after executing the t-1th action, the tth first feature vector is obtained; based on the graph structure information corresponding to the execution object, the second feature vector is obtained; based on the historical action sequence, the tth third feature vector is obtained.

[0100] The historical action sequence at least includes information about operators of at least one action that has been executed before the t-th action is executed, generally information about operators corresponding to the first action to the t-1-th action, and in some embodiments, parameter information corresponding to each operator may also be included.

[0101] The information of the operator may be identity information used to indicate the identity of the operator, such as a number or the name or abbreviation of the operator. The operator is actually a functional unit corresponding to a section of executable code, and the action sequence only includes information such as the name or number of each operator to distinguish different operators. In some embodiments, a limited number of candidate operators may be numbered, for example, using a fixed number of binary digits as numbers to indicate different operators.

[0102] After obtaining the first to third eigenvectors, the tth state eigenvector is obtained based on the tth first eigenvector, the second eigenvector and the tth third eigenvector. In some embodiments, the first to third eigenvectors may be subjected to an element-wise addition operation. The addition operation may be a weighted summation, that is, in some embodiments, a weighted summation operation may be performed on the elements at corresponding positions, thereby integrating the three eigenvectors into one eigenvector, and the integrated state eigenvector carries the feature information of the three eigenvectors. In other embodiments, the three eigenvectors may also be subjected to a horizontal splicing or a vertical splicing operation.

[0103] Specifically, in some embodiments, the tth first feature vector is obtained based on the state of the execution object after executing the t-1th action. This can be done by determining the state vector of the execution object in the current state after executing the t-1th action, inputting the state vector into a preset multilayer perceptron, and outputting the first feature vector through the multilayer perceptron.

[0104] In the same embodiment or other embodiments, obtaining the second feature vector based on the graph structure information corresponding to the execution object may be to determine the graph structure information corresponding to the execution object, input the graph structure information into a preset graph neural network, and output the second feature vector through the graph neural network.

[0105] In the same embodiment or other embodiments, based on the historical action sequence, the t-th third feature vector is obtained, and the historical action sequence can be input into a preset first encoder to obtain the t-th third feature vector; wherein the first encoder is an encoder based on an attention mechanism.

[0106] For example, the execution object is a logic circuit, such as Figure 7 As shown, h t That is, it is the t-th state feature vector, Env is the environment, the logic circuit includes AIG (And-Inverter Graph) circuit and / or LUT circuit; States represents the state vector of the logic circuit in the current state, that is, the state vector corresponding to the intermediate state after executing the t-1th action; Rawgraph represents the original graph structure information of the logic circuit, Previous actions represents the historical action sequence, MLP is a multilayer perceptron (Multilayer Perceptron, MLP), in addition to the input and output layers, there can be at least one hidden layer, the state vector of the intermediate state of the logic circuit is input into the MLP to obtain the first feature vector.

[0107] The AIG circuit or LUT shown in the figure is only an exemplary diagram. The AIG circuit is a circuit format obtained after translating the RTL level code based on an EDA tool. However, it should be noted that the circuit format after translation may be different in different EDA tools. For example, based on another EDA tool, the translation (conversion) is to convert the HDL code into its own GTECH format. Therefore, in other embodiments of the present application, the logic circuit may also be a GTECH (generic technology) netlist or other circuit formats after translation.

[0108] It should be noted that, in the embodiments of the present application, the logic circuit may be a circuit in various formats or states after the translation is performed in the logic synthesis stage, and specifically may be a circuit in an intermediate state before or after the execution of each operator. For example, in some embodiments, when determining the state vector corresponding to the current state of the logic circuit, the logic circuit may include an AIG circuit before the execution of the TM operator, or may include a LUT circuit after the execution of the TM operator. In some embodiments, when determining the graph structure information, the logic circuit may be an AIG circuit before the TM operator is executed.

[0109] For example, the conversion relationship between the AIG circuit, the LUT circuit and various types of operators is as follows: Figure 7 or Figure 6 As shown in , the AIG circuit executes the LO operator, that is, the AIG circuit is logically optimized to obtain the AIG circuit, the TM operator is executed on the AIG circuit to obtain the LUT circuit, the PM operator is executed on the LUT to obtain the LUT circuit, and the LUT circuit executes the backoff operator Strash to obtain the AIG circuit. The operation performed by the backoff operator Strash (Structural Hash) is equivalent to the inverse mapping process of the TM operator.

[0110] In some embodiments, States can be obtained as follows:

[0111] Based on at least one of the first set of information, the second set of information, and the third set of information, a state vector of the logic circuit in a current state is determined.

[0112] The first set of information may include at least one of the number of nodes, logic depth, and number of edges of the logic circuit after executing the t-1th action. Taking the AIG circuit as an example, AIG (AND-INV Graph) is a Boolean network composed of AND gates and NOT gates, each node represents a two-input AND gate, and the arc represents a connection. Therefore, illustratively, the number of nodes of the AIG circuit can be the number of logic gates, such as the number of two-input AND gates.

[0113] In some embodiments, the depth of the logic circuit may be the number of edges of the longest path among multiple paths formed from input to output according to the signal flow direction. A path is a passage of the logic circuit.

[0114] The number of edges in a logic circuit can be the sum of the number of edges in each path from input to output.

[0115] It should be noted that the first set of information is intended to quantitatively express the logic circuit, and different formats of logic circuit files can be expressed in different ways. For example, for the GTECH netlist, the logic gate unit used is a symbol, which contains common logic gates (such as AND, NOR, triggers, etc.) and common operators (addition, subtraction, multiplication, division, shift, comparison, selection, etc.). Therefore, for the GTECH netlist, the complexity of its logic topology relationship can be expressed by the number of logic gates and the number of connecting edges between logic gates.

[0116] The second set of information includes at least one of the number of LUTs and the logical depth of the LUT in the Look-Up-Table (LUT) circuit. For a LUT circuit with a maximum address line number of 6, the number of LUTs refers to the number of 6-address line LUT units and LUT units below 6-bit address lines; or, for a LUT circuit with a maximum address line number of 4, the number of LUTs refers to the number of LUT units with 4-bit address lines and below. A LUT circuit is a circuit containing a LUT obtained after executing the TM operator on a logic circuit; for example, a circuit generated by mapping an AIG circuit with a LUT as the basic unit is a LUT circuit.

[0117] In some embodiments, the logic depth of the LUT circuit may be the number of LUTs that the longest path passes through among multiple paths formed from input to output. For example, in a LUT circuit, there are 7 LUT units, namely LUT1 to LUT7, and the longest path (path) from input to output passes through LUT2→LUT4→LUT5→LUT7 in sequence, that is, it passes through 4 LUT units. Then, in the second set of information corresponding to the LUT circuit, the logic depth is 4 and the number of LUTs is 7.

[0118] The third set of information is the length of the historical action sequence. Figure 7 In (k0 x0)…(k t-1 x t-1 ) is the historical action sequence, where k0…k t-1 represents the 0th to t-1th operators. The length of the historical action sequence is the number of operators, which is t. (k0x0) = Action 0, which is the 0th action and can be understood as the initialization action. t-1 x t-1)=Action t-1, which is the t-1th action. It should be noted that the operator information or operator parameters in the historical action sequence can be numbered from 0 or 1 or any other number. The tth action can be the action numbered t, that is, Action t=(k t x t ), the t-th action is not limited to the action ranked tth in the sequence. For example, the historical action sequence can also be (k1 x1)…(k t-1 x t-1 ), then the length of the historical action sequence is t-1. In addition, some operators may be inserted into the historical action sequence, so the t-th action may not have a clear quantitative relationship with the number of operators in the historical action sequence.

[0119] In this way, after obtaining the first to third sets of information, the obtained values ​​can be used as elements in the vector to splice into a state vector, which is used to characterize the logic circuit of the intermediate state (i.e., the current state). The state vector is the input of the MLP. After feature extraction or dimension conversion by the MLP, the first feature vector is output.

[0120] In this embodiment, the graph neural network can be a graph convolutional neural network (GCN), and the input raw graph structure information Raw graph can be the original graph structure information of the AIG circuit before the operator is executed. Exemplarily, the graph structure information can be an adjacency matrix, that is, the adjacency matrix corresponding to the AIG circuit graph is input into the GCN, so as to perform message passing based on the node features of the graph structure through the GCN, thereby obtaining output features, that is, obtaining the second feature vector.

[0121] In other embodiments, other graph neural networks can be selected, not limited to GCN. For example, in addition to GCN, the graph neural network can also use one of Graph Attention Networks, Graph Autoencoders, Graph Generative Networks, and Graph Spatial-temporal Networks, or a combination of two or more of them including GCN.

[0122] Optionally, the first encoder can be a transformer model, (k0 x0)…(k t-1 x t-1 ) is the historical action sequence. The historical action sequence is input into the transformer model, and the encoding based on the attention mechanism is performed to obtain the third eigenvector.

[0123] Optionally, in some embodiments, the first eigenvector and the second eigenvector may be firstly added element-wise, and then concatenated or added with the third eigenvector to obtain the t-th state eigenvector h t In other embodiments, the first to third eigenvectors may be summed or concatenated together, so that the obtained h t It is sufficient that the feature information in the first to third feature vectors is included in , and there are many specific mixing methods.

[0124] After the t-th state feature vector is obtained, in S102, the t-th state feature vector is input into the policy network, and at least a first parameter is output.

[0125] In some embodiments, only operators may be predicted without predicting operator parameters; in other embodiments, not only operators but also parameters corresponding to each operator are predicted, such as Figure 5 As shown, the t-th state feature vector is input into the policy network, and the first parameter and the second parameter are output; based on the first parameter, the t-th operator is determined, and based on the second parameter, the t-th group of parameters corresponding to the t-th operator is determined. The t-th operator and the t-th group of parameters constitute the t-th action. The t-th action is used to execute the t-th operator on the execution object according to the t-th group of parameters.

[0126] For example, Figure 6 As shown in the figure, the policy network is regarded as an intelligent agent. In the next action predicted by the policy network, "&dch(LO)" is the t-th operator, LO indicates that the operator type is logic optimization, "dch" is the operator name, and "-W 166–C 2127–S3351" is the t-th group of parameters corresponding to the t-th operator. Then, for the logic circuit to execute the t-th action, it is to execute the "dch" operator with the parameter "-W 166–C 2127–S 3351".

[0127] Specifically, Figure 8 As shown, the policy network may include a first actor and a second actor, the first actor is used to predict a first parameter to determine an operator, and the second actor is used to predict a second parameter to determine an operator parameter. Policy control in the policy network may be implemented through a policy function, for example, a first policy function is deployed in the first actor, and a second policy function is deployed in the second actor, and the first policy function is used to execute the operation from h t The mapping to the first parameter, the second strategy function, is used to perform the operation from h tThe policy function may include trainable parameters, and policy control is performed by adjusting the trainable parameters in the policy function. It should be noted that the mapping of the function and the mapping in the above-mentioned mapping operator are not equivalent technical concepts and should not be confused.

[0128] Specifically, the t-th state feature vector h t Input to the first actor, output the first parameter, illustratively, each element in the first parameter is used to represent the probability distribution of each operator as the t-th operator; for example, there are 10 operators, then the output first parameter can be a 10-dimensional vector, and the elements of each dimension represent the probability of the corresponding operator as the t-th operator.

[0129] h t Input to the second actor, output the second parameter. Exemplarily, the second parameter includes a first sub-parameter and a second sub-parameter. In some embodiments, the first sub-parameter is the expected value of the normal distribution, and the second sub-parameter is the standard deviation of the normal distribution. In other embodiments, the first sub-parameter may be the mean, and the second sub-parameter may be the covariance.

[0130] Next, illustratively, the operators and corresponding operator parameters may be determined respectively in the following manners:

[0131] According to the first parameter, categorical distribution sampling is performed to determine the tth operator. Categorical distribution, namely Categorical Distribution, can also be called category distribution or category distribution.

[0132] Exemplarily, the first parameter P(k) can be a set of non-normalized parameters, which can be sampled by inverse transform sampling. Based on the first parameter, a sample in the form of One-hot is sampled. One-hot is unique hot coding, where only one bit is 1 and the rest are 0. Then, as a feasible way, if the bit is 1, the operator corresponding to the bit is selected as the t-th operator kt. For example, assuming that the sampled sample is 0001000000, the operator numbered decimal 4 is selected as the next operator, that is, the t-th operator. In other embodiments, the number of each operator itself can be set to a string of binary digits in the form of unique hot coding, and the sample obtained by sampling is the number of the operator.

[0133] The sampling method based on classification distribution is not limited to inverse transform sampling, but can also be sampling based on the Gumbel-Max technique, or using a random simulation method, namely the Monte Carlo simulation method, to generate samples of the output probability distribution P(k) (the first parameter), namely sampling.

[0134] It should be noted that, in other embodiments, multinomial distribution may be used instead of categorical distribution.

[0135] And, according to the first sub-parameter and the second sub-parameter, a normal distribution sampling is performed to determine the tth group of parameters corresponding to the tth operator. For example, the Metropolis-Hastings algorithm can be used to perform normal distribution sampling, or the Monte Carlo method can be used to generate random numbers that conform to the normal distribution and add them to a sample set, and then statistics such as the mean and variance can be estimated through the sample set.

[0136] In this way, we can get Action t=(k t x t ).

[0137] The currently predicted Action t is added to the historical action sequence to obtain the action sequence.

[0138] In some embodiments, the action sequence may only include operator information, and operator parameters may be called during the execution of the operator. In other embodiments, the action sequence may include operator information and operator parameter information.

[0139] It should be noted that, in the action sequence obtained based on the method provided in the embodiment of the present application, the t-th operator and the t-1-th operator corresponding to the t-1-th action are operators of the same type or operators of different types, and t is a variable. That is, the solution proposed in the present application supports any two adjacent operators in the sequence to be of the same type or different types, and mixed sorting is carried out in units of operators.

[0140] Furthermore, in the embodiment of the present application, the action sequence is a variable-length sequence, which further expands the exploration space compared to a fixed-length sequence.

[0141] After getting Action t, according to parameter x t , perform operator k on the logic circuit t During the training or application phase, the trainable parameters in the policy network are optimized based on the reward function of the environment for Action t feedback.

[0142] Optionally, in some embodiments, the present application also proposes an exploration budget mechanism and designs a loss function based on the exploration budget mechanism. Since both operators and parameters are adjustable and support operator-level mixing, different operators can explore a huge space of action sequences in different orders and with different parameter combinations, making it difficult for the algorithm to explore and learn all reachable circuit states in limited samples. In order to balance the exploration between operators and parameters, in some embodiments of the present application, an exploration budget module is introduced to drive the algorithm to always focus on those search spaces with more promising prospects and better performance, rather than getting entangled in those poorer states. The budget exploration module dynamically adjusts the exploration budget of operator actions and parameter actions to improve sample efficiency and performance.

[0143] like Fig. 9 As shown, the historical action trajectory is input into the exploration budget model to obtain an output. According to the output of the exploration budget model and the output of the policy network, the preset loss function L is calculated. exp The corresponding value, according to L exp To adjust the trainable parameters in the policy network Actor and optimize the agent.

[0144] The historical action trajectory includes at least the operator information of multiple actions that have been executed before the tth action. In some embodiments, the historical action trajectory includes the environment state, operator information, parameter information, and reward function value corresponding to each action numbered 0 to t-1. Fig.10 As shown, the historical action trajectory <s0,k0,x0,r0;s1,k1,x1,r1;…;s t-1 ,k t-1 ,x t-1 ,r t-1 >, where s represents the current environment state, that is, the current state of the logic circuit states; k is the operator, x is the operator parameter, and r is the reward score (reward function value or reward value) obtained by executing a single action.

[0145] In some embodiments, the reward function expression is as follows:

[0146]

[0147] in, It indicates the highest quality score up to time t, that is, up to the tth action; Q(s t+1 ) represents the quality score of the environment’s feedback for the t-th action; α p is a constant, such as 0.01 or 0.001. min indicates a smaller value.

[0148] Q(s t+1 )=β1*Lt +β2*M t ;

[0149] L t It represents the logic depth (level) of the LUT circuit in the logic circuit after the t-th action, M t represents the number of LUT units in the LUT circuit; β1 and β2 represent L t 、M t The corresponding weight.

[0150] Specifically, in some embodiments, Fig.10 As shown, the exploration budget model can include a second encoder based on the attention mechanism and an exploration budget network, and the historical action trajectory (s0 k0 x0 r0)…(s t-1 k t-1 x t-1 r t-1 ) is input to a preset second encoder and outputs a first difference vector to a third difference vector.

[0151] The first difference vector to the third difference vector are used to quantify the changes in the operator, parameter and reward score of the next action (t-th action) predicted by the encoder compared with the previous action (t-1 action). Specifically, the first difference vector Δ k It is used to represent the difference information of the t-th operator relative to the t-1-th operator. The second difference vector Δ x , used to represent the difference information of the t-th group of parameters relative to the t-1-th group of parameters; the third difference vector Δr, used to represent the difference information of the reward function value of the environment's feedback on the t-th action relative to the reward function value of the environment's feedback on the t-1-th action.

[0152] After obtaining the difference vector, the difference vector is input into the exploration budget network and the first coefficient b is output. k and the second coefficient b x The first coefficient is used to characterize the exploration budget corresponding to the operator predicted by the exploration budget model, and the second coefficient is used to characterize the exploration budget corresponding to the parameter predicted by the exploration budget model.

[0153] After obtaining the first coefficient and the second coefficient, the value of the loss function can be calculated based on the first coefficient and the second coefficient. In some embodiments, the loss function L corresponding to the exploration budget model exp , can be a function with the first coefficient and the second coefficient as independent variables.

[0154] In other embodiments, the loss function L may be determined in the following manner: exp Values:

[0155] Determine the entropy of the categorical distribution obeyed by the t-th operator, determine the entropy of the normal distribution obeyed by the t-th group of parameters, and then, based on the first coefficient, the second coefficient, the entropy of the normal distribution and the entropy of the categorical distribution, obtain the loss function value corresponding to the exploration budget model. That is, in this embodiment, the loss function L exp It is calculated using the first coefficient, the second coefficient, the entropy of the normal distribution, and the entropy of the categorical distribution as independent variables.

[0156] Get the loss function L exp After the value of is determined, the trainable parameters in the policy network can be optimized based on this value to control the strategy to search for a more promising and better performing space, avoiding wasting time and computing resources on some worthless sequences.

[0157] The embodiment of the present application further provides a training method, which can be applied to the training stage. According to the foregoing content, it is also possible to train parameters in the application stage, so the training method can also be applied to the application stage.

[0158] Specifically, the training method is based on reinforcement learning execution, such as Fig.11 As shown, reinforcement learning includes a policy network Actor and a value network Critic; the value network includes a value function, and the policy network includes trainable parameters. The training method may include the following process:

[0159] S201, determine the tth state feature vector.

[0160] S202, optimizing trainable parameters in the policy network at least according to the value of the value function.

[0161] The value of the value function is the output obtained by inputting the tth state feature vector into the value network.

[0162] S203, input the t-th state feature vector into the policy network, and output at least a first parameter.

[0163] S204, determining the tth operator based on the first parameter, and obtaining the tth action.

[0164] It should be noted that in terms of execution sequence, the training method can be executed before the action prediction method, and the step number or description order does not serve as a limitation on the execution sequence of the corresponding method or step.

[0165] For the specific implementation of the training method, please refer to the above-mentioned action prediction method and will not be described in detail.

[0166] It should be noted that in the above description of the action prediction method, operations related to training, such as the process of adjusting parameters based on the value of the value function or the value of the loss function, are generally implemented in the training phase. Therefore, the product obtained based on this method can be a mature product that has been trained, and the above-mentioned training-related processes may not be implemented.

[0167] A specific embodiment is listed below, based on which other embodiments can be adaptively obtained:

[0168] First, in this embodiment, the operator sequences of the three stages of logic optimization, process mapping and post-mapping are optimized. There are three types of operators to be predicted, namely LO operator, TM operator and PM operator. Therefore, three-in-one operator modeling is adopted, that is, when building a reinforcement learning model, in the strategy network, the same strategy network is built for different types of operators, without the need to build corresponding strategy networks for different operators. In the traditional logic synthesis optimization process, the operator execution order follows the fixed optimization process of LO→TM→PM, while in the three-in-one modeling of the embodiment of the present application, the next operator predicted may be any of the three types of operators, thus achieving a relatively free mixed order between operators, that is, in the process of logic synthesis, the three-stage operators of LO, TM and PM are allowed to appear mixed, rather than limited to the fixed sequence of LO→TM→PM.

[0169] like Fig.12 As shown, k t That is, the next operator predicted (the tth operator), x t k t The corresponding tth group of parameters, operator k t and x t That constitutes the next action a t , the next operator can be one of LO, TM or PM.

[0170] like Fig.12 As shown, in this embodiment, the environmental state is mixedly represented:

[0171] In the application scenario where the execution object is a logic circuit, the current environment state at least includes the current state of the logic circuit. Taking the translated logic circuit as an AIG circuit as an example, this embodiment uses three parts to hybridly represent the environment state, namely:

[0172] ① The state vector (States) of the logic circuit in its current state after the last action is executed;

[0173] ②The original logic graph structure of AIG circuit (Raw graph);

[0174] ③ Historical action sequence (k0 x0)…(k t-1 x t-1 ).

[0175] Among them, ① the state vector of the logic circuit in the current state characterizes the intermediate state of the logic circuit. In the related art, such as related art 2, the intermediate state information of the netlist is not used, and the sample utilization rate is low. The embodiment of the present application introduces the intermediate state of the circuit into the training process to improve the sample utilization rate. A change in the environmental state can be understood as obtaining a new sample.

[0176] In this embodiment, the logic circuit may be an AIG circuit, or a LUT circuit, or may be an intermediate circuit of other formats that may appear in the logic synthesis stage.

[0177] The state vector of the logic circuit in its current state is equivalent to a set of metrics that describe the current circuit design. It is an initial feature vector that can contain the following values:

[0178] (1) The number of nodes, logical depth, and edges of the AIG circuit. The number of nodes can be the number of logic gates, such as the total number of AND gates and NOT gates. The logical depth is the number of edges or nodes that the longest path from input to output of the AIG circuit passes through according to the signal flow direction. For example, if the longest path of an AIG circuit passes through 100 edges and 101 nodes, the logical depth can be 100 or 101, but the same model needs to maintain a unified logical depth counting method. The number of edges is the sum of the number of edges of multiple paths from input to output in the AIG circuit.

[0179] (2) The number of LUTs with 6 address lines or less and the logical depth of the LUT in the LUT circuit obtained after the most recent TM operator is executed on the AIG circuit. Unlike AIG, the logical depth of the LUT can be the number of address lines n or 2 n .

[0180] (3) Historical action sequence (k0 x0)…(k t-1 x t-1 ). The operators in the historical action sequence are the operators that have actually been executed. The number of operators actually executed may not correspond to the currently predicted action number. For example, the next action to be predicted is the t-th action, but the number of operators in the historical action sequence may be t-4. That is, the length of the historical action sequence is based on the number of operators actually executed.

[0181] By combining the values ​​of (1)-(3) above, we can get the state vector (States). Then, states is input into the multi-layer perceptron MLP, which outputs the first feature vector (not shown in the figure). MLP includes an input layer, at least one hidden layer and an output layer.

[0182] The original logical graph structure (Raw graph) of the AIG circuit in ② can be input into the graph convolutional neural network GCN in the form of an adjacency matrix to obtain a second eigenvector. GCN includes an input layer, an output layer, and at least one convolution layer. The convolution layer is used to perform message passing based on the graph structure. In other embodiments, the input of GCN may include an adjacency matrix and initial features of each node, and node features are extracted through GCN.

[0183] ③ Historical action sequence (k0 x0)…(k t-1 x t-1 ) is input to the first encoder (TE). In this embodiment, the first encoder is a Transformer encoder, that is, a transformer model is used for encoding. Optionally, the historical action sequence can be input to TE after normalization. In the figure, previous actions represent historical actions. It can be understood that the operator and parameters are recorded as a normalized vector to form a historical action vector.

[0184] In this embodiment, the transformer includes an input layer, an encoder, a decoder and an output layer, wherein the encoder may include a self-attention layer (Self-Attention) and a feedforward neural network (Position-wise FeedForward Network, FNN); the decoder includes a self-attention layer, an attention layer (encoder-decoderattention) and an FNN.

[0185] It should be noted that the output layers of MLP, GCN and TE should control the output feature vectors to be of the same dimension. If the dimensions are different, the output can be transformed and then mixed operations such as addition can be performed.

[0186] Therefore, in the mixed representation process of the environment state, the multi-layer perceptron MLP, graph neural network GCN, and Transformer are used to encode the three parts of the environment state respectively, and finally integrated into a complete state feature vector h t .

[0187] In this embodiment, reinforcement learning adopts the Critic-Actor mechanism, where the Actor includes the Operator Actor (operator strategy network) and the Parameter Actor (parameter strategy network). t Input Operator Actor to get p k , p k The number of elements in corresponds to the number of candidate operators. For example, if the next operator predicted is one of the 12 operators, then p k That is, a 12-dimensional vector. t Input Parameter Actor and get μ x and σ x , where μ x represents the expected value of the normal distribution, σ x Represents the standard deviation of the normal distribution. μ x and σ x The number of elements in corresponds to the number of categories of operator parameters. For example, if there are 8 adjustable operator parameters, then μ x and σ x are all 8-dimensional vectors.

[0188] For example, the Operator Actor can be implemented based on the first MLP to perform the operations from h t to p k The Parameter Actor can also be implemented based on the second MLP to perform the mapping from h t to μ x and σ x The first MLP and the second MLP are both fully connected neural networks. The first MLP includes an input layer, an output layer, and at least one hidden layer; the second MLP includes an input layer, an output layer, and at least one hidden layer.

[0189] Next, four operators k are obtained by categorical distribution sampling and normal distribution sampling respectively. t and the tth group parameter x t . Fig.12 In the above figure, categorical distribution represents classification distribution and normal distribution represents normal distribution.

[0190] It should be noted that the solution proposed in the embodiment of the present application supports mixed sorting, but it is not completely unlimited free mixed sorting. In order to deal with the process reversal or inability to continue execution caused by mixed sorting between operators, the embodiment of the present application proposes the following control strategy to improve it:

[0191] When the LO operator is followed by the PM operator, the TM operator is inserted between them;

[0192] When the TM\PM operator is followed by the LO\TM operator, a fallback operator is inserted therein, for example, a Strash operator is inserted. The fallback operator refers to an operator capable of performing a fallback operation from a gate-level netlist to a logic circuit, and is not limited to Strash.

[0193] The above strategy can be added to the strategy network. After determining the type of the next operator, the control strategy can be used to determine whether to insert an operator and whether to insert a TM operator or a fallback operator.

[0194] For example, as shown in Figure 13(a), when it is determined that the next operator is a PM operator, it is necessary to query the operator in the previous action. If it is a LO type operator, a TM operator is inserted before the PM operator.

[0195] As shown in Figure 13(b), when it is determined that the next operator is the LO operator, it is necessary to query the operator in the previous action. If it is a TM or PM operator (not shown), it is necessary to insert a fallback operator before the LO operator.

[0196] As shown in FIG. 13( c ) and FIG. 13( d ), when it is determined that the next operator is a TM operator, it is necessary to query the operator in the previous action. If it is a TM or PM operator, a fallback operator needs to be inserted before the TM operator.

[0197] By modeling in this way, the operator sequence optimized by the present invention will repeatedly perform the mapping from the logic diagram to the gate-level netlist and the fallback operation, thereby greatly expanding the space for operator sequence optimization and further improving the sequence parameter optimization effect of the entire algorithm framework.

[0198] In this embodiment, an exploration budget model is introduced to drive the algorithm to always focus on those search spaces with more promising and better performance, rather than dwelling on those poor states. The budget exploration module dynamically adjusts the exploration budget of operator actions and parameter actions, which can improve sample efficiency and performance.

[0199] Specifically, in this embodiment, the exploration budget model includes a second encoder (TE) and an exploration budget network (Budget Tuner), and Budget Tuner represents budget exploration.

[0200] Collect the most recent historical action trajectories, such as a historical action trajectory <s0,k0,x0,r0;s1,k1,x1,r1;…;s L ,k L ,x L ,r L>, where s is the current environment state, k is the execution operator, x is the operator parameter, and r is the reward obtained by a single operator execution. In some embodiments, s can be the state vector (States) of the logic circuit in the current state; in other embodiments, s can be the current environment state h after mixed representation. t .

[0201] It should be noted that, in this embodiment, the trajectory contains reward values ​​and states, while the action sequence only contains operators and operator parameters, which should be distinguished.

[0202] Explore the budget model, input the historical action trajectory into the transformer, monitor the change of the operator Δk, the change of the parameter Δx and the change of the reward Δr, which can be a difference vector.

[0203] In this embodiment, the next action in the sequence is predicted through Transformer based on the known historical action trajectory. The difference operation is performed between the operator features in the predicted next action and the operator features in the last historical action in the current historical action trajectory to obtain the difference vector corresponding to the operator. The difference operation is only one feasible method. In other embodiments, the change between the tth operator and the t-1th operator can be determined based on other methods. The method for obtaining the changes corresponding to the operator parameters and reward values ​​is similar.

[0204] Then, the three changes Δk, Δx and Δr are used as the query (Query, Q), key (Key, K) and value (Value, V) of the exploration budget network Budget tuner, and the Budget tuner network is used to predict whether to increase or decrease the operator exploration budget b k and parameter exploration budget b x . Budget tuner also uses the attention mechanism.

[0205] For example, the Budget tuner can be used as follows: Fig.14 The attention mechanism shown in the figure, Mask (mask layer) can be Padding Mask or Sequence Mask; SoftMax (normalization layer) uses the softmax function to convert the attention score numerically; Scale is a scaling layer used to reduce variance; from bottom to top as shown in the figure, the first MatMul layer is used to perform scaled dot product multiplication, and the second MatMul layer is used to perform weighted product multiplication. Mask can only be used in Decoder.

[0206] Next, we use the distribution entropy loss of the categorical distribution and the normal distribution sampler to reflect the exploration of operators and parameters. Here, we introduce a novel exploration loss function, which is obtained by adjusting the coefficient b of the loss function. k and b x To determine your exploration budget:

[0207]

[0208] in is the entropy of the categorical distribution, is the entropy of the normal distribution.

[0209] Based on b k and b x ,as well as and A variety of loss function expressions can be obtained, and the above are only examples. In other embodiments, the above loss function can also be appropriately deformed to obtain other forms of loss functions, for example, Multiply by a weight coefficient γ1, Multiply by another weight coefficient γ2, then sum up, and so on. The specific form of the loss function is not unique, and the embodiments of the present application will not list them one by one.

[0210] After obtaining the loss function value, the trainable parameters in the policy network are optimized to adjust the policy towards a more promising prediction direction and reduce the exploration process of invalid operator sequences. exp After derivation, the parameters in the policy network are optimized using gradient descent or back propagation methods.

[0211] After testing, the method provided in the embodiment of the present application can achieve the following technical effects:

[0212] Using ABC as a logic synthesis tool, the method provided by the embodiment of the present application was implemented on the EPFL benchmark circuit (including 10 control circuits and 10 operation circuits). The synthesis result of the well-known sequence Resyn2 was used as the baseline, and the circuit area, that is, the number of LUT-6, was optimized while ensuring that the circuit level was not deteriorated. Fig.15 As shown, select Fig.15 The listed LO, TM, PM operators and their parameters are used to optimize the operator sequence with a length of 20. The length is not limited to 20, but 20 is used as an example here.

[0213] like Fig.16 As shown, under the same number of samples, compared with the baseline Resyn2, the BSBO algorithm improves the optimization effect by an average of 21.9% on 20 circuits. The method provided in the embodiment of the present application improves the optimization effect by an average of 25.4%, which is significantly better than the BSBO algorithm.

[0214] For the 20 circuits on the EPFL benchmark, the EPFL list records the hierarchical optimal and area optimal implementations of these circuits (based on FPGA LUT-6), totaling 40 circuits, which are used to compare the comprehensive optimization algorithms in related technologies.

[0215] like Fig.17 As shown, the circuit optimized by the method provided in the embodiment of the present application surpasses 26 of the 40 best implementations in the EPFL list in 2022. It is worth mentioning that the embodiment of the present application has achieved a significant improvement in 5 of the circuits compared with the best implementation in 2022, with an optimization ratio of up to 20+%.

[0216] In summary, the embodiments of the present application propose all-in-one modeling for logic synthesis, which allows seamless connection between different operators, effectively utilizes the operator space, expands the search space, and improves the optimization results; in addition, the embodiments of the present application also proposes a mixed representation of environmental states, by introducing the intermediate state of the circuit into the training process, and using circuit indicators, circuit diagram structure, and operator sequence to mix and represent the current state, thereby improving the optimization efficiency; in addition, the embodiments of the present application also proposes a budget exploration mechanism, which dynamically adjusts the exploration budget of operator actions and parameter actions, thereby significantly improving sample efficiency and performance.

[0217] It should be noted that the method provided in the embodiment of the present application is not limited to tuning the operator sequence in logic synthesis, but can also be easily applied to other sequence tuning problems, providing a general optimization method for the entire EDA process and even other broader fields.

[0218] The present application also provides a computer program product, which includes a program, and when the program is executed by an electronic device, the electronic device implements the method described in any of the above embodiments. For example, the program product may be an EDA application, or a plug-in that is compatible with an EDA application or other applications.

[0219] An embodiment of the present application also provides an electronic device, which includes: one or more processors; a memory; at least one application; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute a method as described in any of the above embodiments.

[0220] Specifically, the electronic device may be a server or server cluster that provides operator sequence optimization services. Fig.18 ,The hardware structure of an electronic device may include: a processor, a memory and a chipset.

[0221] Processor, the core that handles all computing tasks and requests, including at least the CPU.

[0222] Storage: includes memory and hard disk. The larger the capacity, the more requests and files can be processed simultaneously.

[0223] Chipset: includes northbridge chip and southbridge chip, which connect various I / O devices. Northbridge chip connects CPU, memory and PCI bus, while southbridge chip connects hard disk, network and other I / O devices.

[0224] I / O bus: connects various I / O devices, such as network cards, USB, etc. The larger the I / O bus bandwidth of the server, the more requests it can handle. I / O devices: including network cards, USB, etc. The network card is the main device that connects the server to the external network. Its speed and performance directly affect the response speed and transmission speed of the server.

[0225] Power supply: Provides stable power to the server to ensure normal operation of the server.

[0226] Chassis: The outer shell of the server hardware that protects the hardware from dust, vibration, and temperature.

[0227] Exemplarily, the processor may include one or more processing units, for example, the processor includes a neural-network processing unit (NPU), and may also include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a digital signal processor (DSP), a baseband processor, etc. Among them, different processing units may be independent devices or integrated into one or more processors. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0228] The memory can be used to store computer executable program codes, which include instructions. The internal memory may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one application required for a function, etc. The data storage area may store data (such as input data, output data) created during the use of the electronic device, etc. In addition, the internal memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory and / or instructions stored in a memory provided in the processor.

[0229] Alternatively, the electronic device may also be a terminal device with certain computing capabilities, such as a mobile phone, a personal computer (PC) (such as a laptop), a tablet computer (Tablet Personal Computer, Tablet PC), a wearable electronic device, etc. The terminal device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, etc.

[0230] It is to be understood that the structure illustrated in the embodiment of the present invention is only an example and does not constitute a limitation on the electronic device. The electronic device in the embodiment of the present application may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0231] An embodiment of the present application further provides a storage medium, including computer instructions, which, when executed on an electronic device, enables the electronic device to execute a method as described in any of the above embodiments.

[0232] An embodiment of the present application also provides a chip system, including a communication interface for inputting and / or outputting data; and a processor for executing a computer executable program so that a device equipped with the chip system executes a method as described in any of the above embodiments.

[0233] It should be noted that the solutions provided in the embodiments of the present application include any one or more of action prediction, training methods, program products, electronic devices and chip systems, or may also be other forms of software products, hardware products or virtual service products obtained based on the technical concepts of the methods described in the specification of this application.

[0234] In the description of the device in the embodiment of the present application, various modules for implementing the method flow functions are constructed in the device. The division of modules is merely a division of logical functions. When implementing the embodiment of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0235] Specifically, each functional module can be fully or partially integrated into one physical entity during actual implementation, or can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; or all can be implemented in the form of hardware; or some modules can be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the detection module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or instructions in the form of software.

[0236] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0237] Specifically, in one embodiment of the present application, the above-mentioned one or more computer programs are stored in the above-mentioned memory, and the above-mentioned one or more computer programs include instructions. When the above-mentioned instructions are executed by the above-mentioned device, the above-mentioned device executes the method steps described in the embodiment of the present application.

[0238] Specifically, in one embodiment of the present application, the processor of the electronic device may be a SOC device on a chip, which may include a central processing unit (CPU) and may further include other types of processors, such as a graphics processing unit (GPU). Specifically, in one embodiment of the present application, the processor of the electronic device may be a PWM control chip.

[0239] Specifically, in an embodiment of the present application, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units, NPU) and an image signal processor (Image Signal Processing, ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0240] Specifically, in one embodiment of the present application, the memory of the electronic device may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage devices, or may be any computer-readable medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer.

[0241] Specifically, in an embodiment of the present application, the processor and the memory can be combined into a processing device, more commonly, they are independent components, and the processor is used to execute the program code stored in the memory to implement the method described in the embodiment of the present application. In specific implementation, the memory can also be integrated into the processor, or independent of the processor.

[0242] Furthermore, the devices, apparatuses, and modules described in the embodiments of the present application may be implemented by computer chips or entities, or by products having certain functions.

[0243] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0244] In the several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0245] The embodiments described in this application are described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0246] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0247] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0248] It should also be noted that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0249] In the embodiments of the present application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0250] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0251] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0252] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0253] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0254] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A motion prediction method, characterized in that: The method is performed based on reinforcement learning, and the reinforcement learning includes a policy network; the execution object of the action is a logic circuit or other object; The method comprises: Determine the t-th state feature vector; wherein the t-th state feature vector is used to characterize the state of the environment after executing the t-1-th action; t is an integer variable and t≥1; Inputting the t-th state feature vector into the policy network and outputting at least a first parameter; Based on the first parameter, determine the tth operator and obtain the tth action; wherein the tth action is used to execute the tth operator on the execution object; the tth operator and the t-1th operator corresponding to the t-1th action are operators of the same type or operators of different types.

2. The method according to claim 1, characterized in that Inputting the t-th state feature vector into the policy network and outputting at least a first parameter includes: Inputting the t-th state feature vector into the strategy network, and outputting a first parameter and a second parameter; After outputting at least the first parameter, the method further comprises: Based on the second parameter, a tth group of parameters corresponding to the tth operator is determined; and the tth action is used to execute the tth operator on the execution object according to the tth group of parameters.

3. The method according to claim 2, characterized in that After obtaining the t-th action, the method further includes: An action sequence is obtained, where the action sequence includes information of an operator corresponding to at least one action; and the action sequence is a variable length sequence.

4. The method according to claim 2 or 3, characterized in that The method is applied to the logic synthesis stage in chip design; The action sequence includes information of at least two types of operators among a logic optimization operator, a mapping operator, or a post-mapping operator.

5. The method according to any one of claims 2 to 4, characterized in that: After determining the tth operator based on the first parameter, the method further includes: When the t-1th operator corresponding to the t-1th action is a logic optimization operator and the tth operator is a post-mapping operator, a mapping operator is inserted after the logic optimization operator and before the post-mapping operator.

6. The method according to any one of claims 2 to 5, characterized in that: After determining the tth operator based on the first parameter, the method further includes: When the t-1th operator corresponding to the t-1th action is a mapping operator or a post-mapping operator, and the tth operator is a logic optimization operator or a mapping operator, a fallback operator is inserted after the mapping operator or the post-mapping operator and before the logic optimization operator or the mapping operator.

7. The method according to any one of claims 1 to 6, characterized in that Determine the tth state feature vector, including: At least based on the state of the execution object after executing the t-1th action and the graph structure information corresponding to the execution object, the environment state is mixedly represented to obtain a tth state feature vector.

8. The method according to any one of claims 1 to 7, characterized in that Determine the tth state feature vector, including: Obtaining a t-th first feature vector based on the state of the execution object after executing the t-1-th action; Obtaining a second feature vector based on the graph structure information corresponding to the execution object; Based on the historical action sequence, a t-th third eigenvector is obtained; wherein the historical action sequence includes operator information and parameter information corresponding to at least one action executed before executing the t-th action; The tth state eigenvector is obtained based on the tth first eigenvector, the second eigenvector and the tth third eigenvector.

9. The method according to claim 8, characterized in that Based on the state of the execution object after executing the t-1th action, the tth first feature vector is obtained, including: After executing the t-1th action, determining the state vector of the execution object in the current state; Inputting the state vector into a preset multi-layer perceptron, and outputting a first feature vector through the multi-layer perceptron; Based on the graph structure information corresponding to the execution object, a second feature vector is obtained, including: Determine graph structure information corresponding to the execution object, input the graph structure information into a preset graph neural network, and output a second feature vector through the graph neural network; Based on the historical action sequence, the tth third eigenvector is obtained, including: The historical action sequence is input into a preset first encoder to obtain the tth third feature vector; wherein the first encoder is an encoder based on an attention mechanism.

10. The method according to claim 9, characterized in that The execution object is a logic circuit; Determining a state vector of the execution object in a current state includes: Determine a state vector of the execution object in a current state based on at least one of the first set of information, the second set of information, and the third set of information; The first set of information includes at least one of the number of nodes, the logic depth, and the number of edges of the logic circuit after executing the t-1th action; The second set of information includes at least one of the number of LUTs and the logic depth of the LUT in the lookup table LUT circuit; wherein the LUT circuit is a circuit including the LUT obtained by performing a TM operator on the logic circuit; The third set of information is the length of the historical action sequence.

11. The method according to any one of claims 1 to 10, characterized in that The policy network includes a first actor and a second actor; the first actor and the second actor both include trainable parameters; Inputting the t-th state feature vector into the policy network, and outputting a first parameter and a second parameter, including: Input the t-th state feature vector to the first actor, and output a first parameter; the first parameter is used to represent the probability distribution of each operator as the t-th operator; Input the t-th state feature vector to the second actor, and output a second parameter; the second parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter is the expected value of the normal distribution, and the second sub-parameter is the standard deviation of the normal distribution; Based on the first parameter, determining the tth operator includes: According to the first parameter, perform classification distribution sampling to determine the tth operator; Determining, based on the second parameter, a tth group of parameters corresponding to the tth operator, comprising: Normal distribution sampling is performed according to the first sub-parameter and the second sub-parameter to determine the tth group of parameters corresponding to the tth operator.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Inputting a historical action trajectory into a preset exploration budget model, and outputting a first coefficient and a second coefficient; wherein the historical action trajectory includes the environment state, operator information, parameter information, and reward function value corresponding to at least one action that has been executed before the t-th action; Determine a loss function value corresponding to the exploration budget model based on the first coefficient and the second coefficient; According to the loss function value, the trainable parameters in the policy network are optimized.

13. The method according to claim 12, characterized in that The exploration budget model includes a second encoder based on an attention mechanism and an exploration budget network; Input the historical action trajectory to the preset exploration budget model, and output the first coefficient and the second coefficient, including: Input the historical action trajectory to the second encoder, and output the first difference vector to the third difference vector of the t-th action relative to the t-1-th action; the first difference vector is used to represent the difference information of the t-th operator relative to the t-1-th operator; the second difference vector is used to represent the difference information of the t-th group of parameters relative to the t-1-th group of parameters; the third difference vector is used to represent the difference information of the reward function value fed back by the environment for the t-th action relative to the reward function value fed back by the environment for the t-1-th action; The difference vector is input into the exploration budget network, and the first coefficient and the second coefficient are output.

14. The method according to claim 12 or 13, characterized in that Determining a loss function value corresponding to the exploration budget model based on the first coefficient and the second coefficient includes: Determining the entropy of the normal distribution corresponding to the t-th state eigenvector; Determining the entropy of the classification distribution corresponding to the t-th state feature vector; Based on the first coefficient, the second coefficient, the entropy of the normal distribution, and the entropy of the categorical distribution, a loss function value corresponding to the exploration budget model is obtained.

15. A training method, characterized in that: The method is performed based on reinforcement learning, wherein the reinforcement learning includes a policy network and a value network; the value network includes a value function, and the policy network includes trainable parameters; The method comprises: Determine the t-th state feature vector; wherein the t-th state feature vector is used to characterize the state of the environment after executing the t-1-th action; t is an integer variable and t≥1; Optimizing the trainable parameters in the policy network at least according to the value of the value function; wherein the value of the value function is the output obtained by inputting the t-th state feature vector into the value network; Inputting the t-th state feature vector into the policy network and outputting at least a first parameter; Based on the first parameter, determine the tth operator and obtain the tth action; wherein the tth action is used to execute the tth operator on the execution object; the tth operator and the t-1th operator corresponding to the t-1th action are operators of the same type or operators of different types, and the execution object of the action is a logic circuit or other object.

16. The method according to claim 15, characterized in that The method further comprises: Inputting a historical action trajectory into a preset exploration budget model, and outputting a first coefficient and a second coefficient; wherein the historical action trajectory includes the environment state, operator information, parameter information, and reward function value corresponding to at least one action that has been executed before the t-th action; Determine a loss function value corresponding to the exploration budget model based on the first coefficient and the second coefficient; According to the loss function value, the trainable parameters in the policy network are optimized.

17. The method according to claim 15 or 16, characterized in that Determining a loss function value corresponding to the exploration budget model based on the first coefficient and the second coefficient includes: Determining the entropy of the classification distribution obeyed by the t-th operator; Determining the entropy of the normal distribution obeyed by the tth group of parameters; Based on the first coefficient, the second coefficient, the entropy of the normal distribution, and the entropy of the categorical distribution, a loss function value corresponding to the exploration budget model is obtained.

18. A computer program product, characterized in that The program product includes a program, and when the program is executed by an electronic device, the electronic device implements the method according to any one of claims 1 to 14 or 15 to 17.

19. An electronic device, characterized in that: The electronic device comprises: One or more processors; a memory; at least one application; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform a method as described in any one of claims 1-14 or 15-17.

20. A storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1-14 or 15-17.

21. A chip system, characterized in that: include: A communication interface for inputting and / or outputting data; A processor, used to execute a computer executable program so that a device equipped with the chip system executes a method as described in any one of claims 1-14 or 15-17.

Images