Multi-stage feature selection for training of machine learning (ML) models for electronic design automation (EDA) design flows

Abstract

A method may be performed by a computing system, including steps of accessing an input dataset for an electronic design automation (EDA) design flow and performing a multi-stage feature selection process for the input dataset to determine a feature set to train a machine learning (ML) model for the EDA design flow. Performing the multi-stage feature selection process may include performing a single feature ranking process on the input dataset to obtain a ranked list of the candidate features for the input dataset, performing a multi-feature selection process that sequentially considers the candidate features in an order specified by the ranked list, and determining the feature set to train the ML model for the EDA flow through the multi-feature selection process. The method may also include a step of training the ML model for the EDA design flow through the determined feature set.

Description

MULTI-STAGE FEATURE SELECTION FOR TRAINING OF MACHINE LEARNING (ML) MODELS FOR ELECTRONIC DESIGN AUTOMATION (EDA) DESIGN FLOWSBACKGROUND

[0001] Electronic circuits, such as integrated circuits, are used in nearly every facet of modern society, from automobiles to microwaves to personal computers. Design of circuits may involve many steps, known as a "design flow." The particular steps of a design flow are often dependent upon the type of microcircuit being designed, its complexity, the design team, and the circuit fabricator or foundry that will manufacture the circuit. Electronic design automation (EDA) applications support the design and verification of circuits prior to fabrication. EDA applications may implement various EDA procedures, e.g., functions, tools, or features to analyze, test, or verify a circuit design at various stages of the design flow.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Certain examples are described in the following detailed description and in reference to the drawings.

[0003] Figure 1 shows an example of a computing system that supports multi-stage feature selection for training of machine-learning (ML) models for electronic design automation (EDA) design flows.

[0004] Figure 2 shows an example determination of a feature set through a multistage feature selection process according to the present disclosure.

[0005] Figure 3 shows an example of logic that a computing system may implement to support performance of a multi-feature selection process according to the present disclosure.

[0006] Figure 4 shows an example of logic that a computing system may implement to support multi-stage feature selections for training of ML models for EDA design flows.

[0007] Figure 5 shows an example of a computing system that supports multi-stage feature selections for training of ML models for EDA design flows.DETAILED DESCRIPTION

[0008] Electronic circuits, such as integrated circuits (ICs), are used in nearly every facet of modem society, from automobiles to microwaves to personal computers. The design, verification, physical manufacture, and analysis of circuit devices often involve several steps, sometimes referred to as a "design flow" or “EDA design flow.” As used herein, an EDA design flow may include any steps taken in the design, manufacture, or analysis of circuits. The particular steps of a design flow are dependent upon various factors, such as the type of integrated circuit being designed, its complexity, the design team, and the integrated circuit fabricator (e.g., foundry) that will manufacture the physical circuit. Typically, software and hardware tools can verify the circuit designs at various stages of the design flow, for example through complex rule checks, software-based simulations, hardware-based emulation, and various other techniques supported by modem EDA technology. These steps of a design flow aid in the discovery of errors in circuit designs, and allow design teams and engineers to correct or otherwise improve the designs prior to, during, or after physical manufacture.

[0009] Several steps are common to most design flows of IC design. Initially, the specification for a new circuit can be transformed into or otherwise generated as a logical design. Logical designs are sometimes referred to as a register transfer level (RTL) description of a circuit. With logical designs, a circuit can be described in terms of both the exchange of signals between hardware registers and the logical operations that are performed on those signals. The logical design typically employs a Hardware Design Language (HDL), such as the Very high-speed integrated circuit Hardware Design Language (VHDL). The logic of the circuit is then analyzed to confirm that the design will accurately perform the functions desired for the circuit. This analysis is sometimes referred to as "functional verification."

[0010] After the accuracy of the logical design is confirmed through functional verification, a logical design can be converted into a device design by synthesis software. The device design, which is typically in the form of a schematic or netlist, can describe the specific electronic devices (e.g., transistors, resistors, and capacitors) that form the circuit design, along with the interconnections between these electronic devices. This device design generally corresponds to the level of representation displayed in conventional circuit diagrams. The relationships between the electronic devices are then analyzed to confirm that the circuit described by the device design will correctly perform the desired functions. This analysis is sometimes referred to as "formal verification." Additionally, preliminary timing estimates for portions of the circuit are often made at this stage, using an assumed characteristic speed for each device, and incorporated into the verification process.

[0011] Once the electronic devices components and their interconnections are established, the design can again be transformed in a design flow. In particular, the next transformation may be to a physical design that describes specific geometric elements that form the circuit design. This type of physical version of a circuit design is often referred to as a "layout" design or “physical layout” (and may simply be referred to as a “layout”). The geometric elements, which typically are polygons, define the shapes that will be created in various layers of material to physically manufacture the circuit. Automated place and route tools can be used to define or generate the physical layouts, especially for wires that will be used to interconnect the circuit devices in the physical representation of the circuit design. Each layer of a circuit can have a corresponding layer representation in the layout design, and the geometric shapes described in a layer representation will define the relative locations of the circuit elements that will make up the circuit device (e.g., of transistors, resistors, capacitors, etc.). For example, shapes in the layer representation of a metal layer will define the locations of the metal wires used to connect the circuit devices.

[0012] Integrated circuit layout descriptions can be provided in many different formats. The Graphic Data System II (GDSII) format is a popular format for transferring and archiving two-dimensional graphical IC layout data. Among other features, GDSII contains a hierarchy of structures, each structure containing layout elements (e.g., polygons, paths or poly-lines, circles and textboxes). Other layout formats include an open-source format named Open Access, Milkyway by Synopsys,Inc., EDDM by Siemens EDA (formerly Mentor Graphics Corporation), and the Open Artwork System Interchange Standard (OASIS) format proposed by Semiconductor Equipment and Materials International (SEMI). These various industry formats are used to define the geometrical information in IC layout designs that are employed to manufacture integrated circuits. Once the circuit design is finalized, the layout portion of the design can be used by fabrication tools to manufacture the device using a photolithographic process.

[0013] Typically, a designer will perform a number of verification processes on the layout design. For example, the layout design may be analyzed to confirm that it accurately represents the circuit devices and their relationships described in the device design. In this process, a layout-versus-schematic (LVS) tool can extract a netlist from the layout design and compare it with the netlist taken from the circuit schematic. LVS can be augmented by formal equivalence checking, which checks whether two circuits perform exactly the same function without demanding isomorphism.

[0014] The layout design also may be analyzed to confirm that it complies with various design requirements, such as minimum spacings between geometric elements and minimum linewidths of geometric elements. Such checks may be part of a design rule checking (DRC) process performed on layout design. DRC tools can take, as an input, a physical layout (e.g., in the GDSII or OASIS standard format) as well as a rule deck which specifies the specific rule checks to perform on the layout design. As checks in a DRC process can be specific to a particular circuit fabrication process, rule decks are typically provided by a foundry or circuit manufacturer specifying the particular rules that circuit designs must adhere to for circuit fabrication via the foundry (e.g., at a specified technology node or specific fabrication process parameters). Put another way, foundry-provided rule decks can include a list of rules specific to the semiconductor fabrication process employed by the foundry or otherwise selected for use in circuit manufacture. As such, a set of rules for a particular fabrication process can be referred to as a run-set, rule deck, or just a deck. An example format used for implementation of rule decks is the Standard Verification Rule Format (SVRF) by Siemens EDA (formerly Mentor Graphics Corporation).

[0015] There are many different fabrication processes for manufacturing a circuit, but most processes include a series of steps that deposit layers of different materials on a substrate, expose specific portions of each layer to radiation, and then etch theexposed (or non-exposed) portions of the layer away. For example, a simple semiconductor device component could be manufactured by the following steps. First, a positive-type epitaxial layer is grown on a silicon substrate through chemical vapor deposition. Next, a nitride layer is deposited over the epitaxial layer. Then specific areas of the nitride layer are exposed to radiation, and the exposed areas are etched away, leaving behind exposed areas on the epitaxial layer, (i.e., areas no longer covered by the nitride layer). The exposed areas then are subjected to a diffusion or ion implantation process, causing dopants, for example phosphorus, to enter the exposed epitaxial layer and form charged wells. This process of depositing layers of material on the substrate or subsequent material layers, and then exposing specific patterns to radiation, etching, and dopants or other diffusion materials, is repeated a number of times, allowing the different physical layers of the circuit to be manufactured.

[0016] Each time that a layer of material is exposed to radiation, a photomask (mask) must be created to expose only the desired areas to the radiation, and to protect the other areas from exposure. The mask is created from circuit layout data. That is, the geometric elements described in a physical layout define the relative locations or areas of the circuit wafer that will be exposed to radiation through the mask. A mask or reticle writing tool is used to create the mask based upon the design layout, after which the mask can be used in a photolithographic process for fabrication of physical circuits. One or more resolution enhancement techniques (RETs) are often employed to improve the resolution of the image that the mask forms on the substrate during the photolithographic process. One of these techniques is optical proximity correction (OPC). OPC can be rule-based, model-based, or both. In rule-based OPC, the proximity effects are characterized, and specific solutions are devised for specific geometric configurations. The layout design is then searched using a DRC tool or a geometric-based software engine to find these geometric configurations. Once they are found, the specific solutions are applied. After physical manufacture, an EDA design flow may include analysis of fabricated circuits, for example for hotspot detections, root cause analyses, and other circuit analysis processes that can be used to detect fabrication issues and improve manufacturing yields. Through various steps of a design flow, the design, manufacture, and fabrication of circuits can be performed and supported through EDA technology.

[0017] While various steps of a design flow are described herein, circuit manufacture processes continue to evolve and may include any additional or alternative flow steps. Moreover, the intricacy of each step in a design flow is immense, especially as circuit designs continue to increase in complexity and the transistors and other devices that form a circuit are merely a few atoms wide. As such, accurate and effective design flow steps may increase the efficiency of circuit design and improvements at any given step in the design flow can yield significant benefits.

[0018] With advances in modem technology, machine learning has become increasingly prevalent in various steps of EDA design flows. As used herein, ML models for EDA design flows may refer to any model that is trained or used to support any aspect of circuit design, verification, manufacture, or analysis. Examples of ML models for EDA design flows include ML models in model-based OPC processes, such as OPC models (e.g., etch models or resist models) that can be configured to predict behaviors, values, or characteristics of lithography or etch processes. As other examples, ML models for EDA design flows can be used to down-sample datasets, especially as datasets used in EDA design flows can contain immense amounts of circuit data. For example, clustering ML models can provide capabilities to effectively down select candidate hotspot locations determined through low precision imaging processes, which may result in increased defect confirmation rates for subsequent high precision imaging process verifications. ML models for EDA design flows may also provide predictive capabilities with limit datasets, e.g., defect detection and root causes analyses early in the lifecycle of a technology node with low number of manufactured circuits and measured values thereof. While some examples have been provided, ML models can be used for any aspect or step in EDA design flows.

[0019] One challenge in the use of ML models for EDA design flows is feature selection. As used herein, a feature may refer to any measurable or distinct property. Features may thus take the form of circuit properties (e.g., layout or geometry characteristics), location data, quantified design aspects, and such. In some instances, features may take the form of computations or processes applied to input data or circuit data. An example of such features can be in the form of convolution kernels (and their corresponding kernel parameters) used in etch or resists models for OPC processes. Such kernel features may be expressed as the kernel itself or a property of the kernel, e.g., kernel density computed for convolving various circuitlocations of a circuit design with the kernel. For datasets extracted from circuit designs or manufactured circuits, as well as datasets otherwise relevant to EDA design flows, the number of features applicable to such datasets can be immense.

[0020] Selecting a feature set by which to train ML models for EDA design flows can significantly impact the performance, accuracy, reliability, and efficiency of such ML models. Use of inappropriate feature inputs for training ML models for EDA design flows may adversely affect the model build and its prediction capabilities. Overfitting can result from training ML models with irrelevant features unrelated to the particular value, characteristic, or property ML models are configured to predict or output. Feature overfitting can be particularly severe if the data volume of a training dataset is limited, which may be case in various semiconductor contexts in which ML models for EDA design flows are used. Additionally or alternatively, if a critical feature is not provided for training, the ML model may fail to accurately generate a predictive output and accuracy limitations may arise.

[0021] Another benefit of effective feature set selection is the ability to reduce data volume requirements and improve performance. If the feature set used to train a ML model includes critical features that impact output accuracy without other extraneous or irrelevant features, the amount of data used to train and subsequently use the ML model can be reduced significantly. For EDA processes that can, at times, involve immense amounts of circuit data, proper feature selection can yield significant computational latency improvements. Lack of critical features selected for training can also impact root cause analyses or other downstream applications. Unsupervised and semi-supervised machine learning methods may be more susceptible to training through an improper feature set, further enforcing a need for effective feature selection. Supervised machined learning techniques may require large volumes of training data in order to identify and remove irrelevant features. Effective feature selection can result in improvements of ML models for EDA design flows and overcome various limitations of unsupervised, semi-supervised, and supervised ML techniques.

[0022] Conventional feature selection techniques exist for training of ML models. For many OPC models (e.g., etch models and resists models), manual feature selection is performed based on best practices and user experience. Such manual selections can be prone to error or personal bias. Other conventional feature selection methodsinclude filter methods and wrapper methods. Filter feature selection methods can be supported via statistical measures or using prediction rules to evaluate individual features of a dataset for ML model training. Such methods are typically limited to univariate feature selection and can thus fail to account for feature interactions, ranking redundant features highly, and produce final feature sets that are sensitive to ranking. As such, single feature ranking methods (by themselves) can provide sub- optimal results for feature selections. Some advanced single feature ranking methods can be used, such as Shapley analysis which is based on game theory. Wrapper feature selection methods can support multi-feature analyses. Examples of such features selection processes include forward selection, backwards elimination, and exhaustive search algorithms. However, these processes may suffer from performance issues. While such multi-feature processes can compare multiple different features during selection processes, each may be computationally expensive, with run times on the order of O(N2), O(2N), or higher. For EDA datasets with a large number of features, the runtime limitations of conventional feature selection processes may be computationally infeasible or prohibitively expensive.

[0023] The disclosure herein may provide systems, methods, devices, and logic for multi-stage feature selections for training of ML models for EDA design flows. As described herein, the multi-stage feature selection technology of the present disclosure may provide capabilities to select features for training ML models for EDA design flow from input datasets, doing so through multi-stage feature selection processes. The multi-stage feature selection processes of the present disclosure may include a single feature ranking process that generates a ranked list of candidate features and a multi-feature selection process that sequentially considers the candidate features in an order specified by the ranked list. The single feature ranking process may provide individual feature considerations, and the ranked list may guide a sequential order for the multi-feature selection process.

[0024] Instead of comparing every candidate feature with every other candidate feature (which result in high computational requirements and latency), the multifeature selection process of the present disclosure may sequentially consider individual candidate features in a sequence specified by the ranked list. Such a multifeature selection process can improve performance, potentially significantly so by reducing run times to an order of O(N) instead of O(N2), O(2N), or higher as withconventional methods. The multi-stage feature processes of the present disclosure can detect and remove irrelevant or redundant features and may reduce feature sensitivity limitations in single feature ranking techniques. Thus, the multi-stage feature selection technology of the present disclosure can provide feature selections for training of ML models for EDA design flows with improved efficiency, accuracy, and performance as compared to conventional techniques.

[0025] These and other aspects of the multi-stage feature selection technology according to the present disclosure as well as various technical benefits are described in greater detail herein.

[0026] Figure 1 shows an example of a computing system 100 that supports multistage feature selections for training of ML models for EDA design flows. The computing system 100 may take the form of a single or multiple computing devices such as application servers, compute nodes, desktop or laptop computers, smart phones or other mobile devices, tablet devices, embedded controllers, and more. In some implementations, the computing system 100 hosts, instantiates, executes, supports, or implements an EDA application that supports circuit design and analysis, and may accordingly provide or implement any of the multi-stage feature selection technology described herein.

[0027] As an example implementation to support any combination of the multi-stage feature selection technology described herein, the computing system 100 shown in Figure 1 includes a multi-stage feature selection engine 110. The computing system 100 may implement the multi-stage feature selection engine 110 (including components thereof) in various ways, for example as hardware and programming. The programming for the multi-stage feature selection engine 110 may take the form of processor-executable instructions stored on a non-transitory machine-readable storage medium and the hardware for the multi-stage feature selection engine 110 may include a processor to execute those instructions. A processor may take the form of single processor or multi-processor systems, and in some examples, the computing system 100 implements multiple engines using the same computing system features or hardware components (e.g., a common processor or a common storage medium).

[0028] In operation, the multi-stage feature selection engine 110 may access an input dataset for an EDA design flow and perform a multi-stage feature selection process for the input dataset to determine a feature set to train an ML model for theEDA design flow. The multi-stage feature selection engine 110 may do so by performing a single feature ranking process on the input dataset to obtain a ranked list of the candidate features for the input dataset, performing a multi-feature selection process that sequentially considers the candidate features in an order specified by the ranked list, and determining the feature set to train the ML model for the EDA flow through the multi-feature selection process, doing so in any of the ways described herein. In operation, the multi-stage feature selection engine 110 may also train the ML model for the EDA design flow through the determined feature set.

[0029] These and other aspects of multi-stage feature selection technology according to the present disclosure are described in greater detail next.

[0030] Figure 2 shows an example determination of a feature set through a multistage feature selection process according to the present disclosure. The example technical aspects of Figure 2 are described through the multi-stage feature selection engine 110 as an illustrative implementation. In the example of Figure 2, the multistage feature selection engine 110 may access an input dataset 210. The multi-stage feature selection engine 110 may access the input dataset 210 in any number of ways. For example, the multi-stage feature selection engine 110 may load the input dataset 210 from a memory, receive the input dataset 210 over a network connection, or obtain the input dataset 210 via user input. Any suitable mechanism by which the multi-stage feature selection engine 110 can read or load a dataset may be utilized to access the input dataset 210.

[0031] The input dataset 210 may take the form of any collection of data related to, in support of, or used in an EDA design flow. In particular, the input dataset 210 may be specific to a type of ML model for an EDA design flow that the multi-stage feature selection engine 110 is to determine a feature set for or otherwise train. Example input datasets that the multi-stage feature selection engine 110 may access include physical layout, layer gauge data, circuit data for circuit locations that surround points-of- interest (e.g., 10-15 nanometer windows or any other configurable window size), or other circuit data. The input dataset 210 may include data relevant to OPC ML models (e.g., etch or lithography predictions), candidate defect locations for hotspot detection ML models, and more.

[0032] In some implementations, the input dataset 210 may include measured values (also referred to herein as measurement values) that a ML model is configured topredict or generate an output for. Such measurement values may act as label values for the input dataset 210, e.g., for training of an ML model through supervised learning techniques. For example, OPC input datasets for an etch ML model may include measurement values for etch values for various gauges / locations of a circuit design, such as etch bias measurements, critical dimension measurements, etc.

[0033] The input dataset 210 may include multiple features, and individual features in the input dataset 210 may take the form of any measurable property within the input dataset 210, e.g., circuit layout data, circuit characteristics at circuit locations (e.g., gauge locations), pitch values, geometry characteristics, hotspot location data, any values that can be computed as a function of circuit data (e.g., kernel density), and the like. As such, the number of features for an input dataset 210 may be immense. Selection of a specific set of features from the input dataset 210 through which to train a ML model for EDA design flows can be challenging. As described herein, the multistage feature selection engine 110 may intelligently perform a feature selection process for the input dataset 210 through a multi-stage feature selection process. According to the present disclosure, a multi-stage feature selection process may comprise two stages: (1 ) a single feature ranking process followed by (2) a multifeature selection process, each of which are described in turn.

[0034] In Figure 2, the multi-stage feature selection engine 110 performs a first stage of the multi-stage feature selection process through a single feature ranking process. The multi-stage feature selection engine 110 may perform the single feature ranking process on the input dataset 210 to obtain a ranked list 220 of candidate features. The ranked list 220 generated by the multi-stage feature selection engine 110 may be in the form of any listing than sequentially ranks features identified or extracted from the input dataset 210. In doing so, the multi-stage feature selection engine 110 may access, employ, implement, perform, or use any suitable univariate feature determination technology to perform the single feature ranking process, including via existing single feature ranking processes. Example methods, techniques, algorithms that the multi-stage feature selection engine 110 may employ or implement include Shapley value-based feature importance determinations, SHAP (SHapley Addition exPlanations) techniques, mutual information-based features selection, filter selection methods, or any other suitable single feature process that produces a feature ranking. The multi-stage feature selection engine 110 may implement any such technique,process, or single feature ranking technology to perform the single feature ranking process and generate the ranked list 220.

[0035] The features determined by the multi-stage feature selection engine 110 through the single feature ranking process may be referred to as candidate features. That is, the multi-stage feature selection engine 110 may further process the candidate features to determine a final feature set by which to train an ML model. This may be the case as a ranked list output from the first stage of the multi-stage feature selection process may be inadequate, limited, or deficient. As explained herein, single feature selection processes may face various limitations, including failing to account for multifeature interactions, generating high rankings for multiple redundant features, and producing final feature sets that are sensitive to ranking. The multi-stage feature selection engine 110 may further process the ranked list 220 generated through the first stage of the multi-stage feature selection process to address various weaknesses of single feature selection processes. In particular, the multi-stage feature selection engine 110 may perform a second stage of the multi-stage feature selection process via a multi-stage feature selection process.

[0036] In the example of Figure 2, the multi-stage feature selection engine 110 may use the ranked list 220 generated from the single feature selection process as an input to the multi-feature selection process. Through the multi-feature selection process, the multi-stage feature selection engine 110 may determine a feature set 230 from the ranked list 220. The feature set 230 may specify a specific set of features to use to train an ML model, such as the ML model 240 shown in Figure 2. In some implementations, the multi-stage feature selection engine 110 may train the ML model 240 via the determined feature set 230. Doing so may include preparing training data that comprises data specific to the determined feature set 230 and providing such training data to train the ML model 240. For example, the multi-stage feature selection engine 110 may prepare training data for the ML model 240 by filtering or processing the input dataset 210 with the determined feature set 230 and then training the ML model 240 with the prepared training data.

[0037] The multi-stage feature selection technology described herein may support the use, training, and validation for machine learning technologies of any type or implementation. In some implementations, the multi-stage feature selection engine 110 may itself implement or apply any suitable machine learning capability, algorithm,or technique to train or construct the ML model 240. In that regard, the multi-stage feature selection engine 110 may train ML models via any type of unsupervised, supervised, or semi-supervised machine learning techniques. Neural networks, monotonic machine learning models, cluster-based ML technologies, matrix models, reinforced learning models, or any other suitable machine learning implementations are contemplated herein, and the multi-stage feature selection engine 110 may construct or train any type of ML model accordingly and in support of any aspect of EDA design flows. In any suitable manner, the multi-stage feature selection engine 110 may train the ML model 240 with the determined feature set 230.

[0038] Returning to a second stage of the multi-stage feature selection process, various example technical aspects of the multi-feature selection process are described in further detail. The multi-stage feature selection engine 110 may perform the multifeature selection process by sequentially processing the candidate features in the ranked list 220 and in an order specified by the ranking of the ranked list 220. In performing the multi-feature selection process, the multi-stage feature selection engine 110 may maintain an interim feature set, which may refer to a set of candidate features that the multi-stage feature selection engine 110 has determined to be part of a feature set to use to train an ML model for an EDA design flow. The interim feature set may be referred to as “interim” during performing of the multi-feature selection process, as sequential consideration of candidate features according to the ranked list 220 may result in adding of additional candidate features to the interim feature set. The interim feature set may become final after consideration of the candidate features completes. After considering some (e.g., a threshold number) or all of the candidate features in the ranked list 220, the multi-stage feature selection engine 110 may complete performance of the multi-feature selection process by determining the final feature set by which to train a ML model as the candidate features included in the interim feature set.

[0039] To further illustrate, the multi-stage feature selection engine 110 may start at the top of the ranking (e.g., starting with the highest ranked candidate feature, also referred to as the rank 1 candidate feature) specified in ranked list 220. Then, the multi-stage feature selection engine 110 may sequentially consider each successive candidate feature (e.g., from rank 2, to rank 3, to rank 4, and so forth), and add a currently-considered candidate feature to the interim feature set if doing so wouldimprove training of the ML model with the interim feature set. That is, the multi-stage feature selection engine 110 may add (e.g., keep) a given candidate feature in the interim feature set responsive to a determination that adding the given candidate feature to the interim feature set improves an evaluation metric for training the ML model for the EDA design flow with the interim feature set.

[0040] These and various other example technical aspects of a multi-feature selection process according to the present disclosure are described in greater detail next with reference to Figure 3.

[0041] Figure 3 shows an example of logic that a computing system may implement to support performance of a multi-feature selection process according to the present disclosure. For example, the computing system 100 may implement the logic 300 as hardware, executable instructions stored on a machine-readable medium, or as a combination of both. The computing system 100 may implement the logic 300 via the multi-stage feature selection engine 110, through which the computing system 100 may perform or execute the logic 300 to perform a multi-feature selection process according to the present disclosure, e.g., as a second stage of a multi-stage feature selection process. The following description of the logic 300 is provided using the multi-stage feature selection engine 110 performing a multi-feature selection process for the ranked list 220 of candidate features to train the ML model 240 of Figure 2 as illustrative examples. The technical capabilities of the logic 300 may be consistently implemented for any ranked list, ML model, EDA design flow, or any other relevant context.

[0042] Through performing a multi-feature selection process, the multi-stage feature selection engine 110 may access the ranked list 220 of candidate features, for example as an input provided to the second stage of a multi-stage feature selection process as described herein. The ranked list 220 may be generated from a first step of the multi-stage feature selection process, e.g., via a single feature ranking process. In performing the multi-stage feature selection process, the multi-stage feature selection engine 110 may maintain an interim feature set and do so while sequentially considering the candidate features in an order specified by the ranked list 220 from the first stage of the multi-stage feature selection process. As part of maintaining the interim feature set, the multi-stage feature selection engine 110 may add an initial candidate feature from the ranked list to the interim feature set (302). The initialcandidate feature may be the highest ranked candidate feature in the ranked list 220, e.g., the rank 1 candidate feature specified in the ranked list 220.

[0043] The single feature ranking process performed by the multi-stage feature selection engine 110 may provide an importance or impact ranking of features (e.g., based on SHAP, mutual information, or any other suitable technique). The multi-stage feature selection engine 110 may set a highest-ranked candidate feature from the ranked list 220 as part of the interim feature set for the multi-feature selection process. The highest ranked candidate feature may indicate a high criticality or importance in terms of training the ML model 240 for the EDA design flow, and thus the multi-stage feature selection engine 110 may include the highest ranked candidate feature from the ranked list 220 as an initial candidate feature added to the interim feature set. By doing so, the multi-stage feature selection engine 110 may ensure that a critical or high importance feature is included in the determined feature set 230 to train the ML model 240.

[0044] The multi-stage feature selection engine 110 may then determine whether there are any remaining candidate features to consider in the ranked list (304). Such a determination may serve as an end criterion for consideration of candidate features in the ranked list 220 for the multi-feature selection process. As noted herein, the multi-stage feature selection engine 110 may perform the multi-feature selection process for some or all of the candidate features in the ranked list 220. Each consideration of a candidate feature in the ranked list 220 may be referred to as a consideration iteration of the multi-feature selection process. In some implementations, responsive to a determination that all of the candidate features in the ranked list 220 have been considered (and thus no candidate features remain to consider), the multi-stage feature selection engine 110 may cease the candidate feature considerations and perform no further consideration iterations. In some implementations, the multi-stage feature selection engine 110 may consider some, but not all, of the candidate features in the ranked list 220. For example, a threshold number parameter value may specify a fixed number (e.g., the top 25) candidate features of the ranked list to consider for a determined feature set to train an ML model for an EDA design flow. The threshold number may be predetermined, user-specified, or otherwise controlled by the multi-stage feature selection engine 110. After thethreshold number of candidate features have been considered, the multi-stage feature selection engine 110 may cease the candidate feature considerations.

[0045] If candidate features remain for consideration in the ranked list 220, the multistage feature selection engine 1 10 may perform a next consideration iteration. For a given consideration iteration, the multi-stage feature selection engine 110 may add a next candidate feature from the ranked list into the interim feature set (306). The multistage feature selection engine 110 may do so by identifying a next candidate feature in the ranking specified by the ranked list 220, e.g., the next sequence number in the ranked list 220 following that of the candidate feature considered in the previous consideration iteration. In each consideration iteration, the multi-stage feature selection engine 110 may assess whether adding the next candidate feature to the interim feature set would improve an evaluation metric for training the ML model 240 as compared to an evaluation metric for training the ML model 240 without the next candidate feature added to the interim feature set. Thus, in a given consideration iteration, the multi-stage feature selection engine 110 may perform an assessment for training the ML model 240 for the EDA design flow through the interim feature set with the next candidate feature added (308).

[0046] Any suitable evaluation metric may be implemented, determined, computed, or applied by the multi-stage feature selection engine 110 to assess training effectiveness of the ML model 240, including conventional model evaluation techniques. In that regard, the multi-stage feature selection engine 110 may implement any suitable or conventional model evaluation techniques, technologies, or processes to assess ML trainings and model accuracy. Example evaluation metrics and technology include F1 scores, precision, or recall values. In some implementations, the evaluation metric may be a root mean square (RMS) value, difference, or error between predicted values generated by the ML model 240 for the EDA design flow and measurement values for the EDA design flow. Measurement values may refer to physically-measured values or actual measured values for a dataset, and the difference between the predicted values of a trained ML model 240 and the actually-measured values for the input data may be the evaluation metric employed by the multi-stage feature selection engine 110 (e.g., expressed as RMS error / RSME). Measured values may be specified as part of a validation dataset, within the input dataset 210 itself, or combinations of both. Thus, such validation or inputdatasets may allow testing of the ML model 240 trained with different feature sets in order to assess whether the predictive capabilities or generated outputs of the ML model 240 have improved or not.

[0047] An example is provided for a given consideration iteration through an illustrative evaluation metric example of an RMS value (e.g., difference or RMS error) between model-predicted and actually-measured values. In this example, the multistage feature selection engine 110 may compare (1 ) the RMS value of training the ML model 240 with the interim feature set including the next candidate feature with (2) the RMS value of training the ML model 240 with an interim feature set without the next candidate feature added. If the RMS value of the ML training with an interim feature set that includes the next candidate feature is better than (e.g., lower RMS) than that of training the ML model 240 with an interim feature set that does not include the next candidate feature, then the multi-stage feature selection engine 110 may determine that adding this next candidate feature to the interim feature set would improve the performance, accuracy, or training of the ML model 240. In such cases, the multistage feature selection engine 110 may determine to add the next candidate feature to the interim feature set. However, if the RMS value of the ML training with an interim feature set that includes the next candidate feature is worse than (e.g., higher RMS) than that of training the ML model 240 with an interim feature set that does not include the next candidate feature, then the multi-stage feature selection engine 110 may determine that adding the next candidate feature to the interim feature set would worsen the accuracy of the ML model 240. In such cases, the multi-stage feature selection engine 110 may determine not to add this next candidate feature to the interim feature set, for example by discarding this next candidate feature from the interim feature set.

[0048] An example implementation of such is shown in the logic 300 in which the multi-stage feature selection engine 110 may determine whether or not an improved evaluation metric (e.g., improved RMS) is achieved (310) by adding the next candidate feature to the interim feature set. Responsive to a determination that the evaluation metric does not improve, the multi-stage feature selection engine 110 may discard the next candidate feature considered in this current consideration iteration, e.g., by removing the next candidate feature from the interim feature set (312) and thus discarding the next candidate feature. Responsive to a determination that theevaluation metric does improve (e.g., improved or lower RMS), the multi-stage feature selection engine 110 may add the next candidate feature considered as part of this current consideration iteration to the interim feature set, e.g., by keeping the next candidate feature in the interim feature set (314).

[0049] Keeping the next candidate feature in the interim set may simply involve proceeding to a next step in the logic 300, determining not to discard the next candidate feature of this current consideration iteration from the interim feature set, or by taking no action since the next candidate feature of this current consideration iteration was already previously added to the interim feature set prior to evaluation metric computations. Then, the multi-stage feature selection engine 110 may determine whether any candidate features remain in the ranked list 220 for consideration (304). If no candidate features remain for consideration (e.g., all or a threshold number of candidate features have been considered), then, the multi-stage feature selection engine 110 may finalize the feature set 230 for training the ML model 240, e.g., by determining the interim feature set as the feature set 230 to use to train the ML model 240 for the EDA design flow (316).

[0050] Through the consideration iterations described above, the multi-stage feature selection engine 110 may sequentially consider candidate features in an order specified by the ranked list 220. Some illustrative examples are provided to further explain various technical aspects of this second stage of the multi-stage feature selection process. In the consideration iteration immediately after adding the highest ranked candidate feature to the interim feature set, the multi-stage feature selection engine 110 may perform a consideration iteration for the second highest ranked candidate feature in the ranked list, also referred to herein as the rank 2 candidate feature. Thus, in this consideration iteration, the multi-stage feature selection engine 110 may add the rank 2 candidate feature to the interim feature set and perform an assessment for training the ML model 240 with an interim feature set including the rank 1 and rank 2 candidate features.

[0051] In this consideration iteration, the multi-stage feature selection engine 110 may determine whether the ML model 240 trained with an interim feature set consisting of the rank 1 and rank 2 candidate features has a better RMS value (as an illustrative evaluation metric) than the RMS value for the ML model 240 trained with an interim feature set consisting of only the rank 1 candidate feature. If not, the multi-stage feature selection engine 110 may discard (e.g., remove) the rank 2 candidate feature from the interim feature set. If so, the multi-stage feature selection engine 110 may add (e.g., keep) the rank 2 candidate feature in the interim feature set. And if so, after this example consideration iteration, the interim feature set may thus consist of the rank 1 and rank 2 candidate features. In this example, the multi-stage feature selection engine 110 may determine that the interim feature set comprised of the highest ranked candidate feature and a second highest ranked candidate feature has an improved evaluation metric for training the ML model for the EDA design flow than that of the interim feature set comprising only the highest ranked candidate feature (and thus without the second highest ranked candidate feature). Responsive to such a determination, the multi-stage feature selection engine 110 may add the second highest ranked candidate feature to the interim feature set, e.g., by keeping the second highest ranked candidate feature in the interim feature set (in the case the second highest ranked candidate feature was added in an earlier step of the consideration iteration).

[0052] Next, the multi-stage feature selection engine 110 may proceed to consider subsequent candidate features in an order specified by the ranked list 220, e.g., the rank 3 candidate feature in a next consideration iteration, then the rank 4 candidate feature in a following consideration iteration, and so forth. Note that for any given consideration iteration following that of the rank 2 candidate feature, the multi-stage feature selection engine 110 need not recompute the evaluation metric (e.g., RMS value) for the interim feature set without the next candidate feature added. This may be the case since the RMS value for the interim feature set without the next candidate feature added may have been necessarily computed in a previous consideration iteration.

[0053] For example, in the continuing example presented herein, the multi-stage feature selection engine 110 may perform a consideration iteration for a rank 3 candidate feature in which the previous consideration iteration added the rank 2 candidate feature to the interim feature set. Since the RMS value for training the ML model 240 with an interim feature set that includes the rank 2 candidate feature was computed in the previous consideration iteration, that RMS value can be used as a comparison in the current consideration iteration for the rank 3 candidate feature. Thus, for a given consideration iteration (aside from the first performed considerationiteration), the multi-stage feature selection engine 110 need only compute an evaluation metric for training the ML model 240 with an interim set that has the next candidate feature added, in this case, the rank 3 candidate feature. The multi-stage feature selection engine 110 may then compare the evaluation metric computed for the current consideration iteration with that from a previous consideration iteration. It can be understood that the compared evaluation metric from a previous consideration iteration will be the better of the two evaluation metrics compared in the previous iteration (e.g., lower RMS value from the evaluation metric comparison in the previous consideration iteration). As such, for each of the consideration iterations in the multifeature selection process (aside from that for the rank 2 candidate feature), the multistage feature selection engine 110 may need only compute the evaluation metric for training the ML model 240 with the next candidate feature added to the interim feature set.

[0054] In any manner consistent with those described herein, the multi-stage feature selection engine 110 may perform consideration iterations to sequentially consider the successive features of a ranked list with an interim feature set maintained for the multi-feature selection process. By sequentially comparing individual features with those in an interim feature set, the multi-stage feature selection engine 110 may ensure that only candidate features that improve training (e.g., accuracy) of an ML model for an EDA design flow (e.g., as measured through RMSE) are added to the interim feature set. Such a sequential consideration may yield performance improvements, as this second stage of the multi-stage feature selection process can be performed in O(N) time, where N is the number of features in the ranked list 220 or the threshold number of candidate features that are considered. By sequencing through candidate features in a ranked order, the multi-stage feature selection engine 110 may ensure that higher criticality or important individual features are considered earlier in the stage, and lower-criticality features are considered later. Doing so may ensure critical features are first added to the interim feature set and then later compared with lesser critical features for assessing ML model performance.

[0055] As another technical feature, the multi-feature selection process may support multi-feature analyses, e.g., between an interim feature set that can be comprised of multiple features and a next candidate feature. Doing so may allow the multi-stage feature selection engine 110 to detect and remove redundancies, e.g., for highlyranked features that may be inter-related, based on another, or functionally proximate to one another. In such cases, removal of redundant or overlapping features, including those which are ranked highly in the ranked list 220 may improve ML training and model performance. Consider an illustrative example in which feature “A” is the rank 1 candidate feature in a ranked list, feature “B” is the rank 2 candidate feature, and feature “X” is an irrelevant feature (e.g., rank 10,000). In this illustrative example, Feature “B” may be a function of Features “A” and “X”, e.g., B = 0.99 * A + 0.01 * X.

[0056] In this case, features “A” and “B” may be almost completely redundant, and the high ranking of feature “B” may be due to the contribution of Feature “A”. Univariate single feature ranking processes may rank features “A” and “B” highly, but a training data generated from a feature set that includes both features “A” and “B” may include redundant data for these two features. The multi-stage feature selection technology of the present disclosure may support removal of feature “B” in this illustrative example, as correlated features may fail to improve evaluation metrics (e.g., RMS error) and thus will be excluded from the interim feature set during performance of stage two of the multi-stage feature selection process. Furthermore, the introduction of an irrelevant feature X could degrade the evaluation metrics. Moreover, the multistage feature selection technology described herein may achieve such a redundance removal benefit in O(N) time, e.g., without having to compare feature “B” to all other features in the input dataset 210. Instead, for correlated or redundant features that are each highly ranked, the multi-stage feature selection engine 110 may detect and remove such redundancies relatively early in the multi-feature selection process and thus tangibly improve feature set determinations and ML model trainings.

[0057] Through any of the various aspects described herein, the multi-stage feature selection technology of the present disclosure may improve feature selections for the training of ML models for EDA design flows. ML models for EDA design flows trained with the feature sets determined through the multi-stage feature selection technology described herein may provide increased accuracy or efficiency, and the run-time for such feature selections may be improved as compared to conventional multi-feature selection techniques. The trained ML models for EDA design flows may be used for any suitable manner or application. Trained etch or resist models with feature sets determined through the multi-stage feature selection technology described herein may be used to predict etch values (e.g., bias or critical dimensions) or otherwise supportOPC processes. Any suitable application of the ML models trained accordingly is contemplated herein, as is the physical manufacture of circuits with circuit designs produced through EDA design flows that utilize any of the ML models described herein and trained with determined feature sets.

[0058] Figure 4 shows an example of logic 400 that a system may implement to support multi-stage feature selections for training of ML models for EDA design flows. For example, the computing system 100 may implement the logic 400 as hardware, executable instructions stored on a machine-readable medium, or as a combination of both. The computing system 100 may implement the logic 400 via the multi-stage feature selection engine 110, through which the computing system 100 may perform or execute the logic 400 as a method to support multi-stage feature selections according to the present disclosure. The following description of the logic 400 is provided using the multi-stage feature selection engine 110 as an example. However, other implementation options by computing systems are possible.

[0059] In implementing the logic 400, the multi-stage feature selection engine 110 may access an input dataset for an EDA design flow (402) and perform a multi-stage feature selection process for the input dataset to determine a feature set to train an ML model for the EDA design flow (404). The multi-stage feature selection engine 110 may do so by performing a single feature ranking process on the input dataset to obtain a ranked list of the candidate features for the input dataset (406), performing a multi-feature selection process that sequentially considers the candidate features in an order specified by the ranked list (408), and determining the feature set to train the ML model for the EDA flow through the multi-feature selection process (410), doing so in any of the ways described herein. In implementing the logic 400, the multi-stage feature selection engine 110 may also train the ML model for the EDA design flow through the determined feature set (412).

[0060] The logic 400 shown in Figure 4 provides an illustrative example by which a computing system 100 may support multi-stage feature selections for training of ML models for EDA design flows according to the present disclosure. Additional or alternative steps in the logic 400 are contemplated herein, including according to any of the various descriptions herein for the multi-stage feature selection engine 110.

[0061] Figure 5 shows an example of a computing system 500 that supports multistage feature selections for training of ML models for EDA design flows. Thecomputing system 500 may include a processor 510, which may take the form of a single or multiple processors. The processor(s) 510 may include a central processing unit (CPU), microprocessor, or any hardware device suitable for executing instructions stored on a machine-readable medium. The computing system 500 may include a machine-readable medium 520. The machine-readable medium 520 may take the form of any non-transitory electronic, magnetic, optical, or other physical storage device that stores executable instructions, such as the multi-stage feature selection instructions 522 shown in Figure 5. As such, the machine-readable medium 520 may be, for example, Random Access Memory (RAM) such as a dynamic RAM (DRAM), flash memory, spin-transfer torque memory, an Electrically-Erasable Programmable Read-Only Memory (EEPROM), a storage drive, an optical disk, and the like.

[0062] The computing system 500 may execute instructions stored on the machine- readable medium 520 through the processor 510. Executing the instructions (e.g., the multi-stage feature selection instructions 522) may cause the computing system 500 to perform or implement any of the multi-stage feature selection technology described herein, including according to any aspect of the multi-stage feature selection engine 110.

[0063] For example, execution of the multi-stage feature selection instructions 522 by the processor 510 may cause the computing system 500 to access an input dataset for an EDA design flow and perform a multi-stage feature selection process for the input dataset to determine a feature set to train an ML model for the EDA design flow. The multi-stage feature selection instructions 522 may cause the computing system 600 to do so by performing a single feature ranking process on the input dataset to obtain a ranked list of the candidate features for the input dataset, performing a multifeature selection process that sequentially considers the candidate features in an order specified by the ranked list, and determining the feature set to train the ML model for the EDA flow through the multi-feature selection process, doing so in any of the ways described herein. Execution of the multi-stage feature selection instructions 522 by the processor 510 may also cause the computing system 500 to train the ML model for the EDA design flow through the determined feature set.

[0064] Any additional or alternative aspects of the multi-stage feature selection technology as described herein may be implemented via the multi-stage feature selection instructions 522.

[0065] The systems, methods, devices, and logic described above, including the multi-stage feature selection engine 110, may be implemented in many different ways in many different combinations of hardware, logic, circuitry, and executable instructions stored on a machine-readable medium. For example, the multi-stage feature selection engine 110 may include circuitry in a controller, a microprocessor, or an application specific integrated circuit (ASIC), or may be implemented with discrete logic or components, or a combination of other types of analog or digital circuitry, combined on a single integrated circuit or distributed among multiple integrated circuits. A product, such as a computer program product, may include a storage medium and machine-readable instructions stored on the medium, which when executed in an endpoint, computer system, or other device, cause the device to perform operations according to any of the description above, including according to any features of the multi-stage feature selection engine 110.

[0066] The processing capability of the systems, devices, and engines described herein, including the multi-stage feature selection engine 110, may be distributed among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements. Parameters, databases, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be logically and physically organized in many different ways, and may be implemented in many ways, including data structures such as linked lists, hash tables, or implicit storage mechanisms. Programs may be parts (e.g., subroutines) of a single program, separate programs, distributed across several memories and processors, or implemented in many different ways, such as in a library (e.g., a shared library).

[0067] While various examples have been described above, many more implementations are possible.

Claims

CLAIMS1 . A method comprising: by a computing system: accessing an input dataset for an electronic design automation (EDA) design flow; performing a multi-stage feature selection process for the input dataset to determine a feature set to train a machine learning (ML) model for the EDA design flow, including by: performing a single feature ranking process on the input dataset to obtain a ranked list of the candidate features for the input dataset; performing a multi-feature selection process that sequentially considers the candidate features in an order specified by the ranked list; and determining the feature set to train the ML model for the EDA flow through the multi-feature selection process; and training the ML model for the EDA design flow through the determined feature set.

2. The method of claim 1 , wherein performing the multi-feature selection process that sequentially considers the candidate features in the order specified by the ranked list comprises: maintaining an interim feature set during sequential consideration of the candidate features in the order specified by the ranked list; and adding a given candidate feature to the interim feature set responsive to a determination that adding the given candidate feature to the interim feature set improves an evaluation metric for training the ML model for the EDA design flow with the interim feature set.

3. The method of claim 2, wherein performing the multi-feature selection process that sequentially considers the candidate features in the order specified by the ranked list comprises:adding a highest ranked candidate feature from the ranked list to the interim feature set; determining that the interim feature set comprised of the highest ranked candidate feature and a second highest ranked candidate feature has an improved evaluation metric for training the ML model for the EDA design flow than the interim feature set comprising only the highest ranked candidate feature; and wherein adding the given candidate feature to the interim feature set comprises adding the second highest ranked candidate feature to the interim feature set responsive to the determination that adding the second highest ranked candidate feature to the interim feature set improves the evaluation metric for training the ML model for the EDA design flow with the interim feature set.

4. The method of claims 2 or 3, wherein the evaluation metric comprises a root mean square (RMS) value between predicted values by the ML model for the EDA design flow and measurement values for the EDA design flow.

5. The method of any of claims 2-4, comprising determining the interim feature set as the feature set to train the ML model for the EDA flow after all of the candidate features in the ranked list have been considered.

6. The method of any of claims 2-4, comprising determining the interim feature set as the feature set to train the ML model for the EDA flow after a threshold number of the candidate features in the ranked list have been considered, wherein the threshold number is less than a total number of candidate features in the ranked list.

7. The method of claim 1 , wherein performing the multi-feature selection process comprises sequentially considering a threshold number of the candidate features in the ranked list for inclusion in the determined feature set, wherein the threshold number is less than a total number of candidate features in the ranked list.

8. A system comprising: a processor; anda non-transitory machine-readable medium comprising instructions that, when executed by the processor, causes a computing system to: access an input dataset for an electronic design automation (EDA) design flow; perform a multi-stage feature selection process for the input dataset to determine a feature set to train a machine learning (ML) model for the EDA design flow, including by: performing a single feature ranking process on the input dataset to obtain a ranked list of the candidate features for the input dataset; performing a multi-feature selection process that sequentially considers the candidate features in an order specified by the ranked list; and determining the feature set to train the ML model for the EDA flow through the multi-feature selection process; and train the ML model for the EDA design flow through the determined feature set.

9. The system of claim 8, wherein the instructions, when executed, cause the computing system to perform the multi-feature selection process that sequentially considers the candidate features in the order specified by the ranked list by: maintaining an interim feature set during sequential consideration of the candidate features in the order specified by the ranked list; and adding a given candidate feature to the interim feature set responsive to a determination that adding the given candidate feature to the interim feature set improves an evaluation metric for training the ML model for the EDA design flow with the interim feature set.

10. The system of claim 8, wherein the instructions, when executed, cause the computing system to perform the multi-feature selection process that sequentially considers the candidate features in the order specified by the ranked list by: adding a highest ranked candidate feature from the ranked list to the interim feature set;determining that the interim feature set comprised of the highest ranked candidate feature and a second highest ranked candidate feature has an improved evaluation metric for training the ML model for the EDA design flow than the interim feature set comprising only the highest ranked candidate feature; and wherein the instructions, when executed, cause the computing system to add the given candidate feature to the interim feature set by adding the second highest ranked candidate feature to the interim feature set responsive to the determination that adding the second highest ranked candidate feature to the interim feature set improves the evaluation metric for training the ML model for the EDA design flow with the interim feature set.11 . The system of claims 9 or 10, wherein the evaluation metric comprises a root mean square (RMS) value between predicted values by the ML model for the EDA design flow and measurement values for the EDA design flow.

12. The system of any of claims 9-11 , wherein the instructions, when executed, cause the computing system to determine the interim feature set as the feature set to train the ML model for the EDA flow after all of the candidate features in the ranked list have been considered.

13. The system of any of claims 9-11 , wherein the instructions, when executed, cause the computing system to determine the interim feature set as the feature set to train the ML model for the EDA flow after a threshold number of the candidate features in the ranked list have been considered, wherein the threshold number is less than a total number of candidate features in the ranked list.

14. The system of claim 8, wherein the instructions, when executed, cause the computing system to perform the multi-feature selection process comprises sequentially considering a threshold number of the candidate features in the ranked list for inclusion in the determined feature set, wherein the threshold number is less than a total number of candidate features in the ranked list.

15. A non-transitory machine-readable medium comprising instructions that, when executed by a processor, cause a computing system to perform a method according to any of claim 1-7.

Images