Generating integrated circuit plan using neural network

Abstract

The invention relates to generating an integrated circuit plan using a neural network. Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a computer chip plan are disclosed. One of the methods includes obtaining netlist data of a computer chip; and generating a computer chip plan, comprising placing a respective node at each time step in a sequence comprising a plurality of time steps, the placing comprising, for each time step: generating an input representation of the time step; processing the input representation using a node layout neural network having a plurality of network parameters, where the node layout neural network is configured to process the input representation in accordance with current values of the network parameters to generate a fractional distribution over a plurality of locations on a surface of the computer chip; and assigning the nodes to be placed at the time step length to one of the plurality of locations using the fractional distribution.

Description

Description of the case

[0001] This application is a divisional application of Chinese invention patent application No. 201980078596.8, with a filing date of December 4, 2019. Technical Field

[0002] The present disclosure relates to generating integrated circuit floor plans using neural networks. CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Patent Application No. 62 / 775,284 filed on December 4, 2018, the entire contents of which are incorporated by reference. Background Art

[0004] This specification relates to the use of neural networks for electronic design automation, and more specifically, for generating computer chip floor plans.

[0005] A computer chip floor plan is a schematic representation of the layout of some or all of the circuitry of a computer chip on the surface (i.e., the chip area) of the computer chip.

[0006] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters. Summary of the invention

[0007] This specification describes a system, implemented as a computer program on one or more computers in one or more locations, that generates a chip floor plan for an integrated circuit. The integrated circuit for which the chip floor plan is generated will be referred to in this specification as a "computer chip," but should generally be understood to mean any collection of electronic circuits fabricated on a piece of semiconductor material. The chip floor plan places each node in a node netlist at a corresponding location on the surface of the computer chip.

[0008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0009] Floorplan planning, which involves placing the components of a chip on the surface of the chip, is a key step in the chip design process. The layout of the components should optimize metrics such as area, total wire length, and congestion. If the floorplan performs poorly on these metrics, the computer chip generated based on the floorplan will perform poorly. For example, the computer chip may not operate, may consume too much power, may have unacceptable delays, or have any of a variety of other undesirable characteristics caused by the suboptimal layout of components on the chip.

[0010] The techniques allow for automatic generation of high-quality chip floor plans with minimal user involvement by utilizing the node placement neural network and the training techniques described. As a specific example, when distributed training is employed, high-quality (i.e., super-human) layouts can be generated in hours without any human expert involvement.

[0011] Unlike the described system, conventional floor plan planning solutions employ a weeks-long process that requires significant human involvement. Due to the vast space of potential node layout combinations, conventional automated approaches have been unable to reliably generate high-quality floor plans without consuming excessive computing power and wall-clock time, requiring human expert involvement, or both. However, by effectively utilizing reinforcement learning to train the described node-local neural network, the described technique is able to rapidly generate high-quality floor plans.

[0012] In addition, computer chips produced using the methods may have reduced power consumption compared to computer chips produced by conventional methods. The computer chips may also have increased computing power for a given surface area, or viewed from another perspective, may be produced using fewer resources for a given computing power.

[0013] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following detailed description. Other features, aspects, and advantages of the subject matter will become apparent from the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 An example floor plan generation system is shown.

[0015] Figure 2 The processing of the node layout neural network at the time step is illustrated.

[0016] Figure 3 is a flow chart of an example process for placing nodes at specific time steps.

[0017] Figure 4 is a flowchart of an example process for training a node layout neural network through reinforcement learning.

[0018] Like reference numbers and numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0019] Figure 1 An example floor plan generation system 100 is shown. Floor plan generation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0020] System 100 receives netlist data 102 for a computer chip (i.e., a very large scale integration (VLSI) chip) that is to be manufactured and includes a plurality of integrated circuit components, such as transistors, resistors, capacitors, etc. The plurality of integrated circuit components may be different depending on the desired function of the chip. For example, the chip may be a dedicated chip (i.e., an application specific integrated circuit (ASIC)) for machine learning calculations, video processing, encryption, or another computationally intensive function.

[0021] Netlist data 102 is data describing the connectivity of integrated circuit components of a computer chip. Specifically, netlist data 102 specifies connectivity on a computer chip between a plurality of nodes, each node corresponding to one or more of a plurality of integrated circuit components of a computer chip. That is, each node corresponds to a respective appropriate subset of integrated circuit components and these subsets do not overlap. In other words, for each of a plurality of nodes, netlist data 102 identifies which other nodes (if any) the node needs to be connected to through one or more wires in an already manufactured computer chip. In some cases, the integrated circuit components have been clustered in clusters, for example, by an external system or by using existing clustering techniques, and each node in the netlist data represents a different cluster in the cluster.

[0022] The system 100 generates as output a final computer chip floor plan 152 that places some or all of the nodes in the netlist data 102 at corresponding locations on the surface of the computer chip. That is, for some or all of the nodes in the netlist data 102, and therefore for the integrated circuit components represented by the nodes, the final computer chip floor plan 152 identifies corresponding locations on the surface of the computer chip.

[0023] As an example, netlist data 102 may identify three types of nodes: macros, clusters of standard cells, and ports. System 100 may determine the placement of macros and clusters of standard cells, while the port placement may be fixed or may be determined by another system based on the placement determined by system 100.

[0024] Generally, floor plan 152 assigns each node to a grid square in an N×M grid covering the surface of the chip, where N and M are integers provided as inputs to system 100 .

[0025] The system 100 includes a node placement neural network 110 , and optionally a simulated annealing engine 130 .

[0026] The system 100 uses the node placement neural network 110 to generate an initial computer chip floor plan 122. Similar to the final computer chip floor plan 152, the initial computer chip floor plan 122 also places each node in the netlist data 102 at a corresponding location on the surface of the computer chip.

[0027] The system 100 generates the initial chip floor plan 122 by placing corresponding nodes of the netlist data 102 at each time step in a sequence of multiple time steps. That is, the system 100 generates a layout node by node over multiple time steps, where each node is placed at a location at a different time step in the time steps.

[0028] At each specific time step in the sequence, the system generates an input representation for the specific time step and processes the input representation using the node layout neural network 110 .

[0029] The input representation for a particular time step typically includes (i) except for the first time step in the sequence, partially placed chip data representing a partially placed chip (i.e., a chip design in which a suitable subset of the nodes of the netlist have been placed), the partially placed chip data indicating the layout of the nodes that have been placed at the time step before the particular time step in the sequence, and (ii) current node data representing the node to be placed at the particular time step. In the first time step in the sequence, the partially placed chip data indicates that no nodes have been placed (i.e., representing an empty chip). The input representation may also optionally include (e.g., as part of the current node data) netlist features that represent the netlist of the node, such as representing the connectivity between the nodes specified in the netlist data 102. For example, for one or more of the nodes, the input representation may represent one or more other nodes in the nodes to which the node is connected according to the netlist. Specifically, the input representation may specify some or all nodes in the netlist to which the node to be placed at a particular time is connected according to the netlist, such as at least one of those nodes that have been placed at the time step before the particular time step.

[0030] The node layout neural network 110 is a neural network having parameters (referred to in this specification as "network parameters") and configured to process an input representation according to the current values ​​of the network parameters to generate a distribution of scores (e.g., a probability distribution or a logarithmic distribution) over a plurality of locations on the surface of a computer chip. For example, the distribution may be over grid squares in an N×M grid covering the surface of the chip.

[0031] The system 100 then assigns the node to be placed at a particular time step to one of a plurality of locations using the score distribution generated by the neural network.

[0032] References below Figure 2 and 3 The operations performed by the neural network 110 at a given time step and the placement of nodes at the time step using the score distribution are described in more detail.

[0033] By adding nodes to the floor plan one by one, after the last time step in the sequence, the floor plan will include the corresponding placements of all nodes in the netlist data 102 .

[0034] In order for the score distribution 124 generated by the neural network 110 to produce a high-quality floor plan, the system 100 trains the node layout neural network 110 through reinforcement learning to maximize the reward function.

[0035] As an example, the system 100 may train the neural network 110 through reinforcement learning to determine training values ​​for network parameters, and then after completing the training, generate the initial floor plan 122 by using the neural network 110 and generating the floor plan according to the training values ​​for the network parameters. The system may determine that the training has terminated based on any of a variety of termination criteria. For example, the system may terminate the training after a certain amount of time has passed, after a certain number of candidate floor plans have been evaluated, or after the parameter values ​​of the neural network 110 have converged.

[0036] As another example, the system 100 may track the reward values ​​assigned to the candidate floor plans generated during the training of the neural network 110 and then select the candidate floor plan with the highest reward value as the initial floor plan 122 after the training is completed.

[0037] Typically, the reward function measures the quality of the floor plan generated using the node placement neural network 110, i.e., measures some one or more features of the generated floor plan that, when optimized, result in a chip manufactured using the generated floor plan exhibiting good performance, for example, in one or more of power consumption, thermal and timing performance. Quality is typically an increasing function of these one or more features.

[0038] During training, the system 100 repeatedly generates candidate floorplans using the neural network 110, evaluates the value of the reward function of the generated candidate floorplans by evaluating features measured by the reward function, and adjusts the values ​​of the network parameters based on the value of the reward function.

[0039] References below Figure 4 Training the neural network 110 and reward functions that may be used for training are described in more detail.

[0040] In some embodiments, once the neural network 110 has been trained, the system 100 outputs the initial computer chip floor plan 122 as a final computer chip floor plan 152 .

[0041] In some other embodiments, the system 100 generates a final computer chip floor plan 152 by performing simulated annealing starting from an initial computer chip floor plan 122 generated using the node placement neural network 110 using a cost function (e.g., by performing simulated annealing on a reward function). As a specific example, the system 100 can perform simulated annealing by applying a hill climbing algorithm to iteratively adjust the layout in the initial computer chip floor plan 122 to generate the final computer chip floor plan 152. Hill climbing algorithms and other simulated annealing techniques that can be used to adjust the initial computer chip floor plan 122 are described in more detail in S. Kirkpatrick, CD Gelatt, and MP Vecchi., "Optimization by simulated annealing," Science, 220(4598): 671-680, 1983.

[0042] Optionally, the system 100 or an external system can then manufacture (produce) a chip (integrated circuit) based on one of the candidate plan views (e.g., the candidate plan view with the highest value of the reward function or the candidate plan view generated after the training is terminated). Such an integrated circuit can exhibit improved performance compared to an integrated circuit designed using a conventional design process, such as having one or more of lower power consumption, lower latency, or a smaller surface area, and / or can be produced using fewer resources. Manufacturing can use any known technology.

[0043] Figure 2 The processing of the node placement neural network 110 at a given time step is shown.

[0044] As mentioned above Figure 1 As described, at each time step during generation of the floor plan, the node layout neural network 110 is configured to receive an input representation and process the input representation to generate a score distribution, such as a probability distribution or a logarithmic distribution, over a plurality of locations on the surface of the computer chip.

[0045] Typically, the input representation includes (i) partially placed chip data indicating the layout of nodes that have been placed at the time step before the particular time step in the sequence, and (ii) current node data representing the nodes to be placed at the particular time step.

[0046] like Figure 2 As shown in , the node layout neural network 110 includes an embedding subnetwork 210 , a policy subnetwork 220 , and, optionally, a value subnetwork 230 .

[0047] The embedding subnetwork 210 is configured to process the input representation at each specific time step to generate an embedding of the input representation. An embedding is a digital representation in a fixed dimensional space, i.e., an ordered set of a fixed number of numerical values. For example, an embedding can be a vector or matrix of floating point values ​​or other types of numerical values.

[0048] The policy subnetwork 220 is configured to process the embedding at each particular time step to generate a score distribution. For example, the policy subnetwork 220 can be a recurrent neural network, i.e., a neural network including one or more recurrent neural network layers, such as a long short-term memory (LSTM) layer, a gated recurrent unit (GRU) layer, or other types of recurrent layers, with an output layer that generates position scores. For example, when the scores are probabilities, the output layer can be a softmax layer.

[0049] When used, the value subnetwork 230 is configured to process the embedding at each particular time step to generate a value estimate that estimates the value of the current state of the floor plan as of the particular time step. The value of the current state is an estimate of the output of the reward function of the floor plan generated starting from the current state (i.e., starting from the current partial floor plan). For example, the value subnetwork 230 may also be a recurrent neural network or may be a feed-forward neural network, such as a feed-forward neural network including one or more fully connected layers.

[0050] This value estimate can be used during training of the neural network 110 (i.e., when using a reinforcement learning technique that relies on the available value estimates). In other words, when the reinforcement learning technique used to train the node layout neural network requires a value estimate, the node layout neural network 110 also includes a value subnetwork 230 that generates the value estimates required by the reinforcement learning technique.

[0051] Typically, the embedding subnetwork 210 processes partially placed chip data using a set of neural network layers to generate partially placed chip data embeddings, and processes data representing the current node using another set of neural network layers to generate current node data embeddings, and then connects the two embeddings to generate a final embedding for the state representation.

[0052] One example of a state representation that may be provided as input to the placement neural network 110 is a state representation in which the partially placed chip data is a vector that includes a flattened representation of an N×M grid, i.e., a vector that includes a corresponding entry for each grid square in the N×M grid. For each grid square, the flattened representation of the N×M grid may identify whether a node has been placed at the grid square, and optionally which node (if any) has been placed at the grid square.

[0053] In this example, sub-network 210 may process the partially placed chip data using a feed-forward neural network, such as a multi-layer perceptron (MLP), to generate a partially placed chip embedding.

[0054] Another example of a state representation is one in which the partially placed chip data is an image of the surface of the partially placed chip, the image depicting locations where nodes had been placed at a time step prior to a particular time step in the sequence.

[0055] This example is in Figure 2 In the description.

[0056] Specifically, in Figure 2 , the state representation includes an image 202 showing a partially placed chip of a grid square of an N×M grid. The image 202 depicts the location, connectivity, and optionally size of nodes that have been placed on the chip. Figure 2 As shown in an image of , when a node has been placed at a given grid square, the image depicts a visual representation of the node at the grid square. A connection between two nodes can be represented as an edge between the two nodes in the image, such as an edge drawn using taxi geometry in the image.

[0057] The system also tracks the density of locations on the chip, i.e., the density of squares in the grid. Specifically, the system maintains a density value for each location, which indicates how occupied the location is. When a node has been placed at a given location, the density value for that location is set equal to 1 (or set equal to a different maximum value indicating that the location is fully occupied). When no node has been placed at a given location, the density value for that location indicates the number of edges that pass through that location.

[0058] In some embodiments, when generating the input image, the system uses tracked density values. That is, the input image is generated based on the tracked density values ​​of the locations on the chip surface. As an example, for any given location, the intensity value of a pixel belonging to the visual representation of the chip at that location can be based on the density value of the location, such as where locations with higher density values ​​shave higher intensity values, such as where the intensity value is equal to the density value or is proportional to the density value. As another example, the input image can include an additional channel that includes the intensity value of the location in the input image.

[0059] When the partially placed chip data is an image, sub-network 210 processes the image using a convolutional neural network to generate a partially placed chip embedding.

[0060] The data representing the current node included in the state representation may represent the current node in any suitable manner.

[0061] As an example, the current node data may be a one-hot vector identifying which of the multiple nodes has been placed at the current time step. In this example, the subnetwork 210 may process the current node data using an MLP, a single fully connected layer, or an embedding layer to generate the current node data.

[0062] As another example, the current node data may include an embedding 206 of the current node, ie, an embedding learned jointly with training the node layout neural network 110 .

[0063] As another example, current node data characterizing a node to be placed at a particular time step may include data characterizing the connectivity of the node to be placed at the particular time step specified by the netlist data, i.e., an adjacency vector or matrix identifying nodes to be connected to the current node in the netlist data.

[0064] In some embodiments, the input representation also includes a netlist feature 204 that characterizes the netlist data. Specifically, the netlist feature may include a corresponding feature of each node in the netlist, such as a corresponding feature vector, and may identify connectivity between nodes in the netlist data. The feature of each node may identify any of various features of the node.

[0065] In one example, the feature of each node identifies the size of the node. As another example, for each node that has been placed at an earlier time step in the sequence, the feature of the node identifies the location at which the node has been placed on the chip surface. If the node has not yet been placed, the feature of the node may indicate a default location for the node. Figure 2 In the example of , the system combines netlist features 204 into a feature matrix of feature vectors and an adjacency matrix that specifies connectivity between nodes.

[0066] As another example, the netlist features 204 may also include port information, eg, information about their connectivity to other nodes and their locations, even if the port nodes are not placed by the system.

[0067] As another example, the netlist feature 204 may include a full name of a node in the netlist data, for example, when the name of the node has information related to a level of the node within the netlist graph, a position of the node, a width and a height of the node.

[0068] In embodiments where the input representation includes netlist features, subnetwork 210 may process the netlist features and the current node data using any suitable graph embedding mechanism to generate a current node embedding.

[0069] For example, subnetwork 210 may process the netlist features and the current node data using a graph convolutional neural network (GCN) to generate a current node embedding. An example of a GCN that may be used by subnetwork 210 is described in “Semi-supervised classification with graph convolutional networks” by Thomas N. Kipf and Max Welling (International Conference on Learning Representations, 2017).

[0070] As another example, subnetwork 210 may process the netlist features and the current node data using a Siamese neural network to generate a current node embedding. An example of a Siamese neural network that may be used by subnetwork 210 is described in "Siamese neural networks for one-shot image recognition" by G Koch, R Zemel, and R Salakhutdinov (ICML Deep Learning Workshop, 2015).

[0071] As another example, subnetwork 210 may process netlist features and current node data using Graphsage technology to generate a current node embedding. One example of this technology that may be used by subnetwork 210 is described in "Inductive Representation Learning on Large Graphs" by W. L. Hamilton, R. Ying, and J. Leskovec (arXiv:1706.02216 [cs.SI], 2017).

[0072] As described above, the system tracks the density at every location on the chip surface, i.e., for every grid square of the N × M grid. Once the policy neural network has generated a score distribution at a time step, the system uses the density to generate a modified score distribution, and then uses the modified score distribution to assign nodes corresponding to the time step.

[0073] For example, the system may assign the node to the position with the highest score in the modified score distribution, or sample the positions according to the modified score distribution so that each position has an equal probability of being selected as the probability, and then assign the node to the sampled positions.

[0074] To modify the score distribution, the system sets the score to zero for any location (ie, any grid square) where the density is above a threshold.

[0075] This is Figure 2 250 that can be applied to the score distribution, i.e., a mask that is represented as a value of zero for any location where the density is above a threshold and a mask that is represented as a value of 1 for any location where the density is not above the threshold. As a specific example, the threshold can be equal to 1, and the system can set the score of any location where a node has been placed (i.e., has a density value of 1) to zero. As another example, the threshold can be less than 1, indicating that the system also sets the score of any location that has no nodes but has a number of wires passing through it to zero (i.e., the number of wires associated with the location is above the threshold).

[0076] Figure 3 is a flow chart of an example process 300 for placing nodes at a particular time step. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed floor plan generation system (e.g., Figure 1 The plan view generation system 100 may perform process 300 .

[0077] For each time step in the sequence of time steps, the system can perform process 300 to generate a computer chip floor plan.

[0078] The system generates a state representation for the time step (step 302). As described above, the state representation typically includes partially placed chip data and current node data.

[0079] The system processes the state representation of the time step using a node layout neural network to generate a distribution of scores over the locations (step 304).

[0080] The system uses the score distribution to place the nodes (step 306). As described above, the system can modify the score distribution based on the density of locations on the chip surface and then select the location of the node based on the modified score distribution.

[0081] Figure 4 is a flow chart of an example process 400 for training a node layout neural network. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed floor plan generation system (e.g., Figure 1 The plan view generation system 100 may perform process 400 .

[0082] The system may repeatedly perform process 400 to train the node layout neural network, ie, determine training values ​​for network parameters.

[0083] In some embodiments, the system distributes the training of the node layout neural network among many different workers, i.e., many different homogeneous or heterogeneous computing devices, i.e., devices that use CPUs, GPUs, or ASICs to perform training calculations. In some of these embodiments, some or all of steps 400 can be performed in parallel by many different workers operating asynchronously with each other to accelerate the training of the node layout neural network. In other embodiments, different workers operate synchronously to perform iterations of process 400 in parallel to accelerate the training of the neural network.

[0084] As mentioned above Figure 2 As described, the system uses a node layout neural network and generates a candidate floor plan based on current values ​​of network parameters (step 402).

[0085] The system evaluates the reward function to determine a reward for the generated candidate floor plan (step 404 ).

[0086] As mentioned above, the reward function usually measures the quality of the floor plan generated using the node layout neural network.

[0087] More specifically, the reward function measures certain features of the generated floorplan that, when optimized, result in good performance of a chip manufactured using the generated floorplan, such as in one or more aspects of power consumption, thermal performance, and timing performance.

[0088] Specifically, the reward function includes corresponding terms of one or more features. For example, when there are multiple terms, the reward function can be the sum or weighted sum of the multiple terms.

[0089] As an example, the reward function may include a wire length metric (i.e., a term that measures wire length) that is higher when the wire length between nodes on the chip surface is shorter. For example, the wire length may be the Manhattan distance or other distance metric between all adjacent nodes on the chip surface.

[0090] As another example, the reward function may include a congestion metric (i.e., a term that measures congestion), which is higher when congestion on the surface of the computer chip is lower. Congestion is a measure of the difference between the available routing resources in a given area (not necessarily a continuous area) on the chip and the actual wires that pass through the area. For example, congestion can be defined as the ratio of the wires that pass through the area in the generated floor plan to the available routing resources (e.g., the maximum number of wires that can pass through the area). As a specific example, the congestion metric can track the density of wires across the horizontal and vertical edges of the surface. Specifically, the system can utilize a routing model of the netlist (e.g., a network bounding box, upper L, lower L, A*, minimum spanning tree, or actual routing net, etc.). Based on this routing model, the congestion metric can be calculated by determining the ratio of the available routing resources in the floor plan to the routing estimate from the routing model of the location for each location on the surface.

[0091] As another example, a reward function may include a time term (i.e., a term that measures the timing of digital logic) that is higher when the performance of the chip is better (e.g., the reward function takes a correspondingly higher value for a floor plan of a corresponding chip that takes less time to perform a particular computational task). The timing or performance of a floor plan can be measured using static timing analysis (STA). This measurement can include calculating the level delays on the logic paths (including internal cell delays and wire delays) and finding the critical path, which will determine the maximum speed at which the clock can run safely. For a realistic view of timing, logic optimization may need to be performed to accommodate paths that are getting longer or shorter as node placement is being performed.

[0092] As another example, the reward function may include one or more terms that measure the power or energy to be consumed by the chip, ie, one or more terms that are higher when the power consumed by the chip is lower.

[0093] As another example, the reward function may include one or more terms that measure the area of ​​a floor plan, ie, one or more terms that are higher when the area occupied by the floor plan is smaller.

[0094] The system uses reinforcement learning techniques to determine updates to current values ​​of network parameters (step 406 ).

[0095] Specifically, the system determines gradients with respect to network parameters of a reinforcement loss function corresponding to a reinforcement learning technique, and then computes updates from the determined gradients according to an appropriate update rule (e.g., a stochastic gradient descent update rule, an Adam update rule, or an rmsProp update rule).

[0096] As a specific example of how training can be distributed among multiple workers, each of the multiple workers can perform steps 402 to 406 of the process and then calculate gradients in parallel. The multiple workers can then transmit the calculated gradients to a parameter server, which can calculate updates and then update the network parameters.

[0097] The system can train the node layout neural network using any of a number of reinforcement learning techniques. For example, the system can train using a policy gradient technique such as REINFORCE or Proximal Policy Optimization (PPO). In these cases, the value predictions generated by the value subnetwork can be used to calculate a baseline value that modifies the reward function value when calculating the gradient of the reinforcement learning loss function.

[0098] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0099] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly implemented computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, an optical signal, or an electromagnetic signal, which is generated to encode information transmitted to a suitable receiver device for execution by the data processing device.

[0100] The term "data processing apparatus" refers to data processing hardware and covers all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or may further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof.

[0101] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may (but need not) correspond to a file in a file system. A program may be stored in a portion of a file that stores other programs or data (e.g., one or more scripts stored in a markup language document) in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers, which are located at one location or distributed across multiple locations and interconnected by a communication network.

[0102] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way or at all, and the data may be stored in one or more locations on a storage device. Thus, for example, an index database may include multiple data collections, each of which may be organized and accessed differently.

[0103] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed on and run on the same one or more computers.

[0104] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system (e.g., FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmed computers.

[0105] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or on a central processing unit of any other kind. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for executing or implementing instructions, and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a dedicated logic circuit system or incorporated into the dedicated logic circuit system. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operably coupled to receive data from the one or more mass storage devices or transfer data to the one or more mass storage devices, or receive data from the one or more mass storage devices and transfer data to the one or more mass storage devices. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, for example, to name a few, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).

[0106] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0107] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input generated from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application), and then receiving a response message from the user.

[0108] A data processing device used to implement a machine learning model may also include, for example, dedicated hardware accelerator units that are used to handle both general-purpose and compute-intensive portions of a machine learning training or production (i.e., inference) workload.

[0109] You can implement and deploy machine learning models using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0110] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server; or includes a middleware component, such as an application server; or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an application program, through which a user can interact with an implementation of the subject matter described in this specification; or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0111] A computing system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. The relationship of client and server is created by means of computer programs running on respective computers and having a client-server relationship to each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from the user. Data generated at a user device (e.g., due to user interaction) may be received from the device at the server.

[0112] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the scope of what can be claimed, but rather as a description of the features of a particular embodiment that may be specific to a particular invention. Certain features described in this specification in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the individual features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0113] Similarly, although operations are depicted in a particular order in the drawings and described in a particular order in the claims, this should not be construed as requiring that such operations be performed in the particular order shown or in sequence, or that all of the described operations be performed to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0114] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As another example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: obtaining netlist data for a computer chip, wherein the netlist data specifies connectivity between a plurality of nodes on the computer chip, each node corresponding to one or more of a plurality of integrated circuit components of the computer chip, wherein the plurality of nodes includes nodes representing at least macros and nodes representing clusters of standard cells; and Generating a computer chip floor plan that places each node in the netlist data at a corresponding position on a surface of the computer chip, comprising: Placing at least the node representing the macro on the surface of the computer chip includes placing a corresponding node at each of a plurality of time steps in a sequence of time steps, and for each particular time step in the plurality of time steps, the placing includes: generating an input representation for the particular time step, the input representation characterizing at least (i) a location on the surface of the computer chip of a node that has been placed at a time step prior to the particular time step in the sequence, and (ii) the node to be placed at the particular time step; processing the input representation using a node layout neural network having a plurality of network parameters, wherein the node layout neural network is configured to process the input representation according to current values ​​of the network parameters to generate a distribution of scores over a plurality of locations on the surface of the computer chip; and The node to be placed at the particular time step is assigned to one of the plurality of locations using the score distribution.

2. The method according to claim 1, wherein: Assigning the node to one of the plurality of locations using the score distribution comprises: generating a modified score distribution that sets to zero the score of each location where the density determined based on the corresponding location on the surface of the computer chip of a node that had been placed at a time step prior to the particular time step in the sequence exceeds a threshold; and The nodes are assigned using the modified score distribution.

3. The method according to claim 2, wherein: Allocating the nodes using the modified score distribution includes: The node is assigned to the position having the highest score in the modified score distribution.

4. The method according to claim 2, wherein: Allocating the nodes using the modified score distribution includes: Positions are sampled from the modified score distribution, and the nodes are assigned to the sampled positions.

5. The method according to claim 1, wherein: The plurality of locations comprises grid squares from an N x M grid overlying the surface of the computer chip.

6. The method according to claim 1, wherein: The input representation includes data characterizing connectivity of the nodes specified by the netlist data to be placed at the particular time step.

7. The method according to claim 1, wherein: The input representation includes netlist features characterizing the netlist data.

8. The method according to claim 7, wherein: The netlist features include a corresponding feature vector for each node in the netlist.

9. The method according to claim 8, wherein: The feature vector for each node identifies the size of the node according to its location on the surface of the computer chip.

10. The method according to claim 8, wherein: For each node that has been placed at an earlier time step in the sequence, the feature vector of the node identifies the location where the node was placed.

11. The method according to claim 1, wherein: The data characterizing the node to be placed at the particular time step includes an embedding of the node to be placed at the particular time step.

12. The method according to claim 1, wherein: The partially placed chip data comprises an image of the surface of the partially placed chip, the image depicting the location where the node had been placed at a time step prior to the particular time step in the sequence.

13. The method according to claim 12, wherein: In the image, each location of the plurality of locations is associated with a density value reflecting an occupancy of the location in the partially placed chips.

14. The method according to claim 1, wherein: The node layout neural network comprises: an input subnetwork configured to process the input representation at each specific time step to generate an embedding of the input representation; and A policy subnetwork is configured to process the embeddings to generate the score distribution at each specific time step.

15. The method according to claim 14, wherein: The input subnetwork comprises a convolutional neural network configured to generate an embedding of a partially placed chip depicting the location where the node had been placed at a time step prior to the particular time step in the sequence.

16. The method according to claim 14, wherein: The input subnetwork includes a graph convolutional neural network configured to generate data representing the nodes to be placed and embeddings of the netlist features.

17. The method according to claim 14, wherein: The node layout neural network comprises: A value subnetwork is configured to, at each particular time step, process the embedding to generate a value estimate that estimates a value of a current state of the floor plan as of the particular time step.

18. The method according to claim 1, further comprising: The current values ​​of the network parameters are determined by training the node layout neural network via reinforcement learning to generate a probability distribution that maximizes a reward function that measures the quality of a floor plan generated using the node layout neural network.

19. The method according to claim 18, wherein: The reward function includes a wire length metric that is higher when wire lengths between nodes on the surface of the computer chip are shorter.

20. The method according to claim 18, wherein: The reward function includes a congestion metric that is higher when congestion on the surface of the computer chip is lower.

21. A system comprising one or more computers and one or more storage devices, the storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: obtaining netlist data for a computer chip, wherein the netlist data specifies connectivity between a plurality of nodes on the computer chip, each node corresponding to one or more of a plurality of integrated circuit components of the computer chip, wherein the plurality of nodes includes nodes representing at least macros and nodes representing clusters of standard cells; and Generating a computer chip floor plan that places each node in the netlist data at a corresponding position on a surface of the computer chip, comprising: Placing at least the node representing the macro on the surface of the computer chip includes placing a corresponding node at each of a plurality of time steps in a sequence of time steps, and for each particular time step in the plurality of time steps, the placing includes: generating an input representation for the particular time step, the input representation characterizing at least (i) a location on the surface of the computer chip of a node that has been placed at a time step prior to the particular time step in the sequence, and (ii) the node to be placed at the particular time step; processing the input representation using a node layout neural network having a plurality of network parameters, wherein the node layout neural network is configured to process the input representation according to current values ​​of the network parameters to generate a distribution of scores over a plurality of locations on the surface of the computer chip; and The node to be placed at the particular time step is assigned to one of the plurality of locations using the score distribution.

22. The system of claim 21, wherein: Assigning the node to one of the plurality of locations using the score distribution comprises: generating a modified score distribution that sets to zero the score of each location where the density determined based on the corresponding location on the surface of the computer chip of a node that had been placed at a time step prior to the particular time step in the sequence exceeds a threshold; and The nodes are assigned using the modified score distribution.

23. The system of claim 21, wherein: The plurality of locations comprises grid squares from an N x M grid overlying the surface of the computer chip.

24. The system of claim 21, wherein: The input representation includes data characterizing connectivity of the nodes specified by the netlist data to be placed at the particular time step.

25. The system of claim 21, wherein: The input representation includes a corresponding feature vector for each node in the netlist.

26. The system of claim 25, wherein: For each node that has been placed at an earlier time step in the sequence, the feature vector of the node identifies the location where the node was placed.

27. The system of claim 21, wherein: The data characterizing the node to be placed at the particular time step includes an embedding of the node to be placed at the particular time step.

28. The system of claim 21, wherein: The node layout neural network comprises: an input subnetwork configured to process the input representation at each specific time step to generate an embedding of the input representation; and A policy subnetwork is configured to process the embeddings to generate the score distribution at each specific time step.

29. The system of claim 21, the operations further comprising: The current values ​​of the network parameters are determined by training the node layout neural network via reinforcement learning to generate a probability distribution that maximizes a reward function that measures the quality of a floor plan generated using the node layout neural network.

30. The system of claim 29, wherein: The reward function includes a wire length metric that is higher when wire lengths between nodes on the surface of the computer chip are shorter.

31. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: obtaining netlist data for a computer chip, wherein the netlist data specifies connectivity between a plurality of nodes on the computer chip, each node corresponding to one or more of a plurality of integrated circuit components of the computer chip, wherein the plurality of nodes includes nodes representing at least macros and nodes representing clusters of standard cells; and Generating a computer chip floor plan that places each node in the netlist data at a corresponding position on a surface of the computer chip, comprising: Placing at least the node representing the macro on the surface of the computer chip includes placing a corresponding node at each of a plurality of time steps in a sequence of time steps, and for each particular time step in the plurality of time steps, the placing includes: generating an input representation for the particular time step, the input representation characterizing at least (i) a location on the surface of the computer chip of a node that has been placed at a time step prior to the particular time step in the sequence, and (ii) the node to be placed at the particular time step; processing the input representation using a node layout neural network having a plurality of network parameters, wherein the node layout neural network is configured to process the input representation according to current values ​​of the network parameters to generate a distribution of scores over a plurality of locations on the surface of the computer chip; and The node to be placed at the particular time step is assigned to one of the plurality of locations using the score distribution.

32. The computer-readable storage medium of claim 31, wherein: Assigning the node to one of the plurality of locations using the score distribution comprises: generating a modified score distribution that sets to zero the score of each location where the density determined based on the corresponding location on the surface of the computer chip of a node that had been placed at a time step prior to the particular time step in the sequence exceeds a threshold; and The nodes are assigned using the modified score distribution.

33. The computer-readable storage medium of claim 31, wherein: The plurality of locations comprises grid squares from an N x M grid overlying the surface of the computer chip.

34. The computer-readable storage medium of claim 31, wherein: The input representation includes data characterizing connectivity of the nodes specified by the netlist data to be placed at the particular time step.

35. The computer-readable storage medium of claim 31, wherein: The input representation includes a corresponding feature vector for each node in the netlist.

36. The computer-readable storage medium of claim 35, wherein: For each node that has been placed at an earlier time step in the sequence, the feature vector of the node identifies the location where the node was placed.

37. The computer-readable storage medium of claim 31, wherein: The data characterizing the node to be placed at the particular time step includes an embedding of the node to be placed at the particular time step.

38. The computer-readable storage medium of claim 31, wherein: The node layout neural network comprises: an input subnetwork configured to process the input representation at each specific time step to generate an embedding of the input representation; and A policy subnetwork is configured to process the embeddings to generate the score distribution at each specific time step.

39. The computer-readable storage medium of claim 31, the operations further comprising: The current values ​​of the network parameters are determined by training the node layout neural network via reinforcement learning to generate a probability distribution that maximizes a reward function that measures the quality of a floor plan generated using the node layout neural network.

40. The computer-readable storage medium of claim 39, wherein: The reward function includes a wire length metric that is higher when wire lengths between nodes on the surface of the computer chip are shorter.

41. A method performed by one or more computers, the method comprising: obtaining netlist data for a computer chip, wherein the netlist data specifies connectivity between a plurality of nodes on the computer chip, each node corresponding to one or more of a plurality of integrated circuit components of the computer chip; generating a computer chip floor plan using a node placement neural network, the computer chip floor plan placing each node in the netlist data at a corresponding location on a surface of the computer chip, the node placement neural network comprising: (i) an input subnetwork configured to, at each time step in a plurality of time steps, process an input representation of the time step to generate an embedding of the input representation; and (ii) a policy subnetwork configured to, at each time step in the plurality of time steps, process the embedding of the input representation of the time step to generate a score distribution over a plurality of locations on the surface of the computer chip; generating a reward for the computer chip floor plan using a reward function, the reward function measuring the quality of the computer chip floor plan; and At least the policy sub-network of the node layout neural network is trained via reinforcement learning using at least the reward to generate a probability distribution that maximizes the reward function.

42. The method according to claim 41, wherein: The node layout neural network further includes a value subnetwork configured to, at each of the time steps, process the embedding of the input representation of the time step to generate a value estimate that estimates a value of a current state of the computer chip floor plan as of the time step, and wherein training at least the policy subnetwork of the node layout neural network via reinforcement learning includes training at least the policy subnetwork via reinforcement learning using the value estimates and the rewards for the plurality of time steps.

43. The method of claim 41, wherein: The reward function includes a wire length metric that is higher when wire lengths between nodes on the surface of the computer chip are shorter.

44. The method of claim 41, wherein: The reward function includes a congestion metric that is higher when congestion on the surface of the computer chip is lower.

45. The method of claim 44, wherein: The congestion metric tracks the density of wires across the horizontal and vertical edges of the surface.

46. ​​The method of claim 41, wherein: At each time step, the input representation for the time step represents a location on the surface of the computer chip that characterizes at least (i) a node that has been placed at a time step prior to the particular time step in the sequence, and (ii) a node that is to be placed at the particular time step.

47. The method of claim 46, wherein: Generating the computer chip floor plan includes assigning, at each time step, the node to be placed at the time step to one of the plurality of locations using the score distribution.

48. The method of claim 47, wherein: Allocating the node to be placed at the time step comprises: generating a modified score distribution that sets to zero the score of each location where the density determined based on the corresponding location on the surface of the computer chip of a node that had been placed at the time step prior to the time step exceeds a threshold; and The nodes are assigned using the modified score distribution.

49. The method of claim 41, wherein: The plurality of locations comprises grid squares from an N x M grid overlying the surface of the computer chip.

50. The method of claim 41, wherein: The training of the node layout neural network is distributed across multiple synchronous workers.

51. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: obtaining netlist data for a computer chip, wherein the netlist data specifies connectivity between a plurality of nodes on the computer chip, each node corresponding to one or more of a plurality of integrated circuit components of the computer chip; generating a computer chip floor plan using a node placement neural network, the computer chip floor plan placing each node in the netlist data at a corresponding location on a surface of the computer chip, the node placement neural network comprising: (i) an input subnetwork configured to, at each time step in a plurality of time steps, process an input representation of the time step to generate an embedding of the input representation; and (ii) a policy subnetwork configured to, at each time step in the plurality of time steps, process the embedding of the input representation of the time step to generate a score distribution over a plurality of locations on the surface of the computer chip; generating a reward for the computer chip floor plan using a reward function, the reward function measuring the quality of the computer chip floor plan; and At least the policy sub-network of the node layout neural network is trained via reinforcement learning using at least the reward to generate a probability distribution that maximizes the reward function.

52. A system comprising one or more computers and one or more storage devices, the storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: obtaining netlist data for a computer chip, wherein the netlist data specifies connectivity between a plurality of nodes on the computer chip, each node corresponding to one or more of a plurality of integrated circuit components of the computer chip; generating a computer chip floor plan using a node placement neural network, the computer chip floor plan placing each node in the netlist data at a corresponding location on a surface of the computer chip, the node placement neural network comprising: (i) an input subnetwork configured to, at each time step in a plurality of time steps, process an input representation of the time step to generate an embedding of the input representation; and (ii) a policy subnetwork configured to, at each time step in the plurality of time steps, process the embedding of the input representation of the time step to generate a score distribution over a plurality of locations on the surface of the computer chip; generating a reward for the computer chip floor plan using a reward function, the reward function measuring the quality of the computer chip floor plan; and At least the policy sub-network of the node layout neural network is trained via reinforcement learning using at least the reward to generate a probability distribution that maximizes the reward function.

53. The system of claim 52, wherein: The node layout neural network further includes a value subnetwork configured to, at each of the time steps, process the embedding of the input representation of the time step to generate a value estimate that estimates a value of a current state of the computer chip floor plan as of the time step, and wherein training at least the policy subnetwork of the node layout neural network via reinforcement learning includes training at least the policy subnetwork via reinforcement learning using the value estimates and the rewards for the plurality of time steps.

54. The system of claim 52, wherein: The reward function includes a wire length metric that is higher when wire lengths between nodes on the surface of the computer chip are shorter.

55. The system of claim 52, wherein: The reward function includes a congestion metric that is higher when congestion on the surface of the computer chip is lower.

56. The system of claim 55, wherein: The congestion metric tracks the density of wires across the horizontal and vertical edges of the surface.

57. The system of claim 52, wherein: At each time step, the input representation for the time step represents a location on the surface of the computer chip that characterizes at least (i) a node that has been placed at a time step prior to the particular time step in the sequence, and (ii) a node that is to be placed at the particular time step.

58. The system of claim 57, wherein: Generating the computer chip floor plan includes assigning, at each time step, the node to be placed at the time step to one of the plurality of locations using the score distribution.

59. The system of claim 58, wherein: Allocating the node to be placed at the time step comprises: generating a modified score distribution that sets to zero the score of each location where the density determined based on the corresponding location on the surface of the computer chip of a node that had been placed at the time step prior to the time step exceeds a threshold; and The nodes are assigned using the modified score distribution.

60. The system of claim 52, wherein: The plurality of locations comprises grid squares from an N x M grid overlying the surface of the computer chip.

61. A method performed by one or more computers, the method comprising: obtaining netlist data for a computer chip, wherein the netlist data specifies connectivity between a plurality of nodes on the computer chip, each node corresponding to one or more of a plurality of integrated circuit components of the computer chip; A computer chip floor plan is generated using a node placement neural network, the computer chip floor plan placing each node in the netlist data at a corresponding location on a surface of the computer chip, the node placement neural network having been trained by reinforcement learning to maximize a reward function measuring the quality of the computer chip floor plan, wherein the node placement neural network comprises: (i) an input subnetwork configured to, at each time step in a plurality of time steps, process an input representation of the time step to generate an embedding of the input representation; and (ii) a policy subnetwork configured to, at each time step in the plurality of time steps, process the embedding of the input representation of the time step to generate a score distribution over a plurality of locations on the surface of the computer chip.

62. The method of claim 61, wherein: The node layout neural network further includes a value subnetwork configured to, at each of the time steps, process the embedding of the input representation of the time step to generate a value estimate that estimates a value of a current state of the computer chip floor plan as of the time step, and wherein training via reinforcement learning includes training at least the policy subnetwork and the value subnetwork via reinforcement learning using the value estimates and the rewards for the plurality of time steps.

63. The method of claim 61, wherein: The reward function includes a wire length metric that is higher when wire lengths between nodes on the surface of the computer chip are shorter.

64. The method of claim 61, wherein: The reward function includes a congestion metric that is higher when congestion on the surface of the computer chip is lower.

65. The method of claim 64, wherein: The congestion metric tracks the density of wires across the horizontal and vertical edges of the surface.

66. The method of claim 61, wherein: At each time step, the input representation for the time step represents a location on the surface of the computer chip that characterizes at least (i) a node that has been placed at a time step prior to the particular time step in the sequence, and (ii) a node that is to be placed at the particular time step.

67. The method of claim 66, wherein: Generating the computer chip floor plan includes assigning, at each time step, the node to be placed at the time step to one of the plurality of locations using the score distribution.

68. The method of claim 67, wherein: Allocating the node to be placed at the time step comprises: generating a modified score distribution that sets to zero the score of each location where the density determined based on the corresponding location on the surface of the computer chip of a node that had been placed at the time step prior to the time step exceeds a threshold; and The nodes are assigned using the modified score distribution.

69. The method of claim 61, wherein: The plurality of locations comprises grid squares from an N x M grid overlying the surface of the computer chip.

70. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: obtaining netlist data for a computer chip, wherein the netlist data specifies connectivity between a plurality of nodes on the computer chip, each node corresponding to one or more of a plurality of integrated circuit components of the computer chip; A computer chip floor plan is generated using a node placement neural network, the computer chip floor plan placing each node in the netlist data at a corresponding location on a surface of the computer chip, the node placement neural network having been trained by reinforcement learning to maximize a reward function measuring the quality of the computer chip floor plan, wherein the node placement neural network comprises: (i) an input subnetwork configured to, at each time step in a plurality of time steps, process an input representation of the time step to generate an embedding of the input representation; and (ii) a policy subnetwork configured to, at each time step in the plurality of time steps, process the embedding of the input representation of the time step to generate a score distribution over a plurality of locations on the surface of the computer chip.

71. A system comprising one or more computers and one or more storage devices, the storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: obtaining netlist data for a computer chip, wherein the netlist data specifies connectivity between a plurality of nodes on the computer chip, each node corresponding to one or more of a plurality of integrated circuit components of the computer chip; A computer chip floor plan is generated using a node placement neural network, the computer chip floor plan placing each node in the netlist data at a corresponding location on a surface of the computer chip, the node placement neural network having been trained by reinforcement learning to maximize a reward function measuring the quality of the computer chip floor plan, wherein the node placement neural network comprises: (i) an input subnetwork configured to, at each time step in a plurality of time steps, process an input representation of the time step to generate an embedding of the input representation; and (ii) a policy subnetwork configured to, at each time step in the plurality of time steps, process the embedding of the input representation of the time step to generate a score distribution over a plurality of locations on the surface of the computer chip.

72. The system of claim 71, wherein: The node layout neural network further includes a value subnetwork configured to, at each of the time steps, process the embedding of the input representation of the time step to generate a value estimate that estimates a value of a current state of the computer chip floor plan as of the time step, and wherein training via reinforcement learning includes training at least the policy subnetwork and the value subnetwork via reinforcement learning using the value estimates and the rewards for the plurality of time steps.

73. The system of claim 71, wherein: The reward function includes a wire length metric that is higher when wire lengths between nodes on the surface of the computer chip are shorter.

74. The system of claim 71, wherein: The reward function includes a congestion metric that is higher when congestion on the surface of the computer chip is lower.

75. The system of claim 74, wherein: The congestion metric tracks the density of wires across the horizontal and vertical edges of the surface.

76. The system of claim 71, wherein: At each time step, the input representation for the time step represents a location on the surface of the computer chip that characterizes at least (i) a node that has been placed at a time step prior to the particular time step in the sequence, and (ii) a node that is to be placed at the particular time step.

77. The system of claim 76, wherein: Generating the computer chip floor plan includes assigning, at each time step, the node to be placed at the time step to one of the plurality of locations using the score distribution.

78. The system of claim 77, wherein: Allocating the node to be placed at the time step comprises: generating a modified score distribution that sets to zero the score of each location where the density determined based on the corresponding location on the surface of the computer chip of a node that had been placed at the time step prior to the time step exceeds a threshold; and The nodes are assigned using the modified score distribution.

Images