Macro placement in continuous action space using artificial intelligence solutions

Abstract

A neural network (NN) performs macro placement on a chip. An update mask is updated to mark invalid regions of the chip canvas that are occupied by placed macros. A policy network of the NN generates summary statistics of a two-dimensional (2D) continuous probability distribution over a continuous action space for a given state of the chip canvas. The NN selects an action according to the continuous probability distribution. The selected action corresponds to a coordinate in an unmasked region. The NN generates a trajectory containing (state, action) pairs. A final state in the trajectory corresponds to a complete placement of a macro.

Description

[0001] Cross-references to related applications

[0002] This invention claims the benefit of priority to U.S. Patent Application No. 63 / 254,582, filed October 12, 2021, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Embodiments of the present invention relate to methods and apparatus for generating macro placements on semiconductor chips based on machine learning and artificial intelligence (AI). Background Technology

[0004] In integrated circuit (IC) design, a macro is a set of circuit components that can be viewed as a black box. The logical and electronic behavior of a macro is given, but its internal structure may or may not be known. Mixed-size macro placement is the problem of placing macros of various sizes on a chip canvas to optimize an objective (e.g., line length).

[0005] Typically, the placement area (i.e., the chip canvas) is divided into equal-sized unit regions (called grids). Such a chip canvas is called a grid-discretized chip canvas. During placement, the center of the macro is aligned with the center of one of the grid points. Empirically, placement quality depends on the grid configuration. In practice, increasing the grid resolution does not necessarily improve placement quality.

[0006] The problem with placement regions with low grid resolution (i.e., few grid points and large grid spacing) is that macros may be placed with unacceptable overlap. Furthermore, the placement of very small macros becomes inaccurate because displacement errors can reach half the grid spacing size.

[0007] In placement regions with high grid resolution (i.e., many grid points and small grid spacing), the runtime of an AI placement agent can increase exponentially during the sample collection phase to reach a consistent performance standard. Therefore, in practice, experimenters may observe worse results for higher grid resolutions with the same hyperparameter configuration.

[0008] Therefore, macro deployment techniques need to be improved so that AI agents can perform macro deployments with high performance within an acceptable timeframe. Summary of the Invention

[0009] In one embodiment, a method for macro placement on a chip using a neural network is provided. The method includes: updating a mask to mark invalid regions occupied by macros already placed on a chip canvas; generating a summary statistic of a two-dimensional continuous probability distribution in a continuous action space for a given state of the chip canvas via a policy network of the neural network; selecting an action based on the continuous probability distribution, the selected action corresponding to coordinates in an unmasked region; and generating a trajectory including (state, action) pairs, wherein the final state in the trajectory corresponds to the completed placement of the macro.

[0010] In another embodiment, the system is operable to train a neural network (NN) for macro placement. The system includes processing hardware and memory coupled to the processing hardware to store information about the neural network and the macros to be placed on the chip. The processing hardware is operable to: update a mask to mark invalid regions occupied by macros already placed on the chip canvas; generate a summary statistic of a two-dimensional continuous probability distribution in a continuous action space for a given state of the chip canvas via a policy network of the neural network; select an action based on the continuous probability distribution, the selected action corresponding to coordinates in an unmasked region; and generate a trajectory comprising (state, action) pairs, wherein the final state in the trajectory corresponds to the completed placement of the macro.

[0011] Other aspects and features will become apparent to those skilled in the art upon reading the following description of specific embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0012] The present invention is illustrated by way of example rather than limitation in the accompanying drawings, in which similar reference numerals indicate similar elements. It should be noted that different references to "one" or "a" embodiments in this disclosure do not necessarily refer to the same embodiment, and such references indicate at least one. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed that such feature, structure, or characteristic can be implemented in conjunction with other embodiments within the knowledge of those skilled in the art, whether or not explicitly described.

[0013] Figure 1A This is a block diagram illustrating a neural network (NN) for a macro-arrangement in a continuous action space according to one embodiment.

[0014] Figure 1B This is a block diagram illustrating an NN for macro-arrangement in a discretized action space according to one embodiment.

[0015] Figure 2 A macro layout process according to one embodiment is shown.

[0016] Figure 3 This is a flowchart illustrating a method for training a neural network to perform macro-arrangement in a continuous action space according to one embodiment.

[0017] Figure 4 This is a flowchart of a sample collection operation according to one embodiment.

[0018] Figure 5 This is a flowchart of a training operation according to one embodiment.

[0019] Figure 6 This is a flowchart of an evaluation operation according to one embodiment.

[0020] Figure 7 This is a flowchart illustrating a method for training a neural network to perform macro-arrangement in a continuous action space according to another embodiment.

[0021] Figure 8 This is a flowchart illustrating self-play reinforcement learning according to one embodiment.

[0022] Figure 9 The KR-DL-UCT algorithm for macro layout according to one embodiment is shown.

[0023] Figure 10 This is a flowchart illustrating a weight update operation according to one embodiment.

[0024] Figure 11 An example of a system according to one embodiment is shown.

[0025] Figure 12 This is a flowchart illustrating a method for performing macro deployment for an NN according to one embodiment. Detailed Implementation

[0026] Numerous specific details are set forth in the following description. However, it should be understood that embodiments of the invention can be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail to avoid obscuring the understanding of the invention. However, those skilled in the art will understand that the invention can be practiced without these specific details. Those skilled in the art can achieve appropriate functionality using the included description without excessive experimentation.

[0027] In this disclosure, a neural network (NN) is trained to learn a policy to place an object (e.g., a macro) at a location (x, y) on a circuit block (e.g., a semiconductor chip, also called a chip), where (x, y) are real-valued coordinates on a chip canvas that are not grid-discretized. A macro comprises a set of integrated circuit components, and the chip canvas is a two-dimensional (2D) region on the chip where macros can be placed. The coordinates (x, y) define an action in a continuous action space. Conventional action spaces are grid-discretized. That is, each macro is placed at a location (x′, y′), where the permissible domain of (x′, y′) is a grid. Continuous action spaces eliminate the aforementioned problems associated with grid discretization. In one embodiment, the NN includes a policy network that generates summary statistics of a 2D continuous probability distribution over the action space. In one embodiment, the summary statistics include the mean and standard deviation for each dimension. It may also include alternative or additional statistical data, such as variance, mean, confidence interval, etc.

[0028] Regarding macro placement, action space It is a real coordinate space, representing a chip canvas with width W and height H:

[0029]

[0030] In one embodiment with a deterministic policy, the policy network selects the action with the highest probability based on a continuous probability distribution. Given a canvas of width w and height h, the action is the coordinates of the macro to be placed. The coordinates of this action are a tuple (x, y) directly generated by the policy network, where x ∈ [0, w] and y ∈ [0, h].

[0031] In one embodiment with a stochastic policy, the policy network samples actions based on a continuous probability distribution. For example, the coordinates of an action could be derived from two consecutive Gaussian distributions. and The sampled tuple (x, y).

[0032] Figure 1AThis is a block diagram illustrating an neural network 10 for macro-placement in a continuous action space according to one embodiment. The NN 10 receives input including states s (macro, netlist graph, node ID) and netlist metadata. The NN 10 uses a graph neural network (GNN) 11 to encode the states into a low-dimensional vector, called a GNN embedding 15. The NN 10 also uses a meta encoder 12 to encode the netlist metadata into another low-dimensional vector, called a meta embedding 16. The GNN embedding 15 and the meta embedding 16 are concatenated into a latent state. This latent state is fed into a value network 13 and a policy network 14. The value network 13 generates a value that predicts the reward for action a. The policy network 14 generates a policy π. θ (μ x , σ x μ y , σ y |s), which is specified by a summary statistic of a two-dimensional (2D) probability distribution of action a for a given state s. In one embodiment, the summary statistic includes the mean and standard deviation (e.g., four real numbers: μ) in each dimension of the probability distribution. x , σ x μ y , σ y ).

[0033] A state refers to the state of the chip canvas, including any macros placed on it. An action is a real-valued coordinate on the chip canvas used to place a macro. The probability distribution is a continuous function over the action space. Therefore, the action space is also continuous, called a continuous action space. In a continuous action space, there are infinitely many real-valued coordinates on which macros can be placed. That is, an action can be anywhere in this continuous action space, subject to validity constraints (e.g., only in unmasked regions) and hardware precision (e.g., the bit width of the floating-point representation used in NN computation). The output of policy network 14 is a summary statistic of the continuous probability distribution of action a. Actions can be selected (e.g., by sampling or selection) based on the summary statistic subject to validity constraints. In contrast, a traditional chip canvas is first divided into a grid, and the probability distribution of action a at grid points is computed to generate a probability value for each grid point. Actions are then sampled or selected based on these probability values ​​subject to validity constraints. The large number of probability values ​​at grid points results in significant computational and storage overhead during NN runtime.

[0034] NN 10 is parameterized by θ, which represents the set of parameters defining NN 10. NN 10 applies a mask 18 to the chip canvas and is based on policy π. θ Output the action. This action is based on policy π. θ And generated using either a stochastic or deterministic policy. Using a deterministic policy, NN 20 selects the action with the highest probability within the valid region of the chip canvas (i.e., the unmasked region). Using a stochastic policy, NN 10 samples one or more actions according to a probability distribution until the sampled action falls within the valid region of the chip canvas. In this disclosure, NN 10 following a stochastic policy is referred to as C000, while NN 10 following a deterministic policy is referred to as C001.

[0035] Figure 1B This is a block diagram illustrating an NN 18 for discretizing macro-placement in the action space according to another embodiment. The difference between NN 18 and NN 10 is that the output of NN 18 (i.e., action a) is discretized into a pair of fixed-length numbers (i.e., two-dimensional coordinates). The policy network 14 output of NN 18 is a summary statistic of the continuous probability distribution of the actions. The coordinates (i.e., actions) sampled or selected based on the continuous probability distribution are then discretized by rounding up / down to a given number of decimal places (i.e., fixed-length numbers). The action is based on policy π. θ And generated by either a random or deterministic strategy. Using a random strategy, the NN 18 samples actions and discretizes them. If the action is within the valid region of the chip canvas (i.e., the unmasked region), the action is accepted. If not, sampling and discretization are repeated until the sampled action is within the valid region of the chip canvas. Using a deterministic strategy, the NN 18 selects the discretized action with the highest probability within the valid region of the chip canvas. In this disclosure, the NN 18 following the random strategy is referred to as C002, while the NN 18 following the deterministic strategy is referred to as C003.

[0036] Figure 2 A macro placement process according to one embodiment is illustrated. Given a chip canvas and a trained NN 20, NN 20 performs action a1 to place macro 1 at a first coordinate on the canvas. NN 20 may have the same characteristics as NN 10 ( Figure 1A ) or NN 18( Figure 1BThe same network structure is used. At this point (after executing action a1), the canvas state is represented as s1. Mask 210 is updated to indicate the area around macro 1 that cannot be occupied by the next macro. NN 20 then executes action a2 to place macro 2 at the second coordinate of the unmasked portion of the canvas. The canvas state is updated to s2, and mask 210 is also updated (not shown) to prevent subsequent macros from undesirably overlapping with the first two macros. The chip placement process (including the mask update process) continues until all macros are placed on the chip canvas. Figure 2 The chip placement process shown generates a (state, action) pair (s1, a1), ..., (s n a n The trajectory of ) is used to place n macros, where the final state s n This represents a chip canvas with complete macro placement.

[0037] The following reference Figures 3 to 6 The description provides a first training method for training a neural network (NN) to operate in a continuous action space. (Reference) Figures 7 to 10 The description provides a second training method for training a neural network to operate in a continuous action space. These methods can be performed by a computational system, such as... Figure 11 System 1100 is mentioned, on which placement tools such as neural networks can be trained. Furthermore, some methods described below involve the use of a "threshold." It is understood that the threshold in different methods / stages / operations / steps can refer to different numerical values.

[0038] Figure 3 This is a flowchart illustrating a method 300 for training a neural network (NN) to perform macro-placement in a continuous action space, according to one embodiment. The inputs to method 300 include a chip training set, a chip validation set, and an untrained NN. Method 300 includes three operations: a sample collection operation (S310), a training operation (S320), and an evaluation operation (S330). S310, S320, and S330 are repeated until the reward r output from S330 reaches a predetermined threshold (S340). An example of the reward could be a target, such as line length or other design metrics. At this point, training is complete, and the output is a trained neural network (NN) for macro-placement.

[0039] Figure 4 This is a flowchart of a sample collection operation (S310) according to one embodiment. In the sample collection operation, the NN samples a chip from the training set and samples (i.e. generates) a trajectory on the chip using a stochastic policy (S410). Reference Figure 1A The network C000 in the diagram describes a random policy. To generate trajectories, the neural network uses the current state s of the chip canvas. iAs input (S411). The NN samples action a based on a random policy according to a continuous probability distribution (generated by the NN). i (S412). The sampling action specifies the location to place the macro on the sampled chip. S411 and S412 are repeated until all macros have been placed (S413), and the trajectory is formed by a sequence of (state, action) pairs. The trajectory is then stored in a buffer (S420). When the number of trajectories in the buffer reaches a threshold (S430), the buffer is provided as input for the training operation (S320).

[0040] Figure 5 This is a flowchart of a training operation (S320) according to one embodiment. The inputs to the training operation (S320) include data from... Figure 4 The output trajectory buffer and the untrained neural network NN are used. The training operation begins with the system sampling a mini-batch of trajectories from the buffer (S510). The system uses this mini-batch of trajectories to calculate the loss function L. CLIP+VF+S (θ)(S520), and update the parameters θ of the NN based on gradient descent (S530): Where η is the learning rate. Repeat steps S510, S520, and S530 until the number of updates reaches a predetermined threshold (S540). When the predetermined threshold is reached, the NN has updated parameters θ.

[0041] In S530, the update of the NN parameter θ is computed using a proximal policy optimization (PPO) gradient estimator with generalized advantage estimation. The loss function (L...) CLIP+VF+S The algorithm is described in equation (9) of “Proximal policy optimization algorithms, Schulman et al., arXiv preprint arXiv:1707.06347 (2017)”.

[0042] Figure 6 This is a flowchart of an evaluation operation (S330) according to one embodiment. The evaluation operation (S330) evaluates the training progress of the neural network based on the reward value. The input to the evaluation operation (S330) includes the chip's validation set (in... Figure 3 (in the input) and the NN with updated parameters θ (in Figure 5(In the output). The evaluation operation (S330) begins with the NN sampling a chip in the validation set and sampling (i.e. generating) the trajectory on the chip using a deterministic policy (S610). Reference Figure 1A The deterministic policy is described by the network C001 in the neural network. To generate the trajectory, the neural network uses the current state s. i As input (S611). The NN selects the action a with the highest probability based on a deterministic policy according to a continuous probability distribution (generated by the NN). i (S612). The selected action specifies the location to place the macro on the sampling chip. Repeat S611 and S612 until all macros have been placed (S613), and the trajectory is formed by a sequence of (state, action) pairs. The system is based on the final state s in this trajectory. n Calculate the reward r and collect the reward (S620). Repeat S610 and S620 (including S611-S612) until the number of collected rewards reaches a predetermined threshold (S630). Then the system averages all collected rewards (S640) and outputs a single reward value.

[0043] Return to reference Figure 3 After the evaluation operation (S330), a single reward value is compared with a threshold (S340). Operations S310, S320, and S330 are repeated until the single reward value output from the evaluation operation (S330) reaches the threshold. At this point, the NN is trained to output actions in a continuous action space. A new chip and macros to be placed on this new chip can be provided to the trained NN.

[0044] Figure 7 This is a flowchart illustrating a method 700 for training a neural network to perform macro-arrangement in a continuous action space according to another embodiment. Method 700 begins in a pre-training phase (S710), which performs the same operations as method 300, except that the actions generated by method 700 are discretized as shown in C002 and C003. That is, Figure 4 The sample collection operation (S310) uses C002 for the randomization strategy. Figure 6 The evaluation operation (S330) uses C003 for the deterministic policy. After pre-training the NN in the discretized action space, it is further trained (i.e., fine-tuned) in the continuous action space using self-play reinforcement learning (S720) with C000. The output of self-play reinforcement learning (S720) is the NN with updated weights. The evaluation operation (S730) and... Figure 6The evaluation operation (S330) is the same as in the previous one, using C001 as a deterministic policy in the continuous action space, and is performed on the NN with updated weights. The output of the evaluation operation (S730) is a single reward value. The single reward value is compared with a threshold (S740). Operations S720 and S730 are repeated until the single reward value from the evaluation operation (S730) reaches the threshold (S740). At this point, the NN is trained to output actions in the continuous action space. A new chip and macros to be placed on this new chip can be provided to the trained NN.

[0045] Figure 8 This is a flowchart illustrating self-play reinforcement learning (S720) according to one embodiment. The system uses the KR-DL-UCT algorithm 900 ( Figure 9 The NN is applied to obtain trajectories and the rewards r(S810) for these trajectories. Each trajectory consists of state-action pairs (s1, a1), ..., (s... n a n The system consists of several parts. These trajectories are stored in a buffer (S820), and the trajectories are used to update the weights of the neural network (NN) (S830). The output is the NN with the updated weights.

[0046] Figure 9 An example of an KR-DL-UCT (Kernel Regression-DeepLearning-Upper Confidence bounds applied to Trees) algorithm 900 for macro-layout is illustrated according to one embodiment. The KR-DL-UCT algorithm 900 is based on the algorithm of the same name described in “Deep Reinforcement Learning in Continuous Action Spaces: a Case Study in the Game of Simulated Curling, Lee et al., ICML 2018”, and will not be described in detail here. In algorithm 900, P... θ V represents the policy network. θ This refers to the value network, s t A represents the current state. t s tThe KR-DL-UCT algorithm 900 uses a Monte Carlo tree search (MCTS) method to search for actions and their expected rewards. This method consists of four phases: selection, expansion, simulation, and backpropagation. For macro placement applications, there are some invalid actions in the action space. For example, macros cannot be placed in positions that would cause overlap with other macros to exceed the tolerance. To disable these invalid actions, a mask (e.g., ...) is applied. Figure 1A The masking mechanism (18) is applied to the output of the policy network to ensure the action is valid. Therefore, in line 13 of Algorithm 900, a′ is chosen under some constraints. t Ensure a′ t Valid. For example:

[0047] In the pre-training phase (S710), the PPO method is used to compute and update the NN parameters θ. In the self-play reinforcement learning phase (S720), the NN is further fine-tuned using the MCTS method. The policy obtained from MCTS is then projected back into the function space of the NN's policy network. For each time step t in the self-play, the root state is s. t MCTS returns r and p, where r represents the estimated probability distribution of the reward, and p represents the probability distribution of the action and is proportional to the visit counts estimated based on kernel density estimation. Where τ is the temperature parameter. The parameters of the policy-value network are continuously updated using the following loss function on the data (s, p, r): L(θ) = -rlogV θ (s)-plogπ θ (a;s)+c||θ|| 2 , where π θ (a; s) is the output of the policy network, V θ (s) is the output of the value network, θ is the weight of the NN, and c is a regularization constant. The steps for updating the NN weights will refer to... Figure 10 To provide a more detailed description.

[0048] Figure 10 This is a flowchart illustrating a weight update operation (S830) according to one embodiment. The weight update operation (S830) is... Figure 8 The final step of the self-play reinforcement learning (S720) in the process. The input to S830 includes a buffer with a trajectory and its corresponding reward from the output of the KR-DL-UCT algorithm 900, as well as data from... Figure 7 The pre-trained neural network is output from S710. The weight update operation (S830) begins with the system sampling a mini-batch of trajectories from the trajectory buffer (S1010). The system calculates the loss function L(θ) = -rlogV based on this mini-batch of trajectories. θ (s)-plogπ θ (a;s)+c||θ|| 2 (S1020), where r is the reward, p represents the probability distribution of the action and is proportional to the visit counts estimated based on kernel density estimation. The system then updates the parameters θ of the NN based on gradient descent: Where η is the learning rate (S1030). Repeat S1010, S1020, and S1030 until the number of updates reaches the threshold (S1040). The output is the NN with updated weights.

[0049] Figure 11 An example of a system 1100 according to one embodiment is illustrated. System 1100 includes processing hardware 1110, memory 1120, and network interface 1130. In one embodiment, processing hardware 1110 may include one or more processors and accelerators, such as one or more of the following: central processing unit (CPU), GPU, digital processing unit (DSP), AI processor, tensor processor, neural processor, multimedia processor, and other general-purpose and / or special-purpose processing circuitry.

[0050] System 1100 also includes memory 1120 coupled to processing hardware 1110. Memory 1120 may include memory devices such as dynamic random access memory (DRAM), SRAM, flash memory, and other non-transitory machine-readable storage media, such as volatile or non-volatile storage devices. Memory 1120 may also include storage devices such as any type of solid-state or magnetic storage device. In one embodiment, memory 1120 may store one or more EDA tools 1140, including but not limited to neural networks, AI agents, and other tools for macro deployment. Examples of EDA tools 1140 include C000 and C001 (…). Figure 1A ), and C002 and C003 ( Figure 1BIn some embodiments, memory 1120 may store instructions that, when executed by processing hardware 1110, cause the processing hardware to perform the methods and operations described above for macro placement and / or for training the NN to perform macro placement. However, it should be understood that the methods and operations described above may be performed by methods different from C000 and C001. Figure 1A ) as well as C002 and C003 ( Figure 1B The implementation shall be carried out in accordance with the embodiments of the embodiments of the present invention.

[0051] In some embodiments, system 1100 may further include network interface 1130 for connection to wired and / or wireless networks. It should be understood that simplification has been used for illustrative purposes. Figure 11 Examples of implementations may also include additional hardware components.

[0052] Figure 12 This is a flowchart illustrating a method 1200 for performing macro-layout for a neural network according to one embodiment. Method 1200 can be executed by a computing system, such as... Figure 11 System 1100. Method 1200 begins by updating the mask to mark invalid regions occupied by macros already placed on the chip canvas (S1210). The policy network of the NN generates summary statistics of a two-dimensional (2D) continuous probability distribution over a continuous action space for a given state of the chip canvas (S1220). The NN selects an action based on the continuous probability distribution, the selected action corresponding to coordinates in an unmasked region (S1230). The NN further generates a trajectory comprising (state, action) pairs, where the final state in the trajectory corresponds to the completed placement of the macro (S1240).

[0053] In one embodiment, the summary statistics include at least the mean and standard deviation for each dimension of the continuous probability distribution. The policy network can output four real numbers as the summary statistics to characterize the continuous probability distribution. In one embodiment, the mask is updated after each macro is placed.

[0054] In one embodiment, the neural network (NN) is trained in a continuous action space. The NN can be pre-trained in a discretized action space, where the coordinates corresponding to a given action are rounded to a fixed-length number, and further trained in the continuous action space. In another embodiment, the NN is pre-trained in a discretized action space within repeatedly updated unmasked regions. The pre-trained NN is fine-tuned in the continuous action space within repeatedly updated unmasked regions using self-play reinforcement learning. In one embodiment, the self-play reinforcement learning uses a masked KR-DL-UCT algorithm.

[0055] In one embodiment, the neural network (NN) randomly samples one or more actions from the continuous action space based on a continuous probability distribution until an action is located in an unoccluded region, and generates a set of trajectories based on the repeatedly sampled actions. The NN's parameters are then updated based on a loss function computed from the trajectories. In another embodiment, the NN selects an action from the continuous action space within the unoccluded region, the selected action having the highest probability in the unoccluded region based on a continuous probability distribution. The NN generates a set of trajectories based on the repeatedly selected action. The training progress of the NN is then evaluated based on a reward value computed from the trajectories.

[0056] Various functional components or blocks have been described. As those skilled in the art will understand, functional blocks will preferably be implemented by circuits (dedicated or general-purpose circuits that operate under the control of one or more processors and coded instructions), which typically include transistors configured to control the operation of the circuit according to the functions and operations described herein.

[0057] Although the invention has been described with reference to several embodiments, those skilled in the art will recognize that the invention is not limited to the described embodiments and can be practiced with modifications and variations within the spirit and scope of the appended claims. Therefore, the description is to be considered illustrative rather than restrictive.

Claims

1. A method for macroscopically arranging neural networks on a chip, comprising: Update the mask to mark invalid areas occupied by macros already placed on the chip canvas; Through the policy network of the neural network, a summary statistics of a two-dimensional continuous probability distribution is generated in the continuous action space for a given state of the chip canvas. An action is selected based on the continuous probability distribution, and the selected action corresponds to the coordinates in the unmasked area; and Generate a trajectory consisting of (state, action) pairs, wherein the final state in the trajectory corresponds to the completion placement of the macro.

2. The method as described in claim 1, wherein, The summary statistics include at least the mean and standard deviation of each dimension of the continuous probability distribution.

3. The method of claim 1, wherein the policy network outputs four real numbers as the summary statistics to characterize the continuous probability distribution.

4. The method of claim 1, wherein the neural network is trained in the continuous action space.

5. The method of claim 1, wherein the neural network is pre-trained in a discrete action space, wherein the coordinates corresponding to a given action in the discrete action space are rounded to a fixed-length number, and the neural network is further trained in the continuous action space.

6. The method of claim 1, further comprising: The neural network is pre-trained in a discretized action space within the repeatedly updated unmasked region, wherein the coordinates corresponding to a given action in the discretized action space are rounded to a fixed-length number; and The pre-trained neural network is fine-tuned through self-play reinforcement learning in the continuous action space within the repeatedly updated unmasked region.

7. The method of claim 6, wherein, The self-play reinforcement learning uses the masked KR-DL-UCT algorithm.

8. The method of claim 1, wherein the selection action further comprises: Based on the continuous probability distribution, one or more actions are randomly sampled from the continuous action space until an action is located in the unmasked area; Generate a set of trajectories based on repeated sampling actions; and The parameters of the neural network are updated based on a loss function calculated from the set of trajectories.

9. The method of claim 1, wherein the selection action further comprises: The action is selected from the continuous action space within the unmasked area, and the selected action has the highest probability within the unmasked area according to the continuous probability distribution; Generate a set of trajectories based on repeated selection actions; and The training progress of the neural network is evaluated based on the reward values ​​calculated from the set of trajectories.

10. The method of claim 1, wherein updating the mask further comprises: The mask is updated after each macro is placed.

11. A system operable for training a neural network for macro-layout, comprising: Processing hardware; as well as A memory, coupled to the processing hardware, is used to store information about the neural network and macros to be placed on the chip, wherein the processing hardware is operable to: Update the mask to mark invalid areas occupied by macros already placed on the chip canvas; Through the policy network of the neural network, a summary statistics of a two-dimensional continuous probability distribution is generated in the continuous action space for a given state of the chip canvas. An action is selected based on the continuous probability distribution, and the selected action corresponds to the coordinates in the unmasked area; and Generate a trajectory consisting of (state, action) pairs, wherein the final state in the trajectory corresponds to the completion placement of the macro.

12. The system of claim 11, wherein, The summary statistics include at least the mean and standard deviation of each dimension of the continuous probability distribution.

13. The system of claim 11, wherein the policy network outputs four real numbers as the summary statistics to characterize the continuous probability distribution.

14. The system of claim 11, wherein the neural network is trained in the continuous action space.

15. The system of claim 11, wherein, The neural network is pre-trained in a discrete action space, where the coordinates corresponding to a given action are rounded to a fixed-length number, and the neural network is further trained in the continuous action space.

16. The system of claim 11, wherein the processing hardware further operates to: The neural network is pre-trained in a discretized action space within the repeatedly updated unmasked region, wherein the coordinates corresponding to a given action in the discretized action space are rounded to a fixed-length number; and The pre-trained neural network is fine-tuned through self-play reinforcement learning in the continuous action space within the repeatedly updated unmasked region.

17. The system of claim 16, wherein, The self-play reinforcement learning uses the masked KR-DL-UCT algorithm.

18. The system of claim 11, wherein when the action is selected, the processing hardware further operates: Based on the continuous probability distribution, one or more actions are randomly sampled from the continuous action space until an action is located in the unmasked area; Generate a set of trajectories based on repeated sampling actions; and The parameters of the neural network are updated based on a loss function calculated from the set of trajectories.

19. The system of claim 11, wherein when the action is selected, the processing hardware further operates: The action is selected from the continuous action space within the unmasked area, and the selected action has the highest probability within the unmasked area according to the continuous probability distribution; Generate a set of trajectories based on repeated selection actions; and The training progress of the neural network is evaluated based on the reward values ​​calculated from the set of trajectories.

20. The system of claim 11, wherein, The processing hardware further operates to update the mask after each macro is placed.

Images