System and method for optimal placement of interactive objects over continuous (or discrete or mixed) domain

By reinforcing the optimization of configuration search in system design by reinforcing learning agents and deep neural networks, the problem of inefficiency in large-scale configuration space is solved, and efficient and optimal placement of interactive objects is achieved.

CN119032354APending Publication Date: 2024-11-26HUAWEI TECH CO LTD

Patent Information

Application Number
CN202280094879.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-04-28
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In system design, especially in systems with a large number of configuration dimensions and options, the calculation time for finding the optimal configuration is inefficient, and the optimal placement of interactive objects is not computationally efficient on the domain, making it difficult to effectively search and evaluate.

Method used

Using reinforcement learning (RL) agents, using deep neural networks (DNNs) and bidirectional gating recursive units (Bi-GRUs) as well as self-attention or converter layers, generate a distributed set for optimal placement of interactive objects, through rewards The mechanism updates network parameters and improves search efficiency.

Benefits of technology

It significantly improves the computer operation performance and memory utilization efficiency of finding the optimal configuration in large-scale configuration spaces, and simplifies the optimal placement of interactive objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119032354A_ABST
    Figure CN119032354A_ABST
Patent Text Reader

Abstract

In accordance with an embodiment, a reinforcement learning (RL) agent running on at least one processor receives constant inputs. The RL agent includes a deep neural network (DNN), and the DNN may include one or more layers of bi-directional gated recursive units (Bi-GRUs) and one or more self-attention or converter layers. And the RL intelligent agent outputs N distribution sets. Each of the N distribution sets corresponds to a different object of the N interaction objects and includes a first distribution in a first direction and a second distribution in a second direction. The RL agent generates a batch of sample configurations based on the N distribution sets. Each sample configuration in the batch of sample configurations includes identified or selected locations of N interaction objects. The RL agent outputs the batch of sample configurations to an evaluator. And the RL agent updates parameters of the DNN based on the loss function.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates generally to artificial intelligence (AI), and in particular embodiments to methods and apparatus for optimal placement of interactive objects on a domain. Background Art

[0002] Many system designs may involve a set of adjustable configurations. In the configuration of a system, there may be many configuration dimensions, where each dimension represents a different aspect of the configuration of the system and corresponds to a different configuration parameter for that aspect of the configuration. For each configuration dimension, there may be various available (discrete or continuous) options for that dimension. Due to the large number of configuration dimensions and / or the large number of available options in each dimension, some system designs have very large configuration spaces. For example, for a system whose configuration has 20 dimensions and 8 options per dimension, the number of different configurations of the system may be as large as 20. 8 Furthermore, different configuration dimensions of the system can have different numbers of options. For example, for a discrete system with a fixed number of N dimensions, if each dimension has its own number of (fixed) options M i , then the number of possible configurations of the system can be

[0003] In these configuration spaces (for example, possible configurations) to find the optimal configuration that satisfies the combination goal can be technically challenging. 8 For a system with 20 possible configurations (e.g., 20 dimensions with 8 options per dimension), even if evaluating one configuration of the fully configured system through a combination of objectives would only require one second of computing time, evaluating all possible configurations one by one would require more than 800 years of computing time to find the optimal configuration among all possible configurations, which is inefficient or even infeasible. Furthermore, in many practical applications of system design, evaluating a fully configured system with one configuration directly or through simulation may take much longer than one second. Therefore, techniques are needed to improve the performance and efficiency of computer operations when searching for optimal configurations, especially in search spaces with a large number of dimensions or a large number of options within each dimension.

[0004] The placement of interactive objects on a domain (e.g., a macroelement or a microelement or a cluster of microelements) is a common problem known to be difficult to solve. Examples of placement problems include the placement of electronic components on a die (e.g., a domain) according to some given technology and connection constraints of the electronic components. The electronic components subject to placement may be macroelements, microelements, or a mixture of macroelements and microelements.

[0005] The optimal placement of interactive objects typically requires an external system to evaluate the proposed placement. The search space for the optimal placement of interactive objects may be huge. The number of macrocells and / or microcells (e.g., the number of dimensions) is large, and the selection (e.g., options) of the location of each cell on the domain (e.g., die) may be virtually endless. In this example, the die circuit may be configured in part by selecting the location of circuit elements, blocks, or devices. The location of a cell (e.g., macrocell or microcell) may be the location of the cell on the die. As a non-limiting example, the location may be represented as the coordinates of the cell in the die space (e.g., 2-D coordinates or coordinates with higher dimensions). In addition, macrocells may have various sizes and shapes. In addition, all cells may have very complex connection requirements.

[0006] Complete placement of large graphs of interacting objects (e.g., electronic netlists or hypergraphs) may be computationally impractical. Therefore, techniques are needed to improve the performance of computer operations and the efficiency of memory utilization when finding the optimal configuration (e.g., placement) of interacting objects (macrocells and / or microcells) over a continuous (or discrete or hybrid) domain. Summary of the invention

[0007] Technical advantages are generally achieved by embodiments of the present disclosure describing embodiment methods and apparatus for optimal placement of interactive objects on a continuous (or discrete or hybrid) domain.

[0008] According to an embodiment, a reinforcement learning (RL) agent running on at least one processor receives a constant input. The RL agent includes a deep neural network (DNN), and the DNN may include one or more layers of bidirectional gated recurrent units (Bi-GRU) and one or more self-attention or converter layers. The RL agent outputs N distribution sets. Each of the N distribution sets corresponds to a different object in the N interactive objects, and includes a first distribution in a first direction and a second distribution in a second direction. The RL agent generates a batch of sample configurations based on the N distribution sets. Each sample configuration in the batch of sample configurations includes the positions of the identification or selection of the N interactive objects. The RL agent outputs the batch of sample configurations to an evaluator. The RL agent updates the parameters of the DNN based on a loss function.

[0009] In some embodiments, the loss function may be calculated based on the reward. The reward may be generated based on the performance indicator output by the evaluator. In some embodiments, the RL agent may repeat the following operations for multiple iterations: receiving a constant input, outputting N distribution sets, generating a batch of sample configurations, outputting a batch of sample configurations, and updating parameters. The constant input may remain the same throughout multiple iterations. In some embodiments, the positions of the N interactive objects may include the positions of the N macro units, or a mixture of the positions of the M macro units and the initial centroids of the (NM) micro unit clusters. In some embodiments, the constant input may include a constant value, N different constant unique hot vectors, or N vectors each of which is all 1. In some embodiments, one or more self-attention or converter layers may receive outputs from one or more layers of Bi-GRU layers. The DNN may also include an affine layer that receives outputs from one or more self-attention or converter layers and outputs N distribution sets. In some embodiments, the loss function may be updated by a value estimation layer of a sequence mode. The value estimation layer of the sequence mode may be updated based on a batch of sample configurations. In some embodiments, the loss function may be updated by a value estimation layer of a branch mode, and wherein the value estimation layer of the branch mode is updated based on the task encoding tensor of the DNN. In some embodiments, the N distribution sets may be N categorical distribution sets for discrete 2-D or 3-D placement. In some embodiments, the N distribution sets may be used for continuous 2-D or 3-D placement. In some embodiments, the N distribution sets may be N Beta distribution sets.

[0010] In this way, embodiment techniques improve the performance of computer operations and the efficiency of memory utilization when finding optimal placement of interactive objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description in conjunction with the accompanying drawings, in which:

[0012] Figure 1A and Figure 1B shows general RL theory and algorithms according to some embodiments;

[0013] Figure 2 A possible application of RL for searching for optimal configurations in system design according to some embodiments is shown;

[0014] Figure 3 An improved application scheme of RL for searching for optimal configurations in system design according to some embodiments is shown;

[0015] Figure 4A shows an example architecture of weight resonance for fixed-length optimality search according to some embodiments;

[0016] Figure 4B shows more details of an example architecture for weighted resonance for fixed-length optimality search in accordance with some embodiments;

[0017] Figure 4C shows more details of an example architecture for weighted resonance for fixed-length optimality search in accordance with some embodiments;

[0018] Figure 4D shows more details of an example architecture for weighted resonance for fixed-length optimality search in accordance with some embodiments;

[0019] Figure 4E shows an example architecture of weight resonance for fixed-length optimality search in the case of N discrete configuration dimensions in accordance with some embodiments;

[0020] Figure 4F shows an example architecture of weight resonance for fixed-length optimality search in the case of N discrete configuration dimensions in accordance with some embodiments;

[0021] Figure 5 A flow chart showing a method for weight resonance for fixed-length dimensional optimal configuration search according to some embodiments;

[0022] Figure 6 shows a general scheme for placing interactive objects using the policy gradient family of RL algorithms according to some embodiments;

[0023] Figure 7 An example scheme for placing interactive objects using the policy gradient family of RL algorithms according to some embodiments is shown;

[0024] Fig. 8A shows an example system architecture for placing interactive objects using the policy gradient family of RL algorithms in accordance with some embodiments;

[0025] Figure 8B shows an example system architecture for placing interactive objects using the policy gradient family of RL algorithms in accordance with some embodiments;

[0026] Fig. 9 shows distribution graphs based on different example Beta distributions according to some embodiments;

[0027] Fig.10 An example scheme is shown for processing sample configurations for generation of batches according to some embodiments;

[0028] Fig.11 shows more details of the output of a DNN according to some embodiments;

[0029] Fig.12shows more details of the output of a DNN according to some embodiments;

[0030] Fig.13 A flow chart showing a method for placing an interactive object according to some embodiments is shown;

[0031] Fig.14 is a block diagram of a computing system that can be used to implement the devices and methods disclosed herein, according to some embodiments.

[0032] Corresponding reference numerals in the different figures generally refer to corresponding parts unless otherwise indicated.The figures are drawn to clearly illustrate the relevant aspects of the embodiments and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0033] The formation and use of embodiments of the present disclosure are discussed in detail below. However, it should be recognized that the concepts disclosed herein can be embodied in a variety of specific contexts, and the specific embodiments discussed herein are merely illustrative and are not intended to limit the scope of the claims. In addition, it should be understood that various changes, substitutions, and modifications may be made to this document without departing from the spirit and scope of the present disclosure as defined by the appended claims.

[0034] Searching for the optimal configuration of a combination that satisfies a goal in a large configuration space is technically challenging. If there is a way to evaluate a specific configuration based on some target metric, a reward-driven deep reinforcement learning (D-RL) system can be designed that uses one of a variety of common policy reinforcement algorithms to explore the configuration space. Embodiments of the present disclosure provide a set of D-RL system architectures that make this exploration of the configuration space effective in a fixed-time domain configuration space, thereby improving the performance of computer operations and the efficiency of memory utilization.

[0035] Many system designs have a fixed set of configuration dimensions. In one non-limiting example, the system is designed to determine the best possible placement of block macros defined in the netlist of a central processing unit (CPU). In the netlist, there may be a fixed number of block macros (e.g., dimensions, where each dimension corresponds to a different configuration parameter for configuring the location of different block macros), and the locations of these block macros (e.g., options for each dimension) need to be fixed before the global placer places all other (standard) cells. In another non-limiting example, the system is designed to determine the optimal size of the cache hierarchy in the chip. In the cache hierarchy, there may be multiple cache levels (e.g., L0, L1, L2, L3, etc.). Caches of different levels (e.g., dimensions, where each dimension corresponds to a different configuration parameter for configuring the size of different cache levels) are typically of different sizes, and there may be many size options (e.g., options for each dimension) for each cache. In yet another non-limiting example, the system is designed to determine the optimal amino acid sequence of a given length that will produce the best antibody protein against the virus. Typically there may be 20 amino acid choices (e.g., options per dimension), and the length of a protein is typically between 50 and 2000 amino acids (e.g., dimensions, where each dimension corresponds to a different configuration parameter for configuring a different amino acid choice). Proteins (e.g., amino acid sequences) can be evaluated by topographic biochemical tools / viral analysis / receptor (in this case, antibody) binding. However, evaluating every possible protein configuration to find the optimal protein configuration is computationally intensive.

[0036] The technical problem solved by the present disclosure is to find the optimal configuration (e.g., optimal design) for a given system. System models may be too complex to accurately define, describe, or model (e.g., how cache hierarchy affects CPU design efficiency for a given benchmark set). Furthermore, the combination of possible configurations may define a huge choice space (e.g., the placement problem of block macros defined in the above netlist or the possible proteins of length 100 that can be synthesized from 20 available amino acids).

[0037] To search for optimal solutions, reinforcement learning (RL) can be used. Reinforcement learning is a machine learning training method based on rewarding desired behaviors and / or punishing undesirable behaviors. Reinforcement learning agents are typically able to perceive and interpret their environment, take actions, and learn through trial and error. However, applying RL to the search for optimal solutions can itself be technically challenging.

[0038] The method of reinforcement learning can include taking actions (e.g., generating configurations), collecting rewards (i.e., evaluating configurations), and using a policy reinforcement algorithm to find the optimal action. The policy reinforcement algorithm used can be any policy reinforcement algorithm known in the art, such as REINFORCE, actor-critic (AC), advantage actor-critic (A2C), asynchronous advantage actor-critic (A3C), trust region policy optimization (TRPO), and proximal policy optimization (PPO).

[0039] Additionally, an artificial neural network (ANN) can be used to generate optimal actions (i.e., optimal configurations). Traditionally, an ANN requires input-output data pairs and updates the neural network during a training session using backpropagation of some differentiable loss function (depending on the input-output data set).

[0040] Figure 1A and Figure 1B The general RL theory and algorithms according to some embodiments are shown. Figure 1A As shown, the work material of RL can include an episode of state-action-state-reward sequence. The episode starts at the initial state S0 and ends at the terminal state S T (For example, a round has all dimensions configured, such as in Figure 1A From S0 to S T General RL theory and algorithms can be designed for episodes of arbitrary length.

[0041] like Figure 1B As further shown, in state S i-1 (1≤i≤T), take action A i , and the state changes to state S i The policy gradient family of RL algorithms relies on action estimation. The policy gradient family of RL algorithms attempts to use π(a i |s i-1 ) modeling, π(a i |s i-1 ) is in state s i-1 The action selected in will be a i The probability of i |s i-1,θ). Where θ represents the set of modeling parameters (e.g., weights in a neural network). The modeling parameters can be trained using one of the many members of the PG family of RL algorithms (e.g., REINFORCE, AC, A2C, A3C, TRPO, and PPO), with the goal of forming a maximum (i.e., maximizing the discounted sum of rewards) in each episode.

[0042] Figure 2 A possible application of RL for searching for optimal configurations in system design according to some embodiments is shown. A certain system design configuration exploration may produce the following solution. Figure 2 As shown, the action can be a configuration selection in each configuration dimension. For example, at state S0, the action can be to configure C1 to select the option value of configuration dimension 1; at state S1, the action can be to configure C2 to select the option value of configuration dimension 2; at state S i-1 The action can be to C i Configuration is performed to select option values ​​for dimension i (1≤i≤T) of the configuration, and so on. For system design, the number of configuration dimensions is fixed (e.g., Figure 2 The number of dimensions in the example shown is fixed, namely Figure 2 In each dimension, various options may be available.

[0043] Since the number of configuration dimensions is fixed, RL in fixed-horizon optimality search (e.g., fixed-length rounds) needs to process all rounds of the fixed duration (e.g., the number of dimensions, i.e., Figure 2 In a configuration, a given dimension can provide discrete or continuous options (e.g., actions).

[0044] However, some applications of RL (for example, Figure 2 The scheme described in the previous section may be technically difficult and complex. Rewards (which are used as a measure of design excellence) can only be evaluated after all configuration dimensions are configured (with one option available in each dimension). That is, only fully configured systems can be evaluated through system evaluation mechanisms (e.g., power / performance / area (PPA) in chip designs, which is applicable to chip designs with multiple possible configuration dimensions).

[0045] Figure 3An improved application scenario of RL for searching for an optimal configuration in a system design is shown. Since an episode (e.g., a configuration of all dimensions) has a fixed time domain (e.g., the number of dimensions or the number of configuration parameters is fixed), and since rewards are collected only at the end, embodiments in the present disclosure re-represent the application scenario of RL as a single-step composite step action. For this embodiment, different permutations of configuration options for different dimensions can be evaluated to determine which permutation yields the best reward. The re-represented application scenario improves the problem solution and enables more effective and efficient action probability estimation. As Figure 3 shown, the scenario improvement makes the RL system a single-step stateless episode. Figure 3 The single-step composite action 304 in T-1 is a combination that configures all dimensions (e.g., a combination of "configuring C1 for dimension 1, C2 for dimension 2, C3 for dimension 3, C4 for dimension 4... C T for dimension T - 1, and C Figure 1A and Figure 2 shown state S1, S2, S3... S T-1 ). In the improved application scenario shown in Figure 3 , there are only two states: the blank state 302 of unconfigured dimensions and the fully configured state 306 of configured all dimensions. At the blank state 302, the single-step composite action 304 is executed. Then, the state transitions to the fully configured state 306, and the reward R T is collected. Other application scenarios of RL (e.g., the scenario shown in Figure 2 ) need to consider the constraints between dimensions. In contrast, through the improved application scenario of RL in Figure 3 , if an earlier C i constrains C j (i < j), the embodiment technique ignores such a constraint by modeling the joint probability rather than the more complex conditional probability. The more complex conditional probability usually cannot be parametrically defined because the complex state modifications caused by each configuration choice (in the multi-action multi-step mode) result in a complex domain, making such parametric definition impossible and leading to arbitrary processing of a topologically complex probability domain (e.g., a 2-D placement domain with a large number of holes, including some overlapping domains). In a multi-step stateful method (especially in the continuous case), such a fragmented domain will also be arbitrarily discretized to provide a means for probability assignment, resulting in further loss of fidelity.

[0046] In many traditional RL application scenarios, there is an explicit input-output data set. In the case of embodiments of single-step compound actions (e.g., action 304), there is no input (e.g., no input is required) when it comes to optimality search. The goal of the problem solution here is to find the optimal configuration. Traditional RL libraries assume that multiple-step episodes (e.g., Figure 1A ), and there is no optimization for single-step compound action scenarios. The embodiments of the present disclosure focus on the scenario of single-step compound action RL. The embodiment technology can also solve the same type of problems as various multivariable multi-armed bandit techniques. The embodiment technology is still different from the solution of multivariable multi-armed bandit in many aspects. For example, the embodiment technology handles multiple dimensions. In addition, if the configuration dimension must be represented by a continuous domain, then this dimension alone will present an infinite-armed bandit. In addition, any configuration dimension (of multiple configuration dimensions) that only provides discrete options is each individually a multi-armed bandit.

[0047] Figure 4A An example architecture 400 for weight resonance for fixed-length optimality search is shown in accordance with some embodiments. The example architecture 400 includes a neural network 404 (e.g., an action policy neural network). The neural network 404 includes a node layer with a learnable parameter set θ (e.g., weights between nodes). For a blank state, the input 402 of the neural network 404 can be a constant. This means that the input 402 can remain the same and unchanged throughout the search for the optimal configuration. In some embodiments, the input 402 can be a single constant value (e.g., a non-zero number). In some embodiments, the constant input 402 can be a tensor of 1 or hot-1 (e.g., a set of bits where the only allowable value combinations are combinations with a single high (1) bit and all other bits are low (0)). In some other embodiments, the input 402 can be a single constant value (e.g., a non-zero number), and the neural network 404 can produce an output of the same size as the output produced by the 1 or hot-1 tensor.

[0048] The output 406 of the neural network 404 is a composite action probability density function (PDF) or the parameters of the PDF. A probability distribution gives the likelihood of each outcome of a random event (e.g., possible options for a dimension). The composite action PDF is a function that defines the probabilities of different possible events (e.g., the probabilities of different options for each different dimension). The composite action PDF in the output 406 can be represented as θ(a1, a2, ..., a T-1 ,a T ;θ), where a iis a possible option in configuration dimension i. Based on the composite action PDF, a batch of sample configurations 408 (e.g., candidate configurations) is generated. Each sample configuration in the batch of sample configurations is a full configuration sample of all T dimensions, and each dimension in the total T dimensions is configured with a corresponding dimension option. In some embodiments, a batch of sample configurations is generated based on the composite action PDF so that the distribution of configuration options of the dimensions in the batch of sample configurations is consistent with the composite action PDF π(a1, a2, ..., a T-1 ,a T ; θ)’s probability distribution.

[0049] The generated batch of sample configurations 408 can be sent to an evaluator 410 across the non-differentiable wall between systems. The evaluator 410 can evaluate each fully configured system (using the batch of sample configurations). The evaluator 410 can be any external simulation / evaluation system outside of the neural network 404 and separate from the neural network 404. The evaluator 410 can evaluate the fully configured system to find the optimal configuration for the system design. The actual use of the embodiment technology may require the evaluator 410 to be fast enough, depending on the time available for searching the optimal configuration and the complexity of the configuration space. The evaluator 410 can output a multi-dimensional performance indicator 412 (e.g., power / performance / area (PPA) when designing a chip system) for each fully configured system using the sample configurations in the batch of sample configurations. Then, a scalar reward 414 is generated based on the multi-dimensional performance indicator. For example, the multi-dimensional performance indicator 412 can be normalized and weighted to generate a single number as a scalar reward 414 (e.g., weight of power × normalized power + weight of performance × normalized performance + weight of area × normalized area).

[0050] Scalar reward 414 is used to calculate statistical loss function 416. Statistical loss function 416 can be calculated based on scalar reward 414 using any member of the policy gradient family of RL algorithms known in the art (including but not limited to REINFORCE, AC, A2C, A3C, TRPO, and PPO). A loss function is a function that maps the values ​​of one or more variables to real numbers that intuitively represent some cost associated with these values. The loss function is a function that evaluates how well a neural network (e.g., neural network 404) models a data set. If the modeling effect is not good, the number output by the loss function of the neural network may be greater than the number output by a neural network with better modeling. For backpropagation, the loss function can depend on the parameters ("weights") of neural network 404 (and can be a differentiable function of the parameters of neural network 404), so that these weights can be adjusted by backpropagation. Among these differentiable functions, scalar reward 414 plays the role of a permanent (non-differentiable) parameter, simply because the agent does not know the dependence of reward 414 on the configuration. In other words, the reward 414 is provided by the environment (e.g., an evaluator 410 evaluating a fully configured system (evaluator 410 may be a simulator)). Various objective functions (for maximization or minimization) may be defined by each policy gradient family of algorithms. The objective function (loss) may be evaluated algebraically using algebra supported by some automatic differentiation software library (e.g., TensorFlow, PyTorch, etc.). The parameter set θ of the neural network 404 may then be updated to increase the reward via the back-propagation 418 algorithm supported by the automatic differentiation library described above.

[0051] The above process (e.g., from input 402 to back propagation 418) may be repeated for multiple iterations to find the optimal configuration. The input 402 to the neural network 404 remains constant and unchanged throughout all iterations. In some embodiments, the number of iterations may be a preconfigured number. In some other embodiments, the process may stop when the scalar reward 414 is greater than or equal to a predetermined threshold level. The sample configuration that results in the highest value of the scalar reward 414 may be selected as a selectable configuration.

[0052] Figure 4BMore details of an example architecture 400 for weight resonance for fixed-length optimality search according to some embodiments are shown. As described above, the input 402 of the neural network 404 remains constant throughout the search for the optimal configuration. Even so, the neural network 404 can still learn. The loss of backpropagation (e.g., the loss calculated according to the statistical loss function 416) depends only on the reward of the generated configuration and the PDF / distribution / sampled output 408 of the neural network 404. Typically, the loss is defined so that if the overall discounted reward increases, the loss may decrease. Various policy gradient algorithms determine the extent of this update. Any policy gradient algorithm known in the art (e.g., REINFORCE, AC, A2C, A3C, TRPO, and PPO) can be used to calculate the loss, but the performance of some algorithms may be better than other algorithms. For example, it is generally believed that PPO is better than A3C, A3C is better than A2C, A2C is better than AC, and AC is better than REINFORCE. In addition, each of these algorithms has hyperparameters that determine the importance of various components in the objective function (or loss) represented by each algorithm, and the embodiments are not limited to the algorithms listed above. The neural network 404 can learn because the neural network parameters (e.g., θ) can be updated by stochastic gradient descent to minimize the loss function using a back-propagation algorithm. Thus, despite the constant input, the weights of the neural network 404 can be kept updated toward a joint distribution that produces an optimal configuration (e.g., resonance) using any policy gradient algorithm.

[0053] Figure 4C More details of an example architecture 400 for weight resonance for fixed-length optimality search according to some embodiments are shown. Embodiments in the present disclosure approximate the joint probability in the output 406 as a product of conditional distributions, and these conditional distributions can be parametrically approximated as marginal distributions for modeling these distributions because parameter dependencies are captured by one or more task abstraction layers included in the neural network 404. One or more task abstraction layers can use self-attention neural layers, recurrent neural network (RNN) neural layers, or multi-layer perceptron (MLP) neural layers to establish interdependencies in the internal encoding of the neural network 404 for parameter generator layers or neural network 404 learning. The neural network 404 can include one or more task abstraction layers to jointly model tasks across all available action components. In the limit case of optimality, where the joint distribution is approximated as a conditional distribution, the output 406 of the neural network 404 can result in conditional probabilities.

[0054] Figure 4DMore details of an example architecture 400 for weight resonance for fixed-length optimality search according to some embodiments are shown. The neural network 404 may include one or more task abstraction layers 422, a task encoding tensor 424, and an action distribution head 426. The one or more task abstraction layers 422 may be used to ensure that any distribution (joint) dependencies are captured for the specific configuration optimality search task for which the output 406 of the neural network 404 is generated, and that keeping the input 402 of the neural network 404 at a constant value helps the neural network 404 resonate to that state.

[0055] The task encoding tensor 424 can be used to generate action probability PDFs or parameters for modeling these action PDFs (embodiments in the present disclosure may assume that marginal probability parameters can be used at this stage because the dependencies of marginal probability parameters can be captured in the early stages of the neural network 404 by one or more task abstraction layers 422 (e.g., neck) and one or more action distribution layers 426 (e.g., head). The one or more task abstraction layers 422 may include any or all of self-attention neural layers, RNN (e.g., gated recurrent unit (GRU), long short-term memory (LSTM)) neural layers, or MLP neural layers. The number of layers and the composition of the layers in the one or more task abstraction layers 422 may depend on the complexity of the task and may depend on trade-offs related to the training complexity of more complex networks (e.g., self-attention layers with a larger number of heads or deeper networks). The number of layers and the composition of the layers in the one or more task abstraction layers 422 may also be related to numerical rounding errors and back-propagation signal loss in very deep networks. Furthermore, the action distribution layer 426 may include any or all of a self-attention neural layer, an RNN (e.g., GRU, LSTM) neural layer, or an MLP neural layer.

[0056] Policy gradient algorithms (e.g., REINFORCE, AC, A2C, A3C, TRPO, and PPO) can be predicted by estimating advantages via a value estimator. For advantage estimation, the value estimation tier can be attached to one or more task abstraction layers 422 (e.g., one or more value estimation layers 432 for branching mode), or more directly to the sample configuration 408 (one or more value estimation layers 434 for sequence mode). One or more value estimation layers 432 for branching mode or one or more value estimation layers 434 for sequence mode may include any or all of self-attention neural layers, RNN (e.g., GRU, LSTM) neural layers, or MLP neural layers. In some embodiments, the value estimation tier (e.g., value estimation layer 432 or value estimation layer 434) can also be its own independent parallel neural network, where independent constant inputs are used to estimate the value of the optimal configuration through a similar weight resonance scheme described in the present disclosure. The two networks can be connected at the loss calculation node, and both can be updated (toward optimality resonance) through the same back-propagation mechanism mentioned above.

[0057] If the value estimation layer is attached to one or more value estimation layers 434 of the sequence mode, the value estimation layer can have its own independent loss function, which helps the value estimation layer estimate the reward for any specific configuration. Therefore, in the sequence mode, there can be two loss functions. One loss function (e.g., the statistical loss function 416) is used for the policy estimator, and the other loss function (not shown) is used for the value estimation layer.

[0058] If the value estimation layer is attached to one or more value estimation layers 432 of the branching mode, the two loss functions can be combined into one loss function (e.g., the statistical loss function 416) for backpropagation. There is a difference in how the loss function is used in these two modes because in the sequence mode, the value estimation layer depends on the configuration rather than the strategy for generating the configuration or task abstraction. On the other hand, in the branching mode, the value estimation layer can produce a rough estimate of the most likely optimal reward through multiple iterations.

[0059] Figure 4E shows the N discrete configuration dimensions (N may be a combination of Figure 3 Example architecture 450 of weighted resonance for fixed length optimality search in the case of T). FIG. 4A to FIG. 4D The described embodiment technology can be applied to Figure 4E The example architecture 450 supports the case where all configuration dimensions are categorical configuration dimensions and shows the number of parameters that need to be generated by the policy estimation head 425. For the example architecture 450, it is necessary to know the number of parameters for each dimension (from m1 to m2). N) is the number of discrete options in .

[0060] In some embodiments, the configuration dimensions (in the N-dimensional configuration) may include any combination of discrete and / or continuous distributions. Figure 4E The configuration dimension (not shown) can be modeled as a parametric distribution, such as Gaussian, Beta, etc., where the distribution is appropriately selected for the continuous search of the dimension. For example, if the configuration dimension has bounded continuous options, an embodiment can use a Beta distribution to model the configuration dimension. If the dimension has unbounded options (which may be uncommon), an embodiment can use a Gaussian distribution to model the dimension.

[0061] like Figure 4E As shown, the input 402 of the neural network is a one-hot constant input. In other embodiments, the input 402 of the neural network can be all 1. In other embodiments, the input 402 of the neural network can be a constant value (e.g., a non-zero number). The neural network can include a task encoder 423 (e.g., one or more task abstraction layers 422) and a policy estimation head 425 (e.g., one or more action distribution layers 426).

[0062] The output 406 of the neural network is a discrete distribution of attributes (action probabilities). Based on the discrete distribution of attributes, a batch of sample configurations 408 can be generated. The batch of sample configurations 408 is then sent to an evaluator for evaluation. The evaluator (e.g., evaluator 410) can be a multi-objective reward generator 411. The multi-objective reward generated from the generator 411 can be transformed (e.g., by a weighted sum of all objective indicators) to generate a single (real) reward 415 (e.g., a scalar reward 414), which is used by the policy execution 419 (e.g., a statistical loss function 416 and back propagation 418) to update the parameters (e.g., weights) of the neural network. The weighting can be based on a linear combination of the objectives (e.g., based on ) to generate rewards. For L targets O1, O2, ... L , w j Indicates that it corresponds to the target O j The weight of j Indicates that it corresponds to the target O j In addition, Figure 4E As shown, an estimation hierarchy (eg, one or more value estimation layers 432 of a branching pattern) is attached to the task encoder 423 .

[0063] Figure 4F An example architecture 452 of weight resonance for fixed-length optimality search in the case of N discrete configuration dimensions is shown in accordance with some embodiments. Figure 4F The example architecture in 452 is similar to Figure 4E, except that an estimation layer (eg, one or more value estimation layers 434 for sequence patterns) is attached to the sample configuration 408.

[0064] Figure 5 A flow chart of a method 500 for weight resonance of an optimal configuration search for a fixed-length dimension according to some embodiments is shown. The method 500 may be performed or executed by one or more processing units of a computing device. Examples of processing units include, but are not limited to, a graphics processing unit (GPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an artificial intelligence (AI) accelerator, or a combination thereof. The method 500 may also be performed or executed by a routine, a subroutine, or a software module executed by one or more processing units. The method 500 may also be performed or executed by a combination of hardware and software. The coding of the software for performing or executing the method 500 is entirely within the scope of the present disclosure for those of ordinary skill in the art. The method 500 may include more or fewer operations than those shown and described, and may be performed or executed in a different order. The computer-readable code or instructions of the software executable by one or more processing units may be stored on a non-transitory computer-readable medium such as a memory of a computing device.

[0065] Method 500 begins at operation 502, where a neural network running on at least one processor receives a constant input for a configuration design requiring N dimensions. At operation 504, the neural network outputs N probability distributions for the N dimensions, each of the N probability distributions for a different dimension in the N dimensions. At operation 506, at least one processor generates a batch of sample configurations for the configuration design based on the N probability distributions. Each sample configuration in the batch of sample configurations corresponds to a different full configuration of the system (e.g., each of the N dimensions for configuring the system is configured with a configuration option). At operation 508, at least one processor outputs the batch of sample configurations to an evaluator external to the neural network. At operation 510, at least one processor updates parameters of the neural network based on a loss function.

[0066] In some embodiments, at least one processor may repeat the following operations for multiple iterations: receiving a constant input, outputting N probability distributions, generating a batch of sample configurations, outputting the batch of sample configurations, and updating parameters, wherein the constant input remains the same throughout multiple iterations. In some embodiments, the constant input may include a single constant value, N different constant unique hot vectors, or N vectors each of which is all 1. In some embodiments, receiving a constant input and outputting N probability distributions may be stateless. In some embodiments, updating the parameters of the neural network may include: updating the weights of the neural network. In some embodiments, the N probability distributions may be joint probability distributions, so that all probability distributions in the N probability distributions depend on each other. In some embodiments, a loss function may be calculated based on a reward, and the reward is generated based on a performance indicator output by an evaluator. In some embodiments, the loss function may be updated by a value estimation layer in a sequence mode, and wherein the value estimation layer in the sequence mode is updated based on a batch of sample configurations. In some embodiments, the loss function may be updated by a value estimation layer in a branch mode, and wherein the value estimation layer in the branch mode is updated based on a task encoding tensor of the neural network. In some embodiments, the N probability distributions include N discrete probability density function (PDF) distributions, N continuous PDF distributions, or a mixture of M discrete PDF distributions and (NM) continuous PDF distributions.

[0067] Embodiments of the present disclosure provide techniques for training a policy gradient algorithm that is stateless and relies on weight resonance (with constant inputs) to perform optimality search in a multidimensional design configuration space. A design configuration can be any aspect of a system design that can be represented by discrete / categorical or continuous variables. A category is any set of choices that can be indexed by a fixed set of indices (the probability space of that category).

[0068] The policy gradient algorithm using the embodiment technology is improved by a single-step composite action strategy, and an architectural framework is provided to implement a weight resonance model to generate policy distribution and value estimates used in the policy gradient algorithm. Compared with other known methods (e.g., modeling the complete Pareto front), the embodiments of the present disclosure focus on searching for the optimal configuration on the segment of the front closest to the surface parallel to the desired operating surface. The operating surface is a parallel (L-1) dimensional surface in an L-dimensional multi-objective indicator space, and is based on the multi-objective indicator O1 to O L Functions that are transformed (e.g., by weighted sum combination) into a single reward value are efficiently generated. As the reward value (e.g., weighted sum) varies, such functions define parallel surfaces in the L-dimensional space of target metrics.

[0069] Embodiments of the present disclosure solve the technical problem of searching for optimal configurations where the dimensions of a configuration are fixed and the configuration options selected within each dimension are also fixed, with each possible option within each dimension having a discrete or continuous parameter probability distribution. Searching for optimal configurations becomes a particularly difficult technical problem when the cardinality of these dimensions in the configuration (and the options within each dimension) grows, or when a high-dimensional joint probability space emerges. For example, if there are N configuration dimensions and there are M discrete options in each configuration dimension, then such a configuration space may have M N possible configurations to choose from. More generally, for N configuration dimensions with discrete options (configuration dimension i has M i options), the number of possible configurations can be

[0070] In addition, if the options within a given dimension are from a continuous domain rather than from a discrete domain, the search space provides countless possibilities. Using the embodiment technology, options from the continuous domain can be solved by using parameter distributions (e.g., Gaussian distributions for unbounded domains and or Beta distributions for bounded domains). Therefore, the embodiment technology is still applicable to searching for optimal configurations in continuous space separately from discrete space or jointly with discrete space (a mixture of spaces (e.g., options in certain dimensions in the configuration)). According to an embodiment, the choices within each continuous dimension can be modeled by a parameter probability distribution (with a finite number of parameters). For example, in such a parameter probability distribution that can be defined with a finite set of parameters but can produce countless sets of options, Gaussian distributions and Beta distributions can be used.

[0071] To solve the technical problem of high combinatorial optimality search, embodiments of the present disclosure provide deep RL, weight resonance systems and methods that enable solving the high combinatorial optimality search problem as a stateless single-step compound action RL problem. The optimal configuration may be where the PDF has multidimensional maxima. These are the maximum likelihood optimal positions. In some embodiments, a selection probability ("action probability" or "strategy") is generated, π(a1, a2, ..., a T-1 ,a T; θ)) and value estimation (estimating the value of the optimal configuration) neural network does not receive changing inputs that depend on the state that changes with each selection of the option. Alternatively, the neural network that models the selection probability ("action probability" or "policy") and the optimal value estimation of the embodiment has a constant input. Through D-RL search, the neural network (using any of a variety of policy gradient algorithms known in the art) receives policy gradient feedback based on the reward estimate for the fully configured system generated in a composite action step. The policy network used for these policy gradient algorithms can be the same as the embodiment multidimensional option selection probability ("action probability" or "policy") network (e.g., neural network 404). The value network of these algorithms can be the same as the embodiment multidimensional value estimation network. Therefore, both the action policy network (e.g., neural network 404) and the value estimation network (e.g., one or more value estimation layers 432 or one or more value estimation layers 434) can receive constant inputs.

[0072] Embodiment neural network systems (or any other parameter prediction mechanism capable of being tuned via gradient descent, back-propagation, etc.) may also include a (potentially common) task abstraction hierarchy prior to generating a final layer of action probability distributions and / or value estimates. The action policy neural network (e.g., neural network 404) may generate candidate configurations via probability estimation and sampling techniques.

[0073] The value network can either evaluate the multidimensional sample configuration or generate its estimate of the target value as the optimal value (depending on the task abstraction level). Here, the system attempts to associate the value (which can be a "commentator" in AC, A2C, and A3C), and can pre-estimate the reward signal generated by the evaluator / simulator when evaluating. The reward signal can be generated by the evaluator, which evaluates the generated candidate configuration. For each candidate configuration generated by sampling the multidimensional distribution generated by the action policy network (e.g., neural network 404), the reward generator can evaluate the configuration in a physical reality (simulated system or real system) environment and generate a reward signal (e.g., scalar reward 414). Then, the embodiment technology can use the reward signal in one of a variety of policy gradient algorithms (e.g., REINFORCE, AC, A2C, A3C, TRPO, and PPO, etc., which also depend on the configuration distribution and value estimation from the action policy network) to create a back-propagation signal that updates all weights in the action policy network (e.g., neural network 404).

[0074] In short, while the input of the action policy network is fixed to a constant, the weights of the embodiment neural network (e.g., neural network 404 and the value estimation neural network) can still be updated and resonate with the reward signal, which generates a back-propagation feedback signal for updating the weights of the embodiment neural network. In this way, the embodiment neural network can begin to learn a task abstraction (closer to the constant input) and converge to the optimal solution to the technical problem of efficiently and effectively searching for the optimal configuration by maximizing the reward using any of a variety of policy gradient algorithms known in the art.

[0075] Embodiments of the present disclosure use a weighted resonant neural network (e.g., neural network 404 and a value estimation neural network) that searches for an optimal configuration in a configuration space. Embodiment weighted resonant neural networks (e.g., neural network 404 and a value estimation neural network) may receive a constant input on the input side of the constant input and a reward signal from the output side, which enables the embodiment weighted resonant neural network to resonate the parameters of the network toward ultimately producing an optimal configuration. Embodiment techniques may be used with multi-dimensional objectives (e.g., Figure 4E and Figure 4F The targets O1, O2, O3, ... O shown L ). These objectives can be combined with corresponding weights to produce a single scalar objective (e.g., by Figure 4E and Figure 4F The targets O1, O2, O3, ... O shown L The operating plane is therefore essentially modeled by a weighted sum of objectives. By combining objective values ​​using a weighted sum (or other method), the embodiment technique actually searches a specific region of the Pareto front, and due to its data efficiency, the embodiment technique can search in a much larger combinatorial space than alternative methods that model the entire front.

[0076] In summary, compared to known alternative techniques using multi-step RL (for more complex but lower dimensional problems) or multi-armed bandits (for simpler problems), the embodiment technology utilizes single-step compound action RL. Since there are only two states, namely the initial state (e.g., the blank state 302) and the terminal state (e.g., the full configuration state 306), the embodiment technology does not need to model the state. The embodiment technology relies on updating the policy distribution and value estimation neural network (or other statistical estimator) through weight resonance, gradually generating a policy distribution that is likely to produce the optimal configuration near the area parallel to the operation plane, and optimizing the reward. In addition, compared with known techniques that discretize continuous space unnaturally (which reduces fidelity and greatly increases memory requirements during training), by using single-step compound actions (e.g., action 304), the embodiment technology does not need to model the probability on the segmented domain. Therefore, the embodiment technology does not need to discretize the continuous space. Instead, the embodiment technology relies on reward feedback to learn impermissible moves, which in multiple steps lead to the need for state modeling.

[0077] The above-mentioned embodiment weight resonance technique of finite time domain, compound action, single-step RL for searching optimal configuration can be applied to the placement of interactive objects, such as the placement of macro cells and / or micro cells on a physical plane (e.g., a die). The complete placement of a large graph of interactive objects (e.g., an electronic netlist or a hypergraph) may be computationally impractical. In some embodiments, a large macro (e.g., block type) cell is placed first. Then, the position of the smaller micro (standard cell or core type) cell is to be optimized by a global placement tool known in the prior art using pseudo-physical modeling, so that the micro cell moves to a stable position through a modeled force field / potential energy profile. DREAMPlace can be a non-limiting example of such a global placement tool. Therefore, the embodiments in the present disclosure provide techniques for finding the optimal placement of macro cells. In addition, the embodiments in the present disclosure can also provide techniques for finding the optimal initial position (e.g., the optimal initial center of mass) of a micro cell. Based on the optimal initial position (e.g., the optimal initial center of mass) of the micro cell, the micro cell can be further driven to the final position of the micro cell by an external global placement / evaluation system (e.g., DREAMPlace).

[0078] Figure 6A general scheme for placing interactive objects using a policy gradient family of RL algorithms according to some embodiments is shown. An RL agent 602 (including a neural network such as a deep neural network (DNN)) is used to generate sample configurations. Each sample configuration 604 configures the placement of an interactive object. Each sample configuration 604 is sent to an evaluator 606. The evaluator 606 can run in parallel with the RL agent 602 (e.g., when the RL agent 602 generates a new sample configuration, the evaluator 606 can evaluate the previously generated sample configuration). After evaluating the sample configuration, the evaluator 605 generates a reward for the sample configuration. Each reward 608 can be generated based on multiple performance indicators (e.g., PPA) generated by the evaluator 606. The policy gradient RL algorithm can use the reward 608 to update the neural network of the RL agent 602. The policy gradient RL algorithm can be any policy gradient RL algorithm known in the art, such as REINFORCE, AC, A2C, A3C, TRPO, and PPO.

[0079] Figure 7 An example scheme for placing interactive objects using a policy gradient family of RL algorithms according to some embodiments is shown. RL agent 702 includes a deep neural network (DNN). DNN can be used to generate sample configurations (e.g., placement of macro units). For example, DNN can generate a single-step composite action PDF, such as the single-step composite action PDF in output 406 above. The single-step composite action PDF can include N probability distributions corresponding to N macro units (e.g., N dimensions). Then, a batch of sample configurations (e.g., placement of N macro units) can be generated based on the N probability distributions. Each sample configuration in the sample configuration can provide a placement configuration for N macro units. In some embodiments, a batch of sample configurations is generated based on the composite action PDF, so that the distribution of configuration options (e.g., positions of macro units) in N dimensions (e.g., N macro units) in the batch of sample configurations matches the probability distribution (e.g., N probability distributions) of the composite action PDF. Each sample configuration (e.g., sample configuration 704) configures the placement of macro units 712. Sample configuration 704 can also include an initial position (e.g., initial centroid) of micro units 714. In some embodiments, the initial positions (e.g., initial centroids) of the microcells 714 may all share the same position, e.g. Figure 7The center position of the domain (e.g., the tube core) shown. In some embodiments, the microcell 714 can be grouped into one or more microcell clusters, where all microcells in the same cluster share the same initial position (e.g., the initial center of mass). The RL agent 702 can use its DNN to generate an optimal initial position (e.g., the optimal initial center of mass) for each microcell cluster in the one or more microcell clusters (e.g., the number of dimensions of the DNN can also increase the number of microcell clusters). For example, if there are M macrocells and all microcells are grouped into K microcell clusters, the DNN of the RL agent 702 may have a total of (M+K) dimensions (N=M+K), and the single-step composite action PDF output by the DNN may include M+K probability distributions (M probability distributions corresponding to the M macros, and K probability distributions corresponding to the K microcell clusters).

[0080] A batch of sample configurations (e.g., sample configuration 704) may be intermediate sample configurations, and they are not final sample configurations. Rather, a batch of sample configurations are floor plan files that determine and fix the positions of macro cells. However, the positions of micro cells may be further adjusted. For example, a global placer tool (e.g., DREAMPlace) 706 known in the art may use the initial positions (e.g., initial centroids) of one or more micro cell clusters as the starting point of the micro cells, and find the optimal final positions of all micro cells to generate a sample configuration, such as sample configuration 708. The global placer tool 706 may generate sample configuration 708 based on standard placement solutions known in the art and calculations based on performance metrics (e.g., half-perimeter wire-length (HPWL), overlap, and density). As Figure 7 As shown, the position of macrocell 712 in sample configuration 708 remains the same as the position of macrocell 712 in sample configuration 704. Because global placer tool 706 has changed the position of microcell 714 from its initial position (e.g., initial centroid) in sample configuration 704, the position of microcell 714 in sample configuration 708 may be different from the position of microcell 714 in sample configuration 704.

[0081] A batch of final sample configurations (e.g., sample configuration 708) is sent to the evaluator ( Figure 7) to generate rewards based on the evaluation of a batch of final sample configurations. The reward corresponding to the sample configuration can be calculated as a real-valued function of a performance metric (e.g., power / performance / area (PPA) when designing a chip system). The real-valued function can be a weighted sum of performance metrics. The sign used for each metric can depend on whether they are penalties / costs or rewards / benefits. Then, the parameters (e.g., weights) of the DNN in the RL agent can be updated using a policy gradient algorithm (e.g., REINFORCE, AC, A2C, A3C, TRPO, and PPO) based on the statistical distribution and rewards of the generated samples.

[0082] Fig. 8A An example system architecture 800 for placing interactive objects using the policy gradient family of RL algorithms according to some embodiments is shown. The system architecture 800 may use the above-described embodiment weight resonance technique for finite-horizon, compound-action, single-step RL.

[0083] For the blank state, the input 802 to the DNN 804 can be a constant. That is, the input 802 can remain the same and unchanged throughout the search for the optimal configuration. In some embodiments, the input 602 can be a single constant value (e.g., a non-zero number). In some embodiments, the constant input 602 can be a 1- or 1-hot tensor (e.g., Fig. 8A shown).

[0084] In some embodiments, where the input is a single constant vector value for encoding, the neural network (e.g., DNN 804) may include a decoder architecture that expands the single constant input value into a larger activation tensor before following other network components. In other embodiments, where the constant tensor input consists of 1 or unique heat, a converter architecture or a recurrent neural network (e.g., a gated recurrent unit (GRU)) may be used immediately. Therefore, DNN 804 may include one or more layers of bidirectional gated recurrent units (bi-directional gated recurrent unit, Bi-GRU) 806. Bi-GRU is a sequence processing model consisting of two GRUs, one of which receives an input in the forward direction and the other receives an input in the backward direction. Bi-GRU is a bidirectional recurrent neural network with only an input gate and a forget gate. GRU is part of a specific model of a recurrent neural network that is intended to perform machine learning tasks associated with memory and clustering using connections made through a series of nodes. The weights between the nodes in the layers of Bi-GRU 806 may be the same. For example, from l1 to l2, l2 to l3, ..., l i Toi+1 ,……,l n-1 To n The weights and n To r n-1 ,……,r i To r i-1 , ..., r3 to r2, and r2 to r1 can all have the same weights. When there are multiple layers of Bi-GRU, these Bi-GRU layers can be stacked. For example, the output of the first layer of Bi-GRU can be the input of the second layer of Bi-GRU, and so on. The output of the last Bi-GRU layer can be a network consisting of nodes h1, h2, ..., h n The input of the head layer 808.

[0085] The output of the head layer 808 can be received as an input to the self-attention layer 810. The self-attention layer 810 can take n inputs and generate n outputs. The self-attention layer 810 enables the inputs to interact with each other ("self") and find objects that the inputs should pay more attention to ("attention"). There may be multiple attention heads. In addition, in some embodiments, a transformer neural architecture (e.g., one or more transformer layers) can be used for self-attention. The self-attention layer 810 or one or more transformer layers can be used to capture potential task (internal object interaction) encodings to feed into the following layers. The output of the self-attention layer 810 can be an aggregation of these interactions and attention scores. For DNN 804, the output of the self-attention layer 810 can be a single-step composite action PDF 812 including N beta distribution sets. The N distribution sets can be N beta distribution sets. Each of the N beta distribution sets corresponds to a different interaction object among the N interaction objects (e.g., N macro units, or a mixture of N macro units and micro unit clusters). Each beta distribution set may include a first beta distribution (e.g., (α x ,β x )) and a second beta distribution in a second direction (e.g., the y-axis) (e.g., (α y ,β y )).

[0086] In some embodiments, DNN 804 may also include an affine layer (e.g., a multi-layer perceptron (MLP) strategy). In an affine layer, all nodes included are connected to all nodes of subsequent layers. The affine layer may take the output from the self-attention layer 810, and then the affine layer may output a single-step composite action PDF 812 comprising a set of N beta distributions.

[0087] Based on the single-step composite action PDF 812 including the N distribution sets, a batch of sample configurations 814 is generated. The distribution of configuration options (e.g., positions of macrocells and / or microcell clusters) in the N dimensions (e.g., N macros, or a mixture of N macrocells and microcell clusters) in the batch of sample configurations matches the probability distribution (e.g., N probability distributions) of the composite action PDF 812. If the sample configuration includes initial positions (e.g., initial centroids) of the microcell clusters, a method such as reference can be used. Figure 7 The depicted global placer 706 further adjusts the batch of sample configurations 812 .

[0088] Next, the batch of sample configurations 812 is sent to an external evaluator (e.g., a multidimensional reward generator 816) for evaluation to generate a reward based on the evaluation of the batch of sample configurations 812. The reward corresponding to the sample configuration (e.g., reward 818) can be calculated as a real-valued function of a performance indicator (e.g., power / performance / area (PPA) when designing a chip system). The real-valued function can be a weighted sum of performance indicators. The sign used for each indicator depends on whether they are penalties / costs or rewards / benefits. Then, the parameters (e.g., weights) of the DNN 804 in the RL agent can be updated using a policy gradient algorithm 820 (e.g., REINFORCE, AC, A2C, A3C, TRPO, and PPO) based on the statistical distribution and rewards of the generated samples. The policy gradient algorithm can be predicted by estimating the advantage via a value estimator. For advantage estimation, the value estimation hierarchy can be attached to one or more value estimation layers 822 of the branching pattern. The one or more value estimation layers 822 of the branching pattern can be the same or similar to the one or more value estimation layers 432 of the branching pattern described above.

[0089] Figure 8B An example system architecture 850 is shown for placing interactive objects using the Policy Gradient family of RL algorithms in accordance with some embodiments. Figure 8B The example system architecture 850 in is similar to Fig. 8A , except that the estimation layer (e.g., one or more value estimation layers 824 of the sequence pattern replaces the one or more value estimation layers 824 of the sequence pattern Fig. 8A One or more value estimation layers 822 of the branching pattern in are attached to the sample configuration 818.

[0090] Fig. 9 Distribution graphs based on different example Beta distributions are shown in accordance with some embodiments.

[0091] Fig.10An example scheme of a sample configuration for processing batch generation according to some embodiments is shown. The parallel environment 1002 decomposes B batches of macro placements and distributes them to B controllers 1004. Each of the B controllers 1004 can update a batch of floor plan files for its instance. Each controller can run a global placer (e.g., DREAMPlace) docker with exactly the same configuration file (excluding placement changes). B runners 1006 can run multiple global placers (e.g., DREAMPlace) instances, where multiple macro placements are indicated in a batch of placements. Each controller can collect reward information from each batch of B batches that the B runners 1006 have run. The controller 1004 can then return a batch of rewards to the RL system in an order corresponding to the batch of placements through a reward return queue. Details of a sample configuration for processing batch generation can be found in international application No. PCT / US2021 / 064634, filed on December 21, 2021, entitled “Parallel Processing Using A Memory-based Communication Mechanism,” the entire contents of which are incorporated herein by reference.

[0092] Fig.11 More details of an output of a DNN (e.g., output 812 of DNN 804) according to some embodiments are shown. The output of the DNN can be a single-step composite action PDF comprising N distribution sets. The N distribution sets can be N Beta distribution sets. Each of the N Beta distribution sets corresponds to a different dimension in the N dimensions (N interacting objects (e.g., N macrocells, or a mix of N macrocell and microcell clusters)). Each dimension (e.g., each interacting object) can have its own Beta distribution set (e.g., one Beta distribution set for each axis of the bounded region of the die). For example, dimension 0 can have a Beta distribution in the x-direction (α x0 ,β x0 ) and the Beta distribution in the y-axis direction (α y0 ,β y0 ), dimension 1 can have a Beta distribution in the x-axis direction (α x1 ,β x1 ) and the Beta distribution in the y-axis direction (α y1 ,β y1 )... Dimension i can have a Beta distribution in the x-axis direction (α xi ,β xi ) and the Beta distribution in the y-axis direction (α yi ,β yi ) ... and dimension (N-1) can have a Beta distribution in the x-direction (αx(N-1) ,β x(N-1) ) and the Beta distribution in the y-axis direction (α y(N-1) ,β y(N-1) ).

[0093] After sampling from a beta distribution on the interval (0,1) distribution in each direction (x and y), the full length of the die (in the x and y dimensions) can be scaled to find the physical placement. For the physical placement, (0,1,2,...,M-1) can be the index of the (large) macrocells of the M block types. There are two possible cases for the physical placement.

[0094] In the first case, the positions of all standard cells (e.g., microcells) can be initialized to a single position (e.g., Figure 7 Here, since DNN does not involve generating the beta distribution of microcells, M=N.

[0095] In the second case, if Fig.12 As shown, the N beta distribution sets output by the DNN may include M macro units (e.g., D0, D1, ..., D M-1 ) is a collection of M Beta distributions at fixed positions. DNN can also include (NM) micro-unit clusters (e.g., D M , D M+1 ,……,D N-1 ) of the initial positions (e.g., the centroid of the initial positions of the standard cells, around which the standard cells can be distributed according to a certain Gaussian distribution). For each cluster, the global placer tool can use the initial position of the cluster (e.g., the initial centroid) and also drive the microcells in the cluster to the final optimal position of the microcells. The optimal position can be a position that reduces the linear combination of the indicators produced by the global placer. For the second case, two MLPs can be used. One MLP can be used to generate distribution parameters for (larger) macrocells of M block types based on the lower layer encoding, and the other MLP can be used to generate distribution parameters for NM standard cell clusters (e.g., microcell clusters). Here, the initial position of each cluster in the microcluster can be such a centroid, and the microcells belonging to the cluster will be randomly distributed around the centroid using a distribution centered on the centroid (e.g., a Gaussian distribution).

[0096] In the second case, a graphical neural network (GNN) encoding on the chip's netlist can be used to characterize K clusters of standard cells (e.g., microcells). The GNN encoding can be independent of and prior to global placement, and is used as a technique to cluster standard cells into K clusters.

[0097] Fig.13A flowchart of a method 1300 for placing an interactive object according to some embodiments is shown. The method 1300 may be performed or executed by one or more processing units of a computing device. Examples of processing units include, but are not limited to, a graphics processing unit (GPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an artificial intelligence (AI) accelerator, or a combination thereof. The method 1300 may also be performed or executed by a module of a routine, a subroutine, or software executed by one or more processing units. The method 1300 may also be performed or executed by a combination of hardware and software. The coding of the software for performing or executing the method 1300 is entirely within the scope of the present disclosure for those of ordinary skill in the art. The method 1300 may include more or fewer operations than those shown and described, and may be performed or executed in a different order. The computer-readable code or instructions of the software executable by one or more processing units may be stored on a non-transitory computer-readable medium such as a memory of a computing device.

[0098] Method 1300 begins at operation 1302, where a reinforcement learning (RL) agent running on at least one processor receives a constant input. The RL agent includes a deep neural network (DNN), and the DNN may include one or more layers of bidirectional gated recurrent units (Bi-GRU) and one or more self-attention or transformer layers. At operation 1304, the RL agent outputs N distribution sets. Each of the N distribution sets corresponds to a different object in the N interactive objects and includes a first distribution in a first direction and a second distribution in a second direction. At operation 1306, the RL agent generates a batch of sample configurations based on the N distribution sets. Each sample configuration in the batch of sample configurations includes the positions of the identification or selection of the N interactive objects. At operation 1308, the RL agent outputs the batch of sample configurations to an evaluator. At operation 1310, the RL agent updates the parameters of the DNN based on the loss function.

[0099] In some embodiments, the loss function can be calculated based on the reward. The reward can be generated based on the performance indicator output by the evaluator. In some embodiments, the RL agent can repeat the following operations for multiple iterations: receiving a constant input, outputting N distribution sets, generating a batch of sample configurations, outputting the batch of sample configurations, and updating parameters. The constant input can remain the same throughout multiple iterations. In some embodiments, the positions of the N interactive objects may include the positions of N macro units, or a mixture of the positions of M macro units and the initial centroids of (NM) micro unit clusters. In some embodiments, the constant input may include a constant value, N different constant unique hot vectors, or N vectors each of which is all 1. In some embodiments, one or more self-attention or converter layers may receive outputs from one or more layers of Bi-GRU. The DNN may also include an affine layer that receives outputs from one or more self-attention or converter layers and outputs N distribution sets. In some embodiments, the loss function can be updated by a value estimation layer of a sequence mode. The value estimation layer of the sequence mode can be updated based on a batch of sample configurations. In some embodiments, the loss function can be updated by a value estimation layer of a branch mode, and wherein the value estimation layer of the branch mode is updated based on the task encoding tensor of the DNN. In some embodiments, the N distribution sets may be N categorical distribution sets for discretized 2-D or 3-D placement. In some embodiments, the N distribution sets may be used for continuous 2-D or 3-D placement. In some embodiments, the N distribution sets may be N Beta distribution sets.

[0100] Embodiments use the weighted resonant RL described in this disclosure to estimate policies (e.g., probabilities of composite actions) to discover optimal placement of (larger) macros of block types and optimal initial centroid locations for clusters of standard cells (the standard cells are then moved around the die by a “global placer,” which is an a priori system known in the art that drives standard cells (e.g., microcells) into low-potential configurations using various physical models. Standard cells (e.g., microcells) can be clustered using a priori encodings known in the art. Semi-supervised / unsupervised graph neural networks (GNNs) are used to obtain encodings of edges and nodes in the netlist.

[0101] Using weight resonance reduces the reliance on state maintenance for fixed-time composite actions, enabling efficient modeling of distributions, reducing memory utilization, and accelerating convergence of fixed-time composite action RL. Clustering of microcells ensures that all microcell nodes belonging to the same group can share the same initial centroid before being driven to their final optimal positions using any given global placer.

[0102] The present disclosure uses placement of electronic components on a die as an illustrative example. However, the embodiment techniques are not limited to placement of electronic components on a die, and the embodiment techniques can also be used for any similar problem that requires the generation of any discrete or continuous parameter probability density function ("PDF" or "distribution") D0 to D N-1 These functions are then sampled to generate configuration results (e.g., a batch of sample configurations), each of which can be evaluated and assigned a "reward" for improving the system that generates the PDF.

[0103] In addition, for the purpose of illustration, the output of the DNN includes a Beta distribution as a non-limiting example. In some embodiments, the Beta distribution can be used to place interactive objects in continuous and bounded domains. The output of the DNN can include other types of distributions known in the art (e.g., Poisson distribution or categorical distribution). For example, the output of the DNN can include a set of N categorical distributions for placing N interactive objects in a discrete domain.

[0104] Furthermore, for purposes of illustration, the present disclosure provides non-limiting examples of placing interactive objects in a (continuous or discrete) 2-D domain. The embodiment techniques are not limited to 2-D placement. The embodiment techniques may also be applied to placing interactive objects in a 3-D (z-axis is the third dimension) or 4-D (time is the fourth dimension) domain. For example, for 3-D placement, each set of beta distributions may include a first beta distribution (e.g., (α )) in a first direction (e.g., x-axis) x ,β x )), a second beta distribution in a second direction (e.g., the y-axis) (e.g., (α y ,β y )) and a third beta distribution in a third direction (e.g., z-axis) (e.g., (α z ,β z )). For a 4-D placement, each set of Beta distributions may include a first Beta distribution in a first direction (e.g., the x-axis) (e.g., (α x ,β x )), a second beta distribution in a second direction (e.g., the y-axis) (e.g., (α y ,β y )), a third beta distribution in a third direction (e.g., z-axis) (e.g., (α z ,β z )) and a fourth beta distribution in a third direction (e.g., time axis) (e.g., (α t ,β t)). For 3-D or 4-D placement, a Beta distribution or a mixture of a Beta distribution and other types of distributions can be used. For example, the time axis distribution can be parameterized as a Beta distribution with a bounded time window, or a Poisson distribution with a semi-bounded time window.

[0105] Fig.14 1400 is a block diagram of a computing system 1400 that can be used to implement the devices and methods disclosed herein according to some embodiments. A particular device may use all of the components shown or only a subset of the components, and the level of integration may vary from device to device. In addition, a device may contain multiple instances of components, such as multiple processing units, processors, memories, transmitters, receivers, etc. Computing system 1400 includes processing unit 1402. The processing unit includes a central processing unit (CPU) 1414, a memory 1408, and may also include a mass storage 1404, a video adapter 1410, and an I / O interface 1412 connected to a bus 1420.

[0106] Bus 1420 may be one or more of any type of several bus architectures, including a memory bus or storage controller, a peripheral bus, or a video bus. CPU 1414 may include any type of electronic data processor. Memory 1408 may include any type of non-transient system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or a combination thereof. In an embodiment, memory 1408 may include a ROM for use at startup and a DRAM for storing programs and data used when executing a program.

[0107] The mass storage 1404 may include any type of non-transitory storage device configured to store data, programs, and other information and make the data, programs, and other information accessible via the bus 1420. The mass storage 1404 may include, for example, one or more of a solid-state drive, a hard disk drive, a magnetic disk drive, or an optical disk drive.

[0108] The video adapter 1410 and the I / O interface 1412 provide interfaces for coupling external input and output devices to the processing unit 1402. As shown, examples of input and output devices include a display 1418 coupled to the video adapter 1410 and a mouse, keyboard, or printer 1416 coupled to the I / O interface 1412. Other devices may be coupled to the processing unit 1402, and more or fewer interface cards may be used. For example, a serial interface such as a Universal Serial Bus (USB) (not shown) may be used to provide an interface for external devices.

[0109] The processing unit 1402 also includes one or more network interfaces 1406, which may include wired links such as Ethernet cables or wireless links to access nodes or different networks. The network interface 1406 enables the processing unit 1402 to communicate with remote units via a network. For example, the network interface 1406 can provide wireless communication via one or more transmitters / transmit antennas and one or more receivers / receive antennas. In an embodiment, the processing unit 1402 is coupled to a local area network 1422 or a wide area network for data processing and communication with remote devices (e.g., other processing units, the Internet, or remote storage facilities).

[0110] It should be appreciated that one or more steps in the embodiment methods provided herein may be performed by corresponding units or modules. For example, the methods and techniques described in the present disclosure may be performed by an RL agent running on at least one processor of one or more computing systems (e.g., computing system 600), and the RL agent may include a neural network described in the present disclosure. The evaluator may be located outside or inside the RL agent. In addition, any aspects of the different embodiments may be combined. The corresponding units or modules may be hardware, software, or a combination thereof. For example, one or more of the units or modules may be an integrated circuit, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0111] Although described in detail, it should be understood that various changes, substitutions and modifications may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims. In addition, the scope of the present disclosure is not intended to be limited to the specific embodiments described herein, as one of ordinary skill in the art will readily recognize from the present disclosure that processes, machines, manufactures, compositions of matter, devices, methods or steps currently existing or to be developed in the future may perform substantially the same functions or achieve substantially the same results as the corresponding embodiments described herein. Therefore, the appended claims are intended to include within their scope such processes, machines, manufactures, compositions of matter, devices, methods or steps.

Claims

1. A method comprising: Receiving a constant input by a reinforcement learning (RL) agent running on at least one processor, wherein the RL agent comprises a deep neural network (DNN), and wherein the DNN comprises one or more layers of bidirectional gated recurrent units (Bi-GRUs) and one or more self-attention or transformer layers; Outputting, by a final layer of the DNN of the RL agent, N distribution sets, wherein each of the N distribution sets corresponds to a different object among the N interactive objects and includes a first distribution in a first direction and a second distribution in a second direction; generating, by the RL agent, a batch of sample configurations based on the N distribution sets, wherein each sample configuration in the batch of sample configurations includes the identified or selected positions of the N interactive objects; Outputting the batch of sample configurations to an evaluator by the RL agent; and The RL agent updates the parameters of the DNN based on the loss function.

2. The method according to claim 1, wherein: The loss function is calculated based on a reward, and the reward is generated based on a performance indicator output by the evaluator.

3. The method according to claim 1, further comprising: The RL agent repeats the following operations for multiple iterations: receiving a constant input, outputting a set of N distributions, generating a batch of sample configurations, outputting the batch of sample configurations, and updating parameters, wherein the constant input remains the same throughout the multiple iterations.

4. The method according to claim 1, wherein: The positions of the N interactive objects include positions of N macro-cells, or a mixture of positions of M macro-cells and initial centroids of (NM) micro-cell clusters.

5. The method according to claim 1, wherein: The constant input includes a constant value, N different constant one-hot vectors, or N vectors each containing all 1s.

6. The method according to claim 1, wherein: The one or more self-attention or transformer layers receive outputs from the one or more layers of Bi-GRU, and wherein the DNN further comprises an affine layer that receives outputs from the one or more self-attention or transformer layers and outputs the N distribution sets.

7. The method according to claim 1, wherein: The loss function is updated by a value estimation layer of a sequence pattern, and wherein the value estimation layer of the sequence pattern is updated based on the batch of sample configurations.

8. The method according to claim 1, wherein: The loss function is updated by a branching mode value estimation layer, and wherein the branching mode value estimation layer is updated based on the task encoding tensor of the DNN.

9. The method according to claim 1, wherein: The N distribution sets are N categorical distribution sets for discrete 2-D or 3-D placements.

10. The method according to claim 1, wherein: The N distribution sets are used for continuous 2-D or 3-D placement.

11. The method according to any one of claims 1 to 10, wherein: The N distribution sets are N Beta distribution sets.

12. An apparatus comprising: at least one processor; as well as A non-transitory computer-readable storage medium storing a program, the program comprising instructions that, when executed by the at least one processor, cause the apparatus to perform the following operations: receiving a constant input by a reinforcement learning (RL) agent executing on the at least one processor, wherein the RL agent comprises a deep neural network (DNN), and wherein the DNN comprises one or more layers of bidirectional gated recurrent units (Bi-GRUs) and one or more self-attention or transformer layers; Outputting, by a final layer of the DNN of the RL agent, N distribution sets, wherein each of the N distribution sets corresponds to a different object among the N interactive objects and includes a first distribution in a first direction and a second distribution in a second direction; generating, by the RL agent, a batch of sample configurations based on the N distribution sets, wherein each sample configuration in the batch of sample configurations includes the identified or selected positions of the N interactive objects; Outputting the batch of sample configurations to an evaluator by the RL agent; and The RL agent updates the parameters of the DNN based on the loss function.

13. The device according to claim 12, wherein: The loss function is calculated based on a reward, and the reward is generated based on a performance indicator output by the evaluator.

14. The apparatus of claim 12, wherein the program further comprises instructions that, when executed by the at least one processor, cause the apparatus to perform the following operations: The RL agent repeats the following operations for multiple iterations: receiving a constant input, outputting N distribution sets, generating a batch of sample configurations, outputting the batch of sample configurations, and updating parameters, wherein: The constant input remains the same throughout the multiple iterations.

15. The device according to claim 12, wherein: The positions of the N interactive objects include positions of N macro-cells, or a mixture of positions of M macro-cells and initial centroids of (NM) micro-cell clusters.

16. The device according to claim 12, wherein: The constant input includes a constant value, N different constant one-hot vectors, or N vectors each containing all 1s.

17. The device according to claim 12, wherein: The one or more self-attention or transformer layers receive outputs from the one or more layers of Bi-GRU, and wherein the DNN further comprises an affine layer that receives outputs from the one or more self-attention or transformer layers and outputs the N distribution sets.

18. The device according to claim 12, wherein: The loss function is updated by a value estimation layer of a sequence pattern, and wherein the value estimation layer of the sequence pattern is updated based on the batch of sample configurations.

19. The device according to claim 12, wherein: The loss function is updated by a branching mode value estimation layer, and wherein the branching mode value estimation layer is updated based on the task encoding tensor of the DNN.

20. The device according to claim 12, wherein: The N distribution sets are N categorical distribution sets for discrete 2-D or 3-D placements.

21. The device according to claim 12, wherein: The N distribution sets are used for continuous 2-D or 3-D placement.

22. The device according to any one of claims 12 to 21, wherein: The N distribution sets are N Beta distribution sets.

23. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by a device, causing the device to perform operations comprising: Receiving a constant input by a reinforcement learning (RL) agent running on at least one processor, wherein the RL agent comprises a deep neural network (DNN), and wherein the DNN comprises one or more layers of bidirectional gated recurrent units (Bi-GRUs) and one or more self-attention or transformer layers; Outputting, by a final layer of the DNN of the RL agent, N distribution sets, wherein each of the N distribution sets corresponds to a different object among the N interactive objects and includes a first distribution in a first direction and a second distribution in a second direction; generating, by the RL agent, a batch of sample configurations based on the N distribution sets, wherein each sample configuration in the batch of sample configurations includes the identified or selected positions of the N interactive objects; Outputting the batch of sample configurations to an evaluator by the RL agent; and The RL agent updates the parameters of the DNN based on the loss function.

24. The non-transitory computer readable medium of claim 23, wherein: The loss function is calculated based on a reward, and the reward is generated based on a performance indicator output by the evaluator.

25. The non-transitory computer readable medium of claim 23, the operations further comprising: The RL agent repeats the following operations for multiple iterations: receiving a constant input, outputting a set of N distributions, generating a batch of sample configurations, outputting the batch of sample configurations, and updating parameters, wherein the constant input remains the same throughout the multiple iterations.

26. The non-transitory computer readable medium of claim 23, wherein: The positions of the N interactive objects include positions of N macro-cells, or a mixture of positions of M macro-cells and initial centroids of (NM) micro-cell clusters.

27. The non-transitory computer readable medium of claim 23, wherein: The constant input includes a constant value, N different constant one-hot vectors, or N vectors each containing all 1s.

28. The non-transitory computer readable medium of claim 23, wherein: The one or more self-attention or transformer layers receive outputs from the one or more layers of Bi-GRU, and wherein the DNN further comprises an affine layer that receives outputs from the one or more self-attention or transformer layers and outputs the N distribution sets.

29. The non-transitory computer readable medium of claim 23, wherein: The loss function is updated by a value estimation layer of a sequence pattern, and wherein the value estimation layer of the sequence pattern is updated based on the batch of sample configurations.

30. The non-transitory computer readable medium of claim 23, wherein: The loss function is updated by a branching mode value estimation layer, and wherein the branching mode value estimation layer is updated based on the task encoding tensor of the DNN.

31. The non-transitory computer readable medium of claim 23, wherein: The N distribution sets are N categorical distribution sets for discrete 2-D or 3-D placements.

32. The non-transitory computer readable medium of claim 23, wherein: The N distribution sets are used for continuous 2-D or 3-D placement.

33. The non-transitory computer readable medium of any one of claims 23 to 32, wherein: The N distribution sets are N Beta distribution sets.

Citation Information

Patent Citations

  • Systems and Methods for Weighted Quantization

    US20210064634A1

  • Using hierarchical representations for neural network architecture searching

    CN111465946A

  • Information processing method and device, electronic equipment and computer readable storage medium

    CN113761117A

Cited By

  • Down-sampling DPD method and device based on two-stage attention neural network

    CN120075003A

  • Down-sampling dpd method and apparatus based on two-stage attention neural network

    CN120075003B