A macro placement method for high-speed communication chips based on reinforcement learning.

Abstract

This invention provides a macro-placement method for high-speed communication chips based on reinforcement learning, comprising: extracting static and dynamic features from a netlist; fusing the static and dynamic features and inputting them into a shared graph neural network based on a reinforcement learning architecture; the policy network outputting an action probability distribution based on the current environment state and selecting the legal action with the highest probability value to execute; the value network outputting a cumulative value estimate of the current environment state; obtaining an immediate reward and the next state information after executing the action, and updating the policy network and value network based on the obtained information; and constructing a multi-objective optimization function based on macro-placement constraints. This invention improves the automation level and design quality of macro-cell placement through the deep integration of reinforcement learning and constraint modeling, thereby meeting the requirements of high-speed communication chips for low power consumption, high performance, and compact area.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of integrated circuit design automation, in particular to a high-speed communication chip macro placement method based on reinforcement learning, aiming to optimize chip power consumption, performance and area (PPA), suitable for large-scale integrated circuit design scenarios such as high-speed communication and artificial intelligence accelerator. BACKGROUND

[0002] In recent years, with the rapid development of 5G, data centers and artificial intelligence, the demand for high-speed communication chips has shown explosive growth. To meet the high bandwidth demand, such chips increase power consumption to gain performance improvement, resulting in a significant increase in power density. For example, the power of high-speed communication chips under advanced processes (such as 7nm, 5nm) has exceeded 100W, posing a serious challenge to thermal management and power integrity.

[0003] Chip layout is a key link in the integrated circuit design process, directly affecting signal integrity, timing convergence and manufacturing yield. Its optimization goals include: minimizing wire length (HPWL): reducing interconnection delay and power consumption; controlling density and congestion: avoiding local overheating and routing failure; optimizing area utilization: reducing manufacturing cost.

[0004] Macro units (such as memories, IP cores, analog circuits) have a significant impact on global layout quality due to their large area and many pins. With the exponential growth of chip size (such as million-gate designs), the number of macro units can reach hundreds, and the placement constraints become increasingly complex, making it difficult for traditional methods to efficiently handle them.

[0005] Current macro unit placement techniques mainly include the following three categories:

[0006] 1. Iterative optimization based on heuristic algorithms

[0007] Simulated annealing algorithm: searches for global optimal solution through random perturbation and probability acceptance criterion, but has slow convergence speed and weak adaptability to dynamic constraints;

[0008] Genetic algorithm: uses selection, crossover and mutation operations to optimize layout, but has high computational complexity and is difficult to meet the needs of large-scale design;

[0009] Monte Carlo simulation: evaluates layout quality through sampling statistics, but is inefficient and cannot respond to design rule changes in real time.

[0010] Limitations: many iterations, long design cycle, difficult to adapt to fast iteration design process.

[0011] 2. Semi-automated design tools and human intervention

[0012] Commercial EDA tools (e.g., Cadence Innovus, Synopsys ICC): provide macro placement rules and interactive adjustment functions, but require engineers to manually optimize and rely on experience;

[0013] Layout migration techniques: accelerate layout by reusing historical design data, but lack generalization and adaptability to new designs.

[0014] Limitations: high labor cost, design quality affected by engineer level; unable to automatically adapt to dynamic constraints (e.g., real-time density changes), resulting in suboptimal placement results.

[0015] 3. Preliminary exploration based on reinforcement learning

[0016] Recent research attempts to introduce reinforcement learning (RL) into chip layout, but existing techniques have limitations: unable to respond to constraint evaluation, stack length limitations during dynamic placement; and reward function design is single, mostly oriented towards a single goal (e.g., HPWL), making it difficult to effectively handle complex chip design problems. SUMMARY

[0017] To address the limitations of existing techniques, the present application provides a high-speed communication chip macro placement method based on reinforcement learning, which improves the automation level and design quality of macro placement through the integration of reinforcement learning and constraint modeling, meeting the needs of high-speed communication chips for low power consumption, high performance, and compact area, and has important industrial application value.

[0018] The present application achieves the above-mentioned objectives through the following technical solutions:

[0019] A high-speed communication chip macro placement method based on reinforcement learning, comprising:

[0020] Extracting static features and dynamic features in reinforcement learning environment initialization from a netlist describing the logical structure of a circuit;

[0021] After fusing the static features and dynamic features, input them into a shared graph neural network based on a reinforcement learning architecture, wherein the shared graph neural network includes a policy network and a value network that share feature extraction parts;

[0022] The policy network outputs an action probability distribution based on the current environment state and generates an action mask through action legality checking to select the action with the maximum probability value for execution; the value network outputs the cumulative value estimate of the current environment state based on the fused features;

[0023] After executing the action, obtain the immediate reward and the next state information, and update the parameters of the policy network and the value network based on the obtained information;

[0024] A multi-objective optimization function based on macro placement constraints is constructed to minimize the line length, density, congestion and stack length to achieve the optimization of chip power consumption, performance and area.

[0025] According to the macro placement method for high-speed communication chips based on reinforcement learning provided by the application, the shared graph neural network adopts an interactive training mechanism including at least an outer loop and an inner loop, wherein:

[0026] The outer loop is used to traverse each complete environment interaction sequence, and each sequence contains the whole process from the initial state to the termination state of the macro cell placement task.

[0027] The inner loop is used to iterate each step in a single environment interaction sequence, and the following operations are performed in each step:

[0028] The current environment state is input into the shared graph neural network for feature extraction to generate fusion features containing static constraints and dynamic constraints.

[0029] The policy network outputs an action probability distribution based on the fusion features through a policy head, and a legal action mask is applied to screen legal actions, and the legal action with the maximum probability value is selected for execution.

[0030] The value network outputs the cumulative value estimate of the current state based on the fusion features through a value head.

[0031] After the action is executed, the immediate reward and the next state information are obtained, and the state, action, reward and next state of the current step are stored in an experience buffer.

[0032] According to the macro placement method for high-speed communication chips based on reinforcement learning provided by the application, the current environment state is input into the policy network, wherein represents the state observed from the environment at the first t step;

[0033] After the shared graph neural network detects the current environment state , the policy head of the policy network predicts the probability distribution of the placement action in the current state.

[0034] On the original probability distribution output by the policy head, an action mask output after action legality check is applied to obtain a final action probability distribution constrained by action legality; wherein the final action probability distribution constrained by action legality is used as a policy function , and the policy network samples an action in the current state according to a t ;

[0035] After the action is executed, the immediate reward and the next state .

[0036] According to the macro placement method for high-speed communication chips based on reinforcement learning provided by the application, the current environment state is input into a value network, and the value network and a policy network share a same shared graph neural network; after the value network observes the current environment state , a value head of the value network outputs an estimation of the cumulative state value at the current state .

[0037] After one inner loop ends, all information obtained at the t-th step in the current sequence is stored in a buffer, wherein the information at least includes: the observed environment state at the t-th step, the action at the t-th step, the immediate reward returned after the action is performed, the next state, the policy function at the t-th step, the estimation of the cumulative state value at the t-th step, and the estimation of the cumulative state value at the next step. t t t t t .

[0038] According to the macro placement method for high-speed communication chips based on reinforcement learning provided by the application, the training process of the shared graph neural network includes two stages.

[0039] The first stage is a value network preheating stage, only the parameters of the value head in the value network are updated, and the parameters of the policy head remain unchanged;

[0040] The second stage is a policy-value network joint training stage, the parameters of the policy head and the value head are updated at the same time, and the update is based on the interaction data stored in the experience buffer and the reinforcement learning loss function.

[0041] The preset value preheating step threshold is checked after each iteration in the training process; if the current training step is less than the value preheating step threshold, the first stage is entered; otherwise, the second stage is entered; the training step is incremented after each iteration of the inner loop in the first stage is completed; after the training step is updated each time, the discrimination condition is checked again; if the training step does not meet the condition of being less than the value preheating step threshold, the first stage is terminated immediately and the inner loop of the second stage is started.

[0042] ​​​​​​​​​​​​The application provides a high-speed communication chip macro placement method based on reinforcement learning, in a value network preheating stage, experience data is collected from an environment by using the shared graph neural network, a value head is used to estimate a comprehensive loss function , and a policy loss function is set to zero, that is =0;After gradient calculation and gradient clipping, the value gradient is transmitted to the shared graph neural network to update the value head parameters

[0043] In a policy-value network joint training stage, experience data is collected from the environment by using the shared graph neural network, and the parameters after the value network preheating stage are loaded for joint training;The parameters of the policy head and the value head are updated, the policy head outputs an action probability distribution based on shared features, and the value head outputs a state value estimate;A comprehensive loss function related to the policy and the value is calculated ;After gradient calculation and gradient clipping, the policy gradient and the value gradient are transmitted to the shared graph neural network to synchronously update the parameters of the policy head and the value head.

[0044] The application provides a high-speed communication chip macro placement method based on reinforcement learning, and a comprehensive loss function is calculated as follows

[0045]

[0046] The policy loss function is used to adjust the policy parameters to maximize the expected reward, and the probability of selecting the advantageous action is improved by optimizing the policy function , and is represented as

[0047]

[0048] The advantage function is , and the gradient calculation of the policy function is obtained by deriving the policy loss function , as the policy gradient , and the learning rate is represented as The policy function obtains the update direction of the parameters through the gradient, and is represented as

[0049]

[0050] The value loss function is used to learn an accurate estimate of the cumulative state value , and the value parameters are adjusted to minimize the current cumulative estimate ​​The difference between the target value and the value loss function The formula is expressed as:

[0051]

[0052] Where the discount factor is used to adjust the importance of the next step cumulative state value; the gradient calculation of the value function is obtained by derivation of the value loss function , As the value gradient, The learning rate is represented; the value function is obtained by gradient to get the parameters The direction of the update is expressed as:

[0053]

[0054] According to the macro placement method of the high-speed communication chip based on reinforcement learning provided by the application, the training process of the static feature and the dynamic feature constraint comprises:

[0055] Static constraint feature extraction and setting:

[0056] In the environment initialization stage, the static constraint features are parsed from the input initial layout and netlist, and the static constraint features at least include canvas constraints, spacing constraints, grid constraints, fixed macro constraints and I / O port fixed constraints; the parsed static constraint features are set as part of the initial environment state, and remain unchanged in the whole training process;

[0057] Dynamic constraint feature generation:

[0058] In the cyclic training process, the shared graph neural network continuously interacts with the environment to obtain the current state information of the environment; the shared graph neural network generates dynamic constraint features in real time according to the current environment state, and the dynamic constraint features at least include density constraints and maximum stacking length constraints;

[0059] Feature fusion and model training:

[0060] The static constraint features and the dynamic constraint features are fused, and the fused feature representation is input into the shared graph neural network, and the network generates a corresponding action probability distribution according to the current feature representation.

[0061] According to the macro placement method of the high-speed communication chip based on reinforcement learning provided by the application, the multi-objective optimization function strongly related to the macro placement constraint is constructed and expressed as:

[0062]

[0063]

[0064]

[0065]

[0066] wherein, is the total reward of the instant at the step, t is the total reward of the instant at the step, , , and are the line length reward, the congestion reward, the density reward and the stack reward respectively, , , and are the weight coefficients corresponding to the line length reward, the congestion reward, the density reward and the stack reward respectively;

[0067] In the stack reward calculation, and are the horizontal stack reward and the vertical stack reward respectively, is the length of the vertical stack, i is the maximum allowed vertical stack length, is the length of the horizontal stack, is the maximum allowed horizontal stack length; i After each action is executed, the values of each reward item are calculated in real time according to the current macro cell placement state, and are fed back to the shared graph neural network;

[0068] Through the reward function value, the policy network is guided to select an action that satisfies the manufacturing rules and minimizes the line length, the density, the congestion and the stack length, so that the optimization of the chip power consumption, the performance and the area is realized.

[0069] According to the method, for an action that violates the maximum stack length, an additional penalty term is applied penalty and is directly superimposed on the original total reward

[0070] : r

[0071]

[0072] wherein the before the equal sign represents a new total reward, the after the equal sign represents the original total reward without the penalty term, λ is is a penalty coefficient.

[0073] Therefore, compared with the prior art, the method for macro placement of high-speed communication chips based on reinforcement learning has the following beneficial effects: ​​

[0074] 1、The static constraint introduced in the application is linked with the dynamic constraint linkage mechanism, which can monitor the constraint condition in real time during the macro placement process. The static constraint ensures that the macro unit meets the basic manufacturing and design rules in the initial placement stage, and the dynamic constraint dynamically adjusts and optimizes the position of the macro unit according to the real-time changes of the chip layout, preventing rule violations in the subsequent layout process. Through the effective linkage of this double constraint mechanism, the application can maximize the avoidance of macro placement violations of manufacturing or design rules, thereby improving the manufacturing yield of the chip and reducing production costs.

[0075] 2、The application optimizes the power consumption, performance and area of the chip by constructing a reward function strongly related to the macro placement constraint, minimizing the line length, density, congestion and stacking degree as the core target. The application continuously optimizes the placement position of the macro unit through the reinforcement learning algorithm. During the training process, the agent continuously adjusts the placement strategy according to the feedback of the reward function, so that the layout of the macro unit is more reasonable, thereby effectively reducing the line length, balancing the density distribution, reducing congestion and stacking.

[0076] 3、The macro placement method of the application realizes design automation, and the reinforcement learning algorithm can automatically explore and optimize the macro placement strategy, without manual intervention to search and select among a large number of possible layouts. Through continuous learning and iteration, the agent can quickly find the optimal macro placement scheme that meets the constraint condition, greatly improving the design efficiency.

[0077] The application will be further described in detail below in combination with the drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 is a flowchart of an embodiment of a macro placement method for a high-speed communication chip based on reinforcement learning.

[0079] Figure 2 is a system architecture schematic diagram of an embodiment of a macro placement method for a high-speed communication chip based on reinforcement learning.

[0080] Figure 3 is a schematic diagram of the cycle training of the interaction between the policy network, the value network and the environment in an embodiment of a macro placement method for a high-speed communication chip based on reinforcement learning.

[0081] Figure 4 is a schematic diagram of the cycle training of the interaction between the policy network, the value network and the environment in an embodiment of a macro placement method for a high-speed communication chip based on reinforcement learning. Figure 3 is a schematic diagram of the cycle training of the interaction between the policy network, the value network and the environment in an embodiment of a macro placement method for a high-speed communication chip based on reinforcement learning.

[0082] Figure 5is a training process schematic diagram about static features and dynamic features specific constraints in an embodiment of a high-speed communication chip macro placement method based on reinforcement learning. DETAILED DESCRIPTION

[0083] To make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described below in connection with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0084] Reference to“an embodiment” in this document means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor does it necessarily refer to a particular embodiment that is preferred over other embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined with other embodiments.

[0085] Referring to Figures 1 to 5 The embodiment provides a high-speed communication chip macro placement method based on reinforcement learning, which comprises the following steps:

[0086] Step S1, extracting static features and dynamic features in reinforcement learning environment initialization from a netlist describing the logic structure of a circuit;

[0087] Step S2, after fusing the static features and dynamic features, inputting the fused features into a shared graph neural network based on a reinforcement learning architecture, wherein the shared graph neural network comprises a policy network and a value network sharing a feature extraction part;

[0088] Step S3, the policy network outputs an action probability distribution according to the current environment state, generates an action mask through action legality checking, and selects a legal action with the maximum probability value to execute; the value network outputs an accumulated value estimate of the current environment state based on the fused features;

[0089] Step S4, obtaining instant rewards and next state information after executing the action, and updating the parameters of the policy network and the value network according to the obtained information;

[0090] Step S5, a multi-objective optimization function based on macro placement constraints is constructed, so as to minimize the line length, density, congestion and stack length to realize the optimization of chip power consumption, performance and area. The construction of the multi-objective optimization function includes line length reward, congestion reward, density reward and stack reward, and each reward term is weighted and summed through a corresponding weight coefficient; the stack reward includes horizontal stack reward and vertical stack reward, and the vertical stack reward is determined by calculating the difference between the actual vertical stack length and the maximum allowed vertical stack length.

[0091] In the present embodiment, as shown in Figure 2 The netlist is a data file describing the logical structure of the circuit, and the static features and dynamic features are obtained by inputting the netlist into the feature extractor. The static features and dynamic features are fused as the input of the graph neural network based on the reinforcement learning architecture. The graph neural network to be trained includes a policy network and a value network, wherein the policy network and the value network share the same graph neural network for feature extraction. The advantage of sharing the same graph neural network is to reduce the parameter amount of the trained model, and to allow the two networks to learn common feature representations from the same input, thereby improving the learning speed of the model. The graph neural network outputs an action probability distribution, and the action execution with the maximum probability value is selected in the action execution selection. After the action execution, the total loss function value is calculated, and the trainable parameters in the policy network and the value network are updated through the back propagation of the total loss function.

[0092] In the present embodiment, the reinforcement learning architecture is a subfield of artificial intelligence, and its main components include environment, model, state, policy, action, reward and goal.

[0093] The shared graph neural network adopts an interactive training mechanism including at least an outer loop and an inner loop, wherein:

[0094] The outer loop is used to traverse each complete environment interaction sequence, and each sequence contains the whole process from the initial state to the termination state of the macro cell placement task;

[0095] The inner loop is used to iterate each step in a single environment interaction sequence, and each step iteration performs the following operations:

[0096] The current environment state is input into the shared graph neural network for feature extraction to generate fused features containing static constraints and dynamic constraints;

[0097] The policy network outputs an action probability distribution based on the fused features through a policy head, and selects the legal action with the maximum probability value by applying a legal action mask to filter the legal actions;

[0098] The value network outputs the cumulative value estimate of the current state based on the fused features through a value head;

[0099] After performing the action, the immediate reward and the next state information are obtained, and the state, action, reward, and next state of the current step are stored in the experience buffer.

[0100] In the embodiment, the policy network and the value network are both composed of two parts, the first part is the feature extraction of the shared graph neural network, and the second part is the policy head outputting the action probability distribution or the value head outputting the state value.

[0101] Specifically, as shown in FIG. 1, the current environment state s t is input into the policy network, wherein s t represents the state observed from the environment at the t th step. Figure 3 The shared graph neural network detects the current environment state s t, and then the policy head of the policy network predicts the probability distribution of placing an action in the current state. t

[0102] The shared graph neural network detects the current environment state s t, and then the policy head of the policy network predicts the probability distribution of placing an action in the current state.

[0103] On the original probability distribution output by the policy head, the action mask output after the action legality check is applied to obtain the final action probability distribution constrained by the action legality, wherein the final action probability distribution constrained by the action legality is taken as the policy function , and the policy network samples the action a t in the current state according to a t After performing the action, the immediate reward r t+1 and the next state s t+1 are obtained.

[0104] In addition, the current environment state s t is also input into the value network, and the value network shares the same shared graph neural network GCN with the policy network. After the value network observes the current environment state s t, the value head of the value network outputs the estimate of the cumulative state value in the current state s t.

[0105] After the inner loop ends once, all information obtained at the t th step in the current sequence is stored in the buffer, wherein the information at least includes: the observed environment state at the t th step, the action a t at the t th step, the immediate reward r t+1 returned after performing the action a t, the next state s t+1, and the policy function t t t t t ​​​​​​​​​​​​​​​​​​​​Estimation of step-wise cumulative state value and estimation of next step cumulative state value . Wherein, the subscript old in the above represents the parameters before gradient update, and all the information stored is used for subsequent update of trainable parameters of the policy network and the value network.

[0106] In the embodiment, as shown in Figure 4 , the cycle training process of the shared graph neural network includes two stages:

[0107] The first stage is a value network warm-up stage, only the parameters of the value head in the value network are updated, and the policy head parameters remain unchanged.

[0108] The second stage is a policy-value network joint training stage, the parameters of the policy head and the value head are updated at the same time, and the update is based on the interaction data stored in the experience buffer and the reinforcement learning loss function.

[0109] Among them, the distinction between the two stages is determined by the step check: a preset value warm-up step threshold is set, and the step check is performed after each iteration in the training process; if the current training step is less than the value warm-up step threshold, enter the first stage; otherwise, enter the second stage; the internal loop of the first stage is completed once per iteration, and the training step is incremented; after each training step update, the discrimination condition check is performed again, if the training step does not meet the condition of being less than the value warm-up step threshold, the first stage is terminated immediately and the second stage internal loop is started.

[0110] In the value network warm-up stage, during the interaction with the environment, the shared graph neural network is used to collect experience data from the environment, and only the parameters of the value head are updated, and the parameters of the policy head remain unchanged. The value head is used to estimate the comprehensive loss function , and the policy loss function is set to zero, that is =0; after gradient calculation and gradient clipping, the value gradient is passed to the shared graph neural network to update the value head parameters;

[0111] In the policy-value network joint training stage, during the interaction with the environment, the shared graph neural network is used to collect experience data from the environment, and the parameters after the value network warm-up stage are loaded for joint training; at the same time, the parameters of the policy head and the value head are updated, the policy head outputs the action probability distribution based on the shared features, and the value head outputs the state value estimate; the comprehensive loss function related to policy and value is calculated; after gradient calculation and gradient clipping, the policy gradient and the value gradient are passed to the shared graph neural network to update the parameters of the policy head and the value head synchronously.

[0112] In the embodiment, the comprehensive loss function is calculated as follows:

[0113]

[0114] where the policy loss function is to adjust the policy parameters to maximize the expected reward by optimizing the policy function to improve the probability of selecting the advantage action , denoted as

[0115]

[0116] where is the advantage function, and the gradient of the policy function is calculated by taking the derivative of the policy loss function , as the policy gradient, with the learning rate , the policy function gets the direction of updating the parameters , denoted as

[0117]

[0118] where the value loss function is to learn an accurate estimate of the cumulative state value by adjusting the value parameters to minimize the difference between the current cumulative estimate and the target value, and the value loss function is expressed as

[0119]

[0120] where the discount factor is used to adjust the importance of the next step cumulative state value; the gradient of the value function is calculated by taking the derivative of the value loss function , as the value gradient, with the learning rate , the value function gets the direction of updating the parameters , denoted as

[0121]

[0122] After the gradient calculation, the global norm is used for gradient clipping. In the two stages, the graph neural network receives the corresponding gradient after gradient clipping, gets the direction of parameter update and updates the parameters.

[0123] In this embodiment, as shown in Figure 5 , the training process of static feature and dynamic feature constraint includes:

[0124] Static constraint feature extraction and setting:

[0125] In the environment initialization phase, static constraint features are parsed from the input initial layout and netlist, and the static constraint features at least include canvas constraints, spacing constraints, grid constraints, fixed macro constraints, and I / O port fixed constraints; the parsed static constraint features are set as part of the initial environment state, and remain unchanged during the entire training process;

[0126] Dynamic constraint feature generation:

[0127] In the cyclic training process, the shared graph neural network continuously interacts with the environment to obtain the current state information of the environment; the shared graph neural network generates dynamic constraint features in real time according to the current environment state, and the dynamic constraint features at least include density constraints and maximum stack length constraints;

[0128] Feature fusion and model training:

[0129] The static constraint features and the dynamic constraint features are fused, and the fused feature representation is input into the shared graph neural network, and the network generates a corresponding action probability distribution according to the current feature representation.

[0130] The embodiment introduces a static constraint feature and dynamic feature linkage mechanism through feature fusion, and the fused features are input into the shared graph neural network. Feature fusion can be concat to directly connect the two types of features, or add to add element by element, or cross attention mechanism fusion, etc. First, the policy head applies the action mask output by the action legality check to the original output probability distribution to obtain the final probability distribution constrained by the action legality. A legal action is sampled from the final probability distribution according to the current environment state and executed, and the environment state is updated to obtain the next state and the immediate reward of the current state, wherein the immediate reward function of the current state is shown in the above formula. The updated information is written back to the buffer to store the experience, and the advantage is calculated using the updated information. The advantage and the final action probability distribution are combined and taken logarithm as the policy loss function. The gradient of the policy loss function is calculated and used as the direction of policy update.

[0131] In addition, interact with the environment and collect experience data from the environment: the current state and the next state, the value head outputs the cumulative state value estimate of the current state and the next state as the value loss function, calculates the gradient of the value loss function, and uses it as the direction of value update. Finally, the direction of policy update and value update is combined to update the trainable parameters in the graph neural network.

[0132] In practical applications, the embodiment receives a netlist file describing the circuit logic structure, which contains element types, connection relationships, and initial layout position information; the netlist is parsed and standardized to generate a structured element-connection relationship graph;

[0133] Static constraint features are extracted from the preprocessed netlist, including:

[0134] Canvas constraints: physical boundary size and layout area division of the chip design area;

[0135] Spacing constraints: minimum safe distance rules between different element types;

[0136] Grid constraints: granularity and alignment requirements of the layout grid;

[0137] Fixed macro constraints: preset coordinates of core elements that cannot be moved;

[0138] I / O port fixed constraints: binding position and electrical property requirements of input and output ports;

[0139] The above static features are encoded into a multi-dimensional vector as an initial environment state input;

[0140] During the reinforcement learning training process, the following dynamic constraint conditions are monitored in real time:

[0141] Density constraints: element density threshold and congestion warning of the current layout area;

[0142] Maximum stack length constraints: maximum element stack height allowed in vertical / horizontal directions;

[0143] A dynamic feature vector is generated according to the real-time layout state, which at least includes: a density distribution heat map of the current step; and real-time calculation values of the stack length in each direction.

[0144] The embodiment separates the extraction processes of static rules and dynamic states, which meets strict manufacturing constraints and realizes real-time response to layout process changes.

[0145] In the embodiment, a multi-objective optimization function strongly related to macro placement constraints is constructed and represented as:

[0146]

[0147]

[0148]

[0149]

[0150] wherein, is thet The sum of instant rewards during each step. , , and These are line length reward, congestion reward, density reward, and stacking reward. , , and These are the weighting coefficients corresponding to line length reward, congestion reward, density reward, and stacking reward;

[0151] In the calculation of stacked rewards, and These are horizontal stacking rewards and vertical stacking rewards, For the first i The length of the vertical stack, For the maximum allowed vertical stacking length, For the first i The length of the horizontally stacked reward. The maximum allowed horizontal stack length;

[0152] After each action is executed, the value of each reward item is calculated in real time based on the current macrocell placement state and fed back to the shared graph neural network;

[0153] The reward function value guides the policy network to select components that meet manufacturing rules and minimize line length, density, and congestion, thereby optimizing chip power consumption, performance, and area.

[0154] In addition, extra penalties are applied to actions that violate the maximum stack length. r penalty It will be directly added to the original total reward. middle:

[0155]

[0156] The part before the equal sign Represents the new total reward, after the equals sign. This represents the original total reward without penalties. λ is Penalty coefficient.

[0157] In summary, in the traditional macro cell placement process of high-speed communication chips, the lack of an effective constraint linkage mechanism often leads to violations of manufacturing or design rules. These rules cover multiple aspects such as minimum spacing, maximum size, and specific area limitations. Violation of these rules can cause serious problems such as short circuits, open circuits, and signal interference during chip manufacturing, thereby significantly reducing the chip's manufacturing yield.

[0158] The static constraint and dynamic constraint linkage mechanism introduced by the present application can monitor the constraint conditions in the macro placement process in real time. The static constraint ensures that the macro unit meets the basic manufacturing and design rules in the initial placement stage, and the dynamic constraint dynamically adjusts and optimizes the position of the macro unit according to the real-time changes of the chip layout, preventing rule violations in the subsequent layout process. Through the effective linkage of this double constraint mechanism, the occurrence of macro placement violations of manufacturing or design rules can be avoided to the greatest extent, thereby improving the manufacturing yield of the chip and reducing production costs.

[0159] For example, in the actual design of a certain high-speed communication chip, when using the traditional macro placement method, due to the failure to fully consider the dynamic constraint, a certain probability of chip defects occurs in the manufacturing process due to the violation of the minimum spacing rule. However, after applying the macro placement method of the present application, through the synergistic effect of static constraint and dynamic constraint, the rule violation is successfully reduced to a very low probability, improving the manufacturing quality and reliability of the chip.

[0160] The present application optimizes the power consumption, performance and area of the chip by constructing a reward function strongly related to the macro placement constraint, and minimizing the line length, density, congestion and stacking degree as the core target. In chip design, line length, density and congestion are key factors affecting chip power consumption, performance and area. Excessive line length increases signal transmission delay and power consumption, excessive density causes local overheating and signal interference, and congestion limits the chip's wiring resources and signal transmission efficiency. The reward function of the present application takes minimizing line length, density, congestion and stacking length as the core target, and continuously optimizes the placement position of the macro unit through reinforcement learning algorithm. During the training process, the agent continuously adjusts the placement strategy according to the feedback of the reward function, so that the layout of the macro unit is more reasonable, thereby effectively reducing the line length, balancing the density distribution, reducing congestion and stacking.

[0161] Specifically, in terms of power consumption, shortening the line length can reduce energy loss during signal transmission and reduce the overall power consumption of the chip. In terms of performance, reducing congestion and optimizing density distribution can improve signal transmission speed and stability, enhance the computing power and response speed of the chip. In terms of area, reasonable macro placement can make full use of chip space, avoid unnecessary area waste, and realize the miniaturization and integration of the chip.

[0162] Compared with the traditional method, the power consumption, operation performance and chip area of the chip are improved, and the comprehensive quality and market competitiveness of the chip are increased after the macro placement of the high-speed communication processor chip is performed by the method. The traditional macro placement method of the high-speed communication chip often depends on manual experience and repeated tests, and the design process is tedious and inefficient, and a large amount of time and labor cost is consumed. Moreover, due to the limitation of manual design, it is difficult to find the optimal macro placement scheme in a complex design space. The macro placement method based on reinforcement learning of the application realizes high design automation. The reinforcement learning algorithm can automatically explore and optimize the macro placement strategy, and search and select in a large number of possible layouts without manual intervention. Through continuous learning and iteration, the agent can quickly find the optimal macro placement scheme that meets the constraint conditions, and improve the design efficiency.

[0163] In practical applications, the macro placement design period of the high-speed communication chip can be shortened from several weeks or even several months to several days by using the method of the application, which significantly shortens the design period of the entire chip. This enables chip design companies to quickly bring new products to market, seize market opportunities and improve the economic benefits and market competitiveness of enterprises.

[0164] Different types of high-speed communication chips have great differences in function, performance and design requirements, and the traditional macro placement method can only be designed for a specific type of chip, lacking universality and adaptability. The macro placement method of the application has strong universality and adaptability. The static constraint and dynamic constraint linkage mechanism and the design of the reward function have certain flexibility, and can be customized and adjusted according to the design rules and performance requirements of different chips. The application optimizes the macro placement by reasonably setting the constraint conditions and the reward function. For example, when facing a high-speed communication chip with special signal transmission requirements, the designer can modify the static constraint and the dynamic constraint according to the specific design rules and performance indicators, and adjust the weight parameters in the reward function, so that the method can quickly adapt to new design requirements and provide effective macro placement solutions for different types of high-speed communication chips.

[0165] The technical features of the above embodiments can be combined arbitrarily, and to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0166] The above embodiments are only preferred embodiments of the application, and cannot be used to limit the protection scope of the application, and any non-essential changes and replacements made by those skilled in the art on the basis of the application shall fall within the scope of the application.

Claims

1. A high-speed communication chip macro placement method based on reinforcement learning, characterized in that, The method comprises the following steps: extracting static features and dynamic features from a netlist describing the logic structure of a circuit; inputting the static features and the dynamic features into a shared graph neural network based on a reinforcement learning architecture after fusion, wherein the shared graph neural network comprises a policy network and a value network; the policy network outputs an action probability distribution according to a current environment state, and generates an action mask through action legality checking to select an action with the maximum probability value for execution; the value network outputs a cumulative value estimate of the current environment state based on the fused features; after the action is executed, an immediate reward and next state information are obtained, and the parameters of the policy network and the value network are updated according to the obtained information; a multi-objective optimization function based on macro placement constraints is constructed; wherein the construction of the multi-objective optimization function comprises a line length reward, a congestion reward, a density reward and a stacking reward, and each reward term is weighted and summed through a corresponding weight coefficient; wherein the training process of the static feature and dynamic feature constraints comprises: static constraint feature extraction and setting: in the environment initialization stage, static constraint features are parsed from the input initial layout and netlist, and the static constraint features at least comprise canvas constraints, spacing constraints, grid constraints, fixed macro constraints and I / O port fixed constraints; the parsed static constraint features are set as part of the initial environment state and remain unchanged throughout the training process; dynamic constraint feature generation: in the cyclic training process, the shared graph neural network continuously interacts with the environment to obtain current state information of the environment; the shared graph neural network generates dynamic constraint features in real time according to the current environment state, and the dynamic constraint features at least comprise density constraints and maximum stacking length constraints; feature fusion and model training: the static constraint features and the dynamic constraint features are fused, and the fused feature representation is input into the shared graph neural network, which generates a corresponding action probability distribution according to the current feature representation.

2. The method of claim 1, wherein, The shared graph neural network adopts an interactive training mechanism comprising at least an outer loop and an inner loop, wherein: the outer loop is used to traverse each complete environment interaction sequence, and each sequence contains the whole process from the initial state to the terminal state of the macro unit placement task; the inner loop is used to iterate each step in a single environment interaction sequence, and each step iteration performs the following operations: the policy network outputs an action probability distribution based on the fused features through a policy head, and filters legal actions by applying an action legality mask to select an action with the maximum probability value for execution; the value network outputs a cumulative value estimate of the current state based on the fused features through a value head; after the action is executed, an immediate reward and next state information are obtained, and the state, action, reward and next state of the current step are stored in an experience buffer.

3. The method of claim 2, wherein: the current environment state into the policy network, where represent the state t observed from the environment at the step The shared graph neural network predicts a probability distribution of placing an action in the current state After that, a policy head of the policy network predicts a probability distribution of placing an action in the current state; an action mask output by the action legality checking is applied to the original probability distribution output by the policy head to obtain a final action probability distribution constrained by the action legality. Among them, the final action probability distribution constrained by action legality is used as the policy function. Policy network according to Sample the action in the current state a t ; Upon performing the action, an immediate reward is obtained and the next state .

4. The method of claim 2, wherein: the current environment state into a value network that shares the same shared graph neural network as the policy network, where the value network observes the current environment state after which the value head of the value network outputs an estimate of the cumulative state value at the current state ;​ After one inner loop completes, the current sequence will be... t All information obtained at each step is stored in a buffer, and all information includes at least the observed first step. t Step environmental status, first t walking movements Execution of actions Instant rewards upon return Next state and the t Strategy function during step , No. t Estimation of cumulative state value over time And the estimation of the cumulative state value in the next step .

5. The method of claim 4, wherein, the training process of the shared graph neural network comprises two stages: The first stage is a value network preheating stage, only the parameters of the value head in the value network are updated, and the parameters of the policy head remain unchanged; The second stage is a joint training stage of policy-value network, the parameters of the policy head and the value head are updated simultaneously, and the update is based on the interaction data stored in the experience buffer and the reinforcement learning loss function; Wherein, a preset value preheating step threshold is set, and a step check is performed after each iteration in the training process; if the current training step is less than the value preheating step threshold, the first stage is entered; Otherwise, the second stage is entered; the training step is incremented every time the internal loop of the first stage completes an iteration; after each training step update, the discrimination condition check is performed again, and if the training step does not satisfy the condition of being less than the value preheating step threshold, the first stage is terminated immediately and the internal loop of the second stage is started.

6. The method of claim 5, wherein: In the value network preheating phase, experience data is collected from the environment using the shared graph neural network, and a comprehensive loss function is estimated using a value head , while the policy loss function is set to zero, i.e. = 0; after gradient calculation and gradient clipping, the value gradient is passed to the shared graph neural network to update the value head parameters; In the strategy-value network joint training phase, experience data is collected from the environment by using the shared graph neural network, and the parameters after the value network warm-up phase are loaded for joint training; the parameters of the strategy head and the value head are updated, the strategy head outputs an action probability distribution based on the shared features, and the value head outputs a state value estimate; a comprehensive loss function related to the strategy and the value is calculated ​ After gradient calculation and gradient clipping, the policy gradient and the value gradient are passed to the shared graph neural network to update the parameters of the policy head and the value head simultaneously.

7. The method of claim 6, wherein: Comprehensive loss function The calculation formula is: where the policy loss function The goal is to adjust the policy parameters to maximize the expected reward by optimizing the policy function to increase the probability of selecting the optimal action is represented as: wherein, is the advantage function, ; the gradient computation of the policy function is obtained by derivation of the policy loss function , as the policy gradient, represents the learning rate; the policy function obtains the parameter update direction through the gradient , which is expressed as: where the value loss function The goal is to learn an accurate estimate of the cumulative state value by adjusting the value parameters to minimize the difference between the current cumulative estimate and the target value, the value loss function is formulated as: where the discount factor , is used to adjust the importance of the next step cumulative state value; the gradient of the value function is computed by taking the derivative of the value loss function , as the value gradient, represents the learning rate; the value function gets updated in the direction of the gradient, denoted as: ​ 。 8. The method according to any one of claims 1 to 7, characterized in that, The multi-objective optimization function based on macro placement constraints is constructed and represented as: wherein, is the first t is the sum of the instant rewards at the step, , , and are the line length reward, the congestion reward, the density reward and the stacking reward, respectively, , , and are the weight coefficients corresponding to the line length reward, the congestion reward, the density reward and the stacking reward, respectively. In the stacked reward calculation, and are the horizontal and vertical stacked rewards, respectively, is the length of the i th horizontal stack, is the maximum allowed horizontal stack length, is the length of the j th vertical stack reward, is the maximum allowed vertical stack length; After each action execution, the values of each reward term are calculated in real time according to the current macro cell placement state and fed back to the shared graph neural network; Through the reward function value, the policy network is guided to select an action that satisfies the manufacturing rules and minimizes the wire length, density, congestion, and stack length, so as to optimize the chip power consumption, performance, and area.

9. The method of claim 8, wherein: Applying an additional penalty term for actions violating the maximum stack length r penalty directly to the original total reward : The part before the equal sign Represents the new total reward, after the equals sign. This represents the original total reward without penalties. λ is Penalty coefficient.

Images