Multi-agent reinforcement learning optimization method of analog circuit and related device

Abstract

The invention provides a multi-agent reinforcement learning optimization method of an analog circuit and a related device. The method comprises the following steps: setting an ideal performance parameter group of the analog circuit; clustering the components according to the transient current curves of the components to obtain a plurality of component groups; configuring an intelligent agent for each component group, and enabling the intelligent agent to output a circuit parameter value according to the ideal performance parameter group; applying the circuit parameter value to a simulation circuit for simulation to obtain a simulation circuit performance value; updating the intelligent agent according to the simulation circuit performance value and a preset target performance value to obtain a target intelligent agent; the elements are clustered by analyzing the actual current characteristics of the circuit under transient response, so that the elements with similar functions are divided to the same agent, thereby breaking the limitation of a physical module, realizing parameter decoupling and collaborative optimization, and improving the convergence efficiency.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the field of circuit, in particular to a multi-agent reinforcement learning optimization method of analog circuit and related device. BACKGROUND

[0002] The traditional analog circuit design method relies on the experience and professional knowledge of designers to make parameter adjustment and performance trade-off, which not only leads to a long design cycle, but also makes it difficult to pass on and scale high-quality design capabilities. The design method through machine learning can shorten the design cycle, but there are still the following problems: the grouping method of components still relies on prior knowledge and subjective judgment of specific circuit topology; manual grouping based on physical modules limits the multi-agent collaboration efficiency and convergence speed. SUMMARY

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The purpose of the present application is to solve one of the technical problems in the related art to at least some extent, and the embodiment of the present application provides a multi-agent reinforcement learning optimization method of analog circuit and related device.

[0005] Embodiments of the first aspect of the present application, a multi-agent reinforcement learning optimization method of analog circuit, comprising:

[0006] setting an ideal performance parameter group of the analog circuit;

[0007] clustering components according to their transient current curves to obtain a plurality of component groups;

[0008] configuring an agent for each of the component groups, so that the agent outputs a circuit parameter value according to the ideal performance parameter group;

[0009] applying the circuit parameter value to a simulation circuit for simulation to obtain a simulation circuit performance value;

[0010] updating the agent according to the simulation circuit performance value and a preset target performance value to obtain a target agent.

[0011] According to certain embodiments of the first aspect of the present application, the setting of the ideal performance parameter group of the analog circuit comprises:

[0012] parameterizing the components in the circuit to set the parameter range of the components;

[0013] setting the range of performance indicators according to the preset circuit design requirements;

[0014] converting the design intention of the analog circuit into a quantitative weight for guiding the evaluation of the overall performance of the circuit;

[0015] The ideal performance parameter group of the analog circuit is composed of the parameter range of the components, the range of the performance indicators and the quantization weight.

[0016] According to certain embodiments of the first aspect of the present application, the design intention of the analog circuit is converted into a quantization weight for guiding the evaluation of the overall performance of the circuit, including:

[0017] According to the design intention, the importance comparison matrix between different performance indicators is obtained by comparing different performance indicators;

[0018] The importance weight value of each performance indicator is extracted from the importance comparison matrix;

[0019] The importance weight value of the performance indicator is subjected to consistency test, and the quantization weight is calculated after the test is passed.

[0020] According to certain embodiments of the first aspect of the present application, the components are clustered according to the transient current curve of the components, and a plurality of component groups are obtained, including:

[0021] The dynamic time warping distance between different components is calculated according to the transient current curve of the components;

[0022] The initial clustering clusters with similar distances are clustered and merged according to the dynamic time warping distance, and the clustering clusters are obtained;

[0023] The silhouette coefficient of the target component is determined according to the average dynamic time warping distance between the target component and other components in the same cluster and the average dynamic time warping distance between the target component and the nearest neighbor cluster, and the optimal clustering number is determined according to the silhouette coefficient, and the silhouette system is used to evaluate the clustering quality under different clustering numbers;

[0024] The components are clustered according to the optimal clustering number, and the candidate component groups are generated;

[0025] According to the similarity between the candidate component groups, the component groups are selected from the candidate component groups.

[0026] According to certain embodiments of the first aspect of the present application, the agent outputs the circuit parameter value according to the ideal performance parameter group, including:

[0027] The target ideal performance parameter group is selected from a plurality of ideal performance parameter groups;

[0028] The policy network in the agent outputs the circuit parameter value according to the target ideal performance parameter group.

[0029] According to certain embodiments of the first aspect of the application, the applying the circuit parameter value to the simulation circuit to obtain a simulation circuit performance value comprises:

[0030] updating a circuit netlist according to the circuit parameter value to obtain an instance of the simulation circuit;

[0031] performing simulation and circuit performance analysis on the simulation circuit by a simulator to obtain the simulation circuit performance value.

[0032] According to certain embodiments of the first aspect of the application, the updating the agent according to the simulation circuit performance value and a preset target performance value to obtain a target agent comprises:

[0033] obtaining a reward value according to a difference between the simulation circuit performance value and the preset target performance value;

[0034] constructing observation information according to the reward value and state information of the simulation circuit;

[0035] updating the agent according to the observation information to obtain the target agent.

[0036] According to certain embodiments of the first aspect of the application, the updating the agent according to the observation information to obtain the target agent comprises:

[0037] standardizing reward values of samples in the same batch to obtain advantage values;

[0038] determining a target function according to the advantage values;

[0039] updating the agent according to the target function to obtain the target agent.

[0040] Embodiments of the second aspect of the application, an electronic device, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-agent reinforcement learning optimization method for analog circuit as described in the embodiments of the first aspect of the application.

[0041] Embodiments of the third aspect of the application, a computer storage medium, storing computer executable instructions, wherein the computer executable instructions are used to execute the multi-agent reinforcement learning optimization method for analog circuit as described in the embodiments of the first aspect of the application.

[0042] The above scheme has at least the following beneficial effects: clustering elements by analyzing the actual current characteristics of the elements under transient response allows functionally similar elements to be divided into the same agent, thereby breaking the physical module limit, more scientifically realizing parameter decoupling and collaborative optimization, and significantly improving convergence efficiency. Quantifying the weight of multiple performance indicators in the reward function ensures that the automated optimization process is highly consistent with the design intent. Meanwhile, to address the weak correlation between adjacent states caused by the direct mapping of the entire parameter space to the action space in the reinforcement learning environment, a single-step trajectory configuration and policy optimization algorithm without an evaluation network are used, avoiding the problems of inaccurate advantage estimation and slow training caused by traditional long trajectories and evaluation networks in a sparse reward environment, thereby further improving the stability and optimization performance of the training. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the technical solutions of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical solutions of the present application, and do not constitute a limitation on the technical solutions of the present application.

[0044] Figure 1 is a step diagram of the multi-agent reinforcement learning optimization method of the analog circuit;

[0045] Figure 2 is a sub-step diagram of step S100;

[0046] Figure 3 is a sub-step diagram of step S130;

[0047] Figure 4 is a sub-step diagram of step S200;

[0048] Figure 5 is a sub-step diagram of step S300;

[0049] Figure 6 is a sub-step diagram of step S400;

[0050] Figure 7 is a sub-step diagram of step S500;

[0051] Figure 8 is a sub-step diagram of step S530. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0053] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the sequence in the flowchart. The terms "first", "second", etc. in the specification, claims or above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0054] Embodiments of the present application provide a multi-agent reinforcement learning optimization method for analog circuits.

[0055] Referring to Figure 1 , the multi-agent reinforcement learning optimization method for analog circuits comprises the following steps:

[0056] Step S100, setting a group of ideal performance parameters of the analog circuit;

[0057] Step S200, clustering the components according to the transient current curve of the components, to obtain a plurality of component groups;

[0058] Step S300, configuring an agent for each component group, so that the agent outputs circuit parameter values according to the group of ideal performance parameters;

[0059] Step S400, applying the circuit parameter values to the simulation circuit for simulation to obtain a simulation circuit performance value;

[0060] Step S500, updating the agent according to the simulation circuit performance value and the preset target performance value to obtain a target agent.

[0061] It can realize automatic optimization of parameters of complex analog circuits such as Low Dropout Regulator (LDO). The overall technical architecture adopts a hierarchical progressive design idea, from the bottom layer of circuit netlist parameterization processing to the upper layer of intelligent optimization decision, forming a complete end-to-end automated design process.

[0062] Referring to Figure 2 , for step S100, setting a group of ideal performance parameters of the analog circuit, comprising the following steps:

[0063] Step S110, parameterizing the components in the circuit, and setting the parameter range of the components;

[0064] Step S120, setting the range of performance indicators according to the preset circuit design requirements;

[0065] Step S130, converting the design intention of the analog circuit into a quantitative weight for guiding the evaluation of the overall performance of the circuit;

[0066] Step S140, the ideal performance parameter group of the analog circuit is composed of the parameter range of the component, the range of the performance index and the quantization weight.

[0067] For step S110, the original circuit SPICE netlist to be optimized is parameterized. This process defines the physical parameters of key components in the circuit (such as transistors, resistors, capacitors, etc.) as variable design variables, such as the width, length, and number of fingers of the transistor. At the same time, reasonable value ranges are set for these parameters to ensure that the generated circuit parameter combinations are physically realizable.

[0068] For step 120, a series of target performance index ranges are defined according to the specific circuit design requirements. These indexes cover various performance dimensions of the circuit, such as quiescent current, phase margin, gain-bandwidth product, load regulation, etc. The system randomly generates ideal performance samples within these ranges to provide a rich, actual demand-compliant target distribution for subsequent reinforcement learning training.

[0069] For step S130, analog circuit design usually involves more than ten or even dozens of performance indexes, and there are often mutual constraints between these indexes, and designers have different emphasis on different indexes. In analog circuit design, the optimization process often needs to balance multiple, even conflicting performance indexes. Different design projects have different priority requirements for each index, and traditional methods are difficult to quantitatively express the designer's subjective design preferences and intentions. This step converts the designer's qualitative judgment into precise quantitative weights, thereby guiding the subsequent automated optimization process.

[0070] Reference Figure 3 The design intention of the analog circuit is converted into a quantization weight for guiding the evaluation of the overall performance of the circuit, including the following steps:

[0071] Step S131, according to the design intention, compare different performance indexes to obtain an importance comparison matrix between different performance indexes;

[0072] Step S132, extract the importance weight value of each performance index from the importance comparison matrix;

[0073] Step S133, perform consistency test on the importance weight value of the performance index, and calculate the quantization weight after the test is passed.

[0074] For step S131, the purpose is to transform the designer's qualitative understanding of the importance of performance metrics into a structured mathematical form. The designer makes pairwise comparisons between all the performance metrics that need to be optimized. For example, between the two metrics "static current" and "phase margin", the designer needs to judge which one is more important and give a quantitative relative importance score according to a pre-defined scale (e.g. from equally important to extremely important), following a 1 to 10 scale.

[0075] The comparison process goes through all possible pairs of metrics. The results of the comparisons are systematically organized into a consistent importance comparison matrix. In this matrix, the element c ij represents the importance of metric i with respect to metric j, and the matrix satisfies the reciprocal property, i.e. c ji = 1 / c ij , and the diagonal elements c ii = 1.

[0076] For step S132, after the importance comparison matrix is constructed, the objective weights of each performance metric are extracted from it through mathematical operations.

[0077] First, the importance aggregation value SL i of each metric i is calculated. This value is obtained by summing the elements in the i-th row of the importance comparison matrix C, and represents the overall relative importance of metric i with respect to all other metrics.

[0078] The formula for calculating the importance aggregation value is as follows:

[0079] Next, in order to obtain the final weights of each metric, the aggregation values are normalized. The final importance weight I i of each metric is obtained by dividing its own aggregation value SL i by the sum of the aggregation values of all metrics. This weight I i precisely represents the quantitative importance of metric i in the overall optimization goal.

[0080] The formula for calculating the importance weight is as follows:

[0081] The weights I i of all metrics together form the final weight vector. Each element in this vector precisely corresponds to a performance metric, and its numerical value quantitatively represents the importance of that metric in this design task.

[0082] The consistency check step ensures that there are no logical contradictions in the pairwise comparison process (e.g. considering A more important than B, B more important than C, but C more important than A).

[0083] A consistency ratio (CR) is automatically calculated based on the mathematical properties of the comparison matrix (e.g., by calculating its largest eigenvalue λmax). The consistency ratio is automatically calculated based on the mathematical properties of the comparison matrix. The value is calculated based on the largest eigenvalue of the comparison matrix and the dimension of the matrix (i.e., the number of performance indicators) N.

[0084] The formula for calculating the consistency ratio is as follows:

[0085] The CR value is used to measure the degree of consistency of the comparison judgment. If the CR value is less than a predetermined threshold (e.g., 0.1), it indicates that the judgment in the pairwise comparison process has good consistency, and the calculated weight vector is reliable and can be accepted. If the CR value exceeds the threshold, it indicates that there may be logical contradictions in the judgment in the pairwise comparison process, and the pairwise comparison process needs to be reviewed and adjusted.

[0086] The abstract design intent is systematically and reliably converted into a set of precise, quantified weights that can be used to guide the optimization of the machine through step S130. This set of verified weights will be directly applied to the reward function design in the subsequent training phase, ensuring that the optimization direction of multi-agent reinforcement learning is highly aligned with the designer's desired goals, thereby efficiently solving complex multi-objective optimization problems.

[0087] Randomly combine the parameter ranges of the components, the ranges of the performance indicators, and the quantified weights to form multiple sets of ideal performance parameter groups for the simulated circuits.

[0088] For step S200, this step realizes an automated data-driven circuit component grouping method. This method replaces the manual grouping method that relies on the designer's prior knowledge and experience, and achieves more reasonable and efficient parameter grouping by analyzing the intrinsic functional similarity of components in the circuit, thereby significantly improving the training speed and optimization effect of multi-agent reinforcement learning. The goal of this stage is to determine the optimal parameter grouping strategy for subsequent multi-agent training. Through rigorous mathematical methods, the objectivity and robustness of the grouping are ensured.

[0089] Referring to Figure 4 , the components are clustered according to their transient current curves to obtain multiple component groups, including the following steps:

[0090] Step S210, calculate the dynamic time warping distance between different components according to their transient current curves;

[0091] Step S220, cluster and merge the initial clustering clusters with similar distances according to the dynamic time warping distance to obtain clustering clusters;

[0092] Step S230, determining the silhouette coefficient of the target component according to the average dynamic time warping distance between the target component and other components in the same cluster and the average dynamic time warping distance between the target component and the nearest neighbor component in the cluster, and determining the optimal cluster number according to the silhouette coefficient; the silhouette system is used to evaluate the clustering quality under different cluster numbers;

[0093] Step S240, clustering the components according to the optimal cluster number to generate a candidate component group;

[0094] Step S250, selecting a component group from the candidate component groups according to the similarity between the candidate component groups.

[0095] In analog circuits, functionally similar components will exhibit similar current change patterns when responding to external excitation (such as load transient changes). This characteristic is used for automatic clustering of components.

[0096] To accurately measure the similarity between different component current transient curves (i.e. time series data), dynamic time warping (DTW) distance is used. DTW distance can effectively handle the time axis shift problem caused by RC delay and other factors in the circuit, and focuses on capturing the intrinsic shape similarity of waveforms.

[0097] The calculation formula of DTW distance is as follows:

[0098] Where π represents the optimal warping path between two time series, and d is the Euclidean distance between the corresponding path points.

[0099] After calculating the DTW distance between all pairs of components, the present application uses a hierarchical clustering algorithm to group them. This algorithm uses the average linkage criterion and iteratively merges the closest clusters until all components are grouped into one cluster.

[0100] The distance between two clusters C p and C q is defined as follows:

[0101] Where |C p | and |C q | represent the number of components contained in the two clusters.

[0102] To obtain an optimal grouping that is robust under different operating conditions, the following screening process is used.

[0103] For each circuit simulation sample, the system needs to determine an optimal cluster number k. To this end, the silhouette coefficient is used to evaluate the clustering quality under different k values. For each component i in the cluster, its silhouette coefficient si The calculation requires determining two values: a i (the average DTW distance of the component to all other components in the same cluster) and b i (the average DTW distance of the component to all components in the nearest neighbor cluster).

[0104] The formula for calculating the silhouette coefficient is as follows:

[0105] wherein,

[0106] The value of k that maximizes the average silhouette coefficient of the entire sample is selected as the optimal number of clusters under the sample.

[0107] To ensure the generalization ability of the grouping strategy, the system applies all candidate optimal cluster numbers k identified in the previous step to all circuit simulation samples. This process generates a comprehensive candidate solution set containing multiple clustering results, which contains multiple candidate solutions, i.e., candidate component groups.

[0108] From the numerous candidate solutions, a unique optimal grouping solution is selected. The Adjusted Rand Index (ARI) is used to evaluate the consistency or similarity between different clustering solutions.

[0109] As the basis of ARI, the Rand Index (RI) measures the similarity by calculating the proportion of component pairs that are consistently handled (i.e., belong to the same cluster or belong to different clusters) in two clustering solutions.

[0110] The definition formula of the Rand Index is as follows:

[0111] wherein, a is the number of component pairs that belong to the same cluster in both solutions C1 and C2, d is the number of component pairs that belong to different clusters in both solutions, and is the total number of component pairs.

[0112] The Adjusted Rand Index is a modification of the Rand Index, which eliminates the consistency that may be accidentally achieved by random grouping, and thus more accurately measures the true similarity between clustering solutions.

[0113] The calculation formula of the Adjusted Rand Index is as follows:

[0114] wherein C1 and C2 are two candidate component groups to be compared, E[RI] is the expected Rand Index under random clustering. The ARI scores between any two candidate component groups in the candidate scheme set are calculated to construct a similarity matrix. Finally, the candidate component group with the highest average similarity score compared with all other candidate component groups will be selected as the final optimal component grouping strategy. This consensus-based selection method ensures that the final grouping scheme has the highest consistency and stability under diverse circuit operating conditions.

[0115] For step S300, the prepared grouping scheme and design preference weight are used to optimize the multi-agent (Multi-Agent) strategy through repeated interaction with the circuit simulation environment. A multi-agent proximal policy optimization algorithm specially designed for grouping optimization is adopted to efficiently and stably learn complex circuit parameter design strategies.

[0116] Configure an agent for each component group.

[0117] Referring to Figure 5 , the agent outputs circuit parameter values according to the ideal performance parameter group, including the following steps:

[0118] Step S310, selecting a target ideal performance parameter group from a plurality of ideal performance parameter groups;

[0119] Step S320, the strategy network in the agent outputs circuit parameter values according to the target ideal performance parameter group.

[0120] The training process consists of a series of interactions, each step representing a decision by the agent and a feedback from the environment.

[0121] At the beginning of each training episode, a specific design target is first randomly sampled from the plurality of ideal performance parameter groups generated in the preprocessing stage as the target ideal performance parameter group, which is input into the agent. Then, according to the current state, each agent (corresponding to a component grouping) will output a set of actions according to its internal strategy network, determining specific values for the parameters it is responsible for, i.e. circuit parameter values.

[0122] Referring to Figure 6 For step S400, the circuit parameter values are applied to the simulation circuit for simulation to obtain the simulation circuit performance values, including the following steps:

[0123] Step S410, updating the circuit netlist according to the circuit parameter values to generate an instance of the simulation circuit;

[0124] Step S420, simulating and analyzing the circuit performance of the simulation circuit through the simulator to obtain the simulation circuit performance values.

[0125] All the parameter values outputted by the agents are integrated and updated into the parameterized netlist, thus generating a simulation circuit instance to be evaluated.

[0126] The configured netlist is submitted to an industry-standard SPICE (Simulation Program with Integrated Circuit Emphasis) simulator (e.g. Spectre) for comprehensive performance evaluation. The simulator performs a series of predefined circuit analyses, such as DC Analysis, AC Analysis, and Transient Analysis, to accurately calculate the performance indicators of the circuit instance under the current parameters. This step ensures that the feedback of the training process is based on physically accurate circuit behavior, thus guaranteeing the effectiveness of the learned strategy in practical applications.

[0127] After the simulation is completed, the system enters the feedback and learning update link. It is beneficial to improve the decision-making ability of the agent.

[0128] Reference Figure 7 For step S500, the agent is updated according to the simulation circuit performance value and the preset target performance value to obtain a target agent, including the following steps:

[0129] Step S510, obtaining a reward value according to the difference between the simulation circuit performance value and the preset target performance value;

[0130] Step S520, constructing observation information according to the reward value and the state information of the simulation circuit;

[0131] Step S530, updating the agent according to the observation information to obtain the target agent.

[0132] From the SPICE simulation results, various performance indicators are extracted and compared with the current design targets. These performance indicators and other related circuit state information (such as the working zone of components) together constitute the "state observation" of the agent at the current time step. At the same time, the system calculates a comprehensive reward value (Reward) according to the difference between the simulation performance and the target, combined with the weights determined by the AHP (Analytic Hierarchy Process) in the preprocessing stage. This reward value quantitatively evaluates the goodness of the agent's previous decision, and is the core signal driving its strategy optimization.

[0133] To accelerate exploration, the action space of the agent is directly mapped to the entire parameter space. This setup results in very weak correlations between adjacent states during training. For traditional, critic network-dependent algorithms (e.g., standard PPO), this weak correlation makes it difficult for the critic network to learn an accurate state value function (V(s)), resulting in unstable value estimates and high-variance advantage function (Advantage Function) calculations, which ultimately lead to incorrect gradient signals, severely impacting training efficiency and convergence speed. To solve the above problems, the multi-agent grouping relative policy optimization algorithm does not estimate the advantage indirectly through the "state value", but directly calculates the advantage of each action through relative comparison in the collected batch of sample data. Specifically, the algorithm standardizes the rewards obtained by all samples in a batch (for example, subtract the mean and divide by the standard deviation), and directly uses this standardized reward as the advantage estimate of the action.

[0134] Reference Figure 8 , specifically, updating the agent according to the observation information to obtain a target agent, including the following steps:

[0135] Step S531, standardizing the reward values of the same batch of samples to obtain advantage values;

[0136] Step S532, determining a target function according to the advantage values;

[0137] Step S533, updating the agent according to the target function to obtain a target agent.

[0138] The reward values of the same batch of samples are standardized to obtain the advantage values. The calculation formula of the advantage value is:

[0139] Where A i is the advantage estimate of the action taken by the agent i, is the reward obtained by the action, and and are the mean and standard deviation of all rewards in the current batch, respectively.

[0140] The advantage estimate A i obtained in this way is more direct and accurate, as it directly reflects the relative good or bad of the current action in a group of contemporaneous experiments, without introducing errors caused by unstable value prediction. This reliable advantage estimate is then substituted into the clipped surrogate target function of the proximal policy optimization to update the policy network.

[0141] The calculation formula of the target function is:

[0142] where r t (θ) is the ratio of the old and new policy output action probabilities, which is the advantage estimation obtained by the relative comparison above, indicating the goodness or badness of the current action relative to the average level. ε is a hyperparameter that defines the clipping range, which limits the magnitude of policy update within a reliable interval, avoiding training collapse caused by excessively large single-step updates.

[0143] The above algorithm has two significant advantages: first, it provides a correct advantage estimation. In the special setting where the action space corresponds to the entire parameter space, this method avoids the evaluation network's estimation error and can provide stable and reliable gradient signals, significantly improving training efficiency and convergence speed. Second, it reduces computational overhead. Since the evaluation network and its related training and inference processes are completely removed, the computational complexity and resource consumption of the entire algorithm are reduced, further improving the overall method's efficiency.

[0144] The above single-step interaction, simulation evaluation, reward calculation, and policy update process will continue for multiple times. Through this continuous trial and error and iterative optimization, all agent policy networks are gradually improved, and eventually learn a set of general design strategies that can generate high-performance circuit parameter combinations for diverse design objectives. The trained target agent is obtained.

[0145] After training, the system obtains a set of optimized agent policy networks that can achieve high-performance circuit design. These policy networks contain general circuit design knowledge and can be directly reused in new design tasks. For example, when a designer needs to optimize for new performance targets (such as lower power consumption) or different manufacturing processes, these trained strategies can be directly loaded and fine-tuned without starting from scratch. This strategy reuse capability significantly reduces the computational overhead and time cost of subsequent design, greatly improving design efficiency.

[0146] By analyzing the actual current characteristics of the elements under the transient response of the analysis circuit, the elements are clustered, so that functionally similar elements are divided into the same agent, thereby breaking the limitation of the physical module, more scientifically realizing parameter decoupling and collaborative optimization, and significantly improving the convergence efficiency. In addition, improvements have been made in terms of optimization target alignment and learning efficiency. When dealing with up to dozens of performance indicators, the traditional method is difficult to quantify the designer's preference for different indicators systematically, resulting in the optimization direction sometimes deviating from the original design intention. By using the analytic hierarchy process to quantify the weight of multiple performance indicators in the reward function, the automated optimization process is ensured to be highly consistent with the designer's intention. At the same time, in order to solve the problem of weak correlation between adjacent states caused by the direct mapping of the entire parameter space to the action space in the reinforcement learning environment, a single-step trajectory configuration and policy optimization algorithm without evaluation network are used. This strategy avoids the problems of inaccurate advantage estimation and slow training caused by traditional long trajectories and evaluation networks in a sparse reward environment, thereby further improving the stability and optimization performance of the training.

[0147] Embodiments of the present application provide an electronic device. The electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-agent reinforcement learning optimization method of the analog circuit as described above when executing the computer program.

[0148] The electronic device can be any intelligent terminal including a computer.

[0149] In general, for the hardware structure of the electronic device, the processor can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0150] The memory can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the relevant program codes are saved in the memory and executed by the processor to implement the method of the embodiments of the present application.

[0151] The input / output interface is used to realize information input and output.

[0152] The communication interface is used to realize the communication interaction between the device and other devices, and the communication can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0153] The bus transmits information between various components (such as the processor, the memory, the input / output interface, and the communication interface) of the device. The processor, the memory, the input / output interface, and the communication interface realize the communication connection between each other inside the device through the bus.

[0154] Embodiments of the present application provide a computer storage medium. The computer storage medium stores computer executable instructions for executing the multi-agent reinforcement learning optimization method of the analog circuit as described above.

[0155] Those of ordinary skill in the art can understand that all or some steps in the above disclosed method and system can be implemented as software, firmware, hardware and appropriate combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, as known to those of ordinary skill in the art, communication media generally includes computer readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and can include any information delivery medium. In the above description of the specification, the description of the terms "one embodiment", "another embodiment" or "some embodiments" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples.

[0156] Those of ordinary skill in the art understand that all or some of the steps in the method disclosed above and the functional modules / units in the system and device can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0157] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0158] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0159] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0160] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the apparatus embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices, or units, and can be in electrical, mechanical, or other forms.

[0161] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit and principles of the present application. These equivalent modifications or replacements are all included in the scope of the claims of the present application and their equivalents.

Claims

1. A method for multi-agent reinforcement learning optimization of analog circuits, characterized in that, The method comprises the following steps: setting a set of ideal performance parameters of the simulation circuit; clustering components according to their transient current curves to obtain a plurality of component groups; configuring an agent for each of the component groups, so that the agent outputs circuit parameter values according to the set of ideal performance parameters; applying the circuit parameter values to the simulation circuit for simulation to obtain simulation circuit performance values; updating the agent according to the simulation circuit performance values and a preset target performance value to obtain a target agent.

2. The method of claim 1, wherein, The setting of the set of ideal performance parameters of the simulation circuit comprises: parameterizing components in the circuit to set parameter ranges of the components; setting ranges of performance indicators according to preset circuit design requirements; converting the design intention of the simulation circuit into quantitative weights for guiding the evaluation of the overall performance of the circuit; composing the set of ideal performance parameters of the simulation circuit from the parameter ranges of the components, the ranges of the performance indicators, and the quantitative weights.

3. The multi-agent reinforcement learning optimization method of analog circuits according to claim 2, wherein, The conversion of the design intention of the simulation circuit into quantitative weights for guiding the evaluation of the overall performance of the circuit comprises: comparing different performance indicators according to the design intention to obtain an importance comparison matrix between different performance indicators; extracting importance weight values of each performance indicator from the importance comparison matrix; performing consistency test on the importance weight values of the performance indicators, and calculating the quantitative weights after the test is passed.

4. The method of claim 1, wherein, The clustering of components according to their transient current curves to obtain a plurality of component groups comprises: calculating dynamic time warping distances between different components according to their transient current curves; clustering and merging initial clustering clusters with similar distances according to the dynamic time warping distances to obtain clustering clusters; determining a silhouette coefficient of a target component according to the average dynamic time warping distance between the target component and other components in the same cluster and the average dynamic time warping distance between the target component and components in the nearest neighbor cluster, determining an optimal clustering number according to the silhouette coefficient, and using the silhouette system to evaluate the clustering quality under different clustering numbers; clustering components according to the optimal clustering number to generate candidate component groups; selecting component groups from the candidate component groups according to the similarity between the candidate component groups.

5. The method of claim 1, wherein, The agent outputs circuit parameter values according to the set of ideal performance parameters, which comprises: selecting a target set of ideal performance parameters from a plurality of sets of ideal performance parameters; a policy network in the agent outputs circuit parameter values according to the target set of ideal performance parameters.

6. The method of claim 1, wherein, The application of the circuit parameter values to the simulation circuit for simulation to obtain simulation circuit performance values comprises: updating a circuit netlist according to the circuit parameter values to generate an instance of the simulation circuit; performing simulation and circuit performance analysis on the simulation circuit through a simulator to obtain simulation circuit performance values.

7. The method of claim 1, wherein, The updating of the agent according to the simulation circuit performance values and a preset target performance value to obtain a target agent comprises: obtaining a reward value according to the difference between the simulation circuit performance values and the preset target performance value; constructing observation information according to the reward value and state information of the simulation circuit; updating the agent according to the observation information to obtain the target agent.

8. The method of claim 7, wherein, The updating of the agent according to the observation information obtains a target agent, and the method comprises the steps of: standardizing the reward values of the same batch of samples to obtain advantage values; determining a target function according to the advantage values; updating the agent according to the target function to obtain a target agent.

9. An electronic device, comprising: comprise: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the multi-agent reinforcement learning optimization method of the analog circuit according to any one of claims 1 to 8 when executing the computer program.

10. A computer storage medium, characterized in that computer executable instructions are stored, and the computer executable instructions are used for executing the multi-agent reinforcement learning optimization method of the analog circuit according to any one of claims 1 to 8.

Images