Parasitic parameter extraction neural network distributed training method and device

Abstract

The embodiment of the invention provides a parasitic parameter extraction neural network distributed training method and device, and relates to the technical field of integrated circuits. According to the method, a local batch of each processing device is obtained by using a data pushing process in each training batch, the data reading efficiency is improved, the memory is saved, and parallel training is performed on a parasitic parameter extraction neural network on the processing device by using a Horovod parallel framework to obtain a local gradient. Local BN layer statistical parameters of each processing device are obtained to calculate global statistical parameters, the global statistical parameters are distributed to each processing device to perform BN layer data updating, local gradients are updated at the same time, all local gradients are synchronized by using a Horovod framework to determine model updating parameters, and the model updating parameters are updated; and training is continued until the trained parasitic parameter extraction neural network is obtained. On the basis of a Horovod parallel framework, cross-card-batch normalization synchronization operation is realized, the situation that training is not converged due to the fact that local batches are too small is avoided, the training precision can be maintained, and the training efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of integrated circuit technology, and in particular to a parasitic parameter extraction neural network distributed training method, device, equipment and storage medium. Background Art

[0002] In the design process of integrated circuits, it is very important to accurately extract parasitic parameters (such as parasitic capacitance, parasitic resistance, etc.). The circuit simulation after adding the extracted parasitic parameters to the layout is called "post-simulation". Design engineers continue to modify transistor parameters or adjust circuit structures based on the post-simulation results until the simulation results of the post-simulation meet the design indicators and system function requirements. The design of the integrated circuit is considered complete.

[0003] With the continuous development of artificial intelligence, related technologies use neural network models to extract parasitic parameters on integrated circuit boards. In order to improve the prediction accuracy of the model, it is necessary to obtain more sampling points to generate a large-scale data set, so as to accurately characterize the conductor and calculate the area to extract parasitic parameters with high accuracy. This makes the training of the current neural network model used for parasitic parameter extraction relatively time-consuming. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a distributed training method, device, equipment and storage medium for parasitic parameter extraction neural network to improve the training efficiency of the parasitic parameter extraction neural network.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a distributed training method for a parasitic parameter extraction neural network, comprising:

[0006] generating at least one training batch based on point cloud data corresponding to interconnect conductors in the integrated circuit board;

[0007] Generate at least one data push process, and in each training batch, use the data push process to obtain a local batch corresponding to each processing device, each of the local batches including at least one training sample data;

[0008] The parasitic parameter extraction neural network deployed on the processing device is trained in parallel using the Horovod parallel framework to obtain local gradients. For each BN layer in the parasitic parameter extraction neural network, the local BN layer statistical parameters corresponding to the local batch of each processing device are obtained, and then the global BN layer statistical parameters are calculated. The global BN layer statistical parameters are distributed to each processing device to update the data of the BN layer, and the local gradients are updated at the same time.

[0009] All the local gradients are synchronized using the Horovod parallel framework, the model update parameters of the parasitic parameter extraction neural network on each of the processing devices are determined, and the training is continued until the termination condition is reached to obtain the trained parasitic parameter extraction neural network.

[0010] In some embodiments, generating at least one training batch according to the point cloud data corresponding to the interconnect conductors in the integrated circuit board includes:

[0011] Obtaining point cloud coordinates, point cloud annotations, the number of conductors, and the number of Gaussian points of the interconnect conductor as the point cloud data;

[0012] Generate a plurality of training sample data according to the point cloud data, and divide the training sample data into a preset number of training batches;

[0013] A data pipeline object is generated, and the training sample data in the training batch are sequentially stored in a data queue of the data pipeline object.

[0014] In some embodiments, the obtaining of the local batch corresponding to each processing device by using the data push process includes:

[0015] Obtaining at least one of the training sample data of the current training batch from the data queue of the data pipeline by using the data push process;

[0016] The training sample data is pushed to the processing device through a socket by using the data pushing process, and all the training sample data received by each processing device is used as the corresponding local batch.

[0017] In some embodiments, there are multiple processing devices, and using the data push process to push the training sample data to the processing device through a socket includes:

[0018] load balancing the processing device using the socket;

[0019] Based on the result of load balancing, the processing device for receiving the training sample data is selected, and the training sample data is sent to the processing device.

[0020] In some embodiments, there are multiple processing devices, and using the data push process to push the training sample data to the processing device through a socket includes:

[0021] If the number of the training sample data received by each of the processing devices reaches the preset number of samples, or there is no processing device as a downstream node;

[0022] Entering a waiting state until one of the processing devices is selected to receive the training sample data.

[0023] In some embodiments, the obtaining of the local BN layer statistical parameters corresponding to the local batch of each of the processing devices and then calculating the global BN layer statistical parameters includes:

[0024] Obtain input data corresponding to each of the BN layers, and obtain an input square mean according to the mean of the squares of the input data;

[0025] Calculating the average of the local means to obtain a global mean, and calculating the square of the global mean to obtain a global mean square;

[0026] Obtaining a global variance according to a difference between the input square mean and the global mean square;

[0027] The global BN statistical parameter is obtained according to the global mean and the global variance.

[0028] In some embodiments, the parallel training of the deployed parasitic parameter extraction neural network on the processing device using the Horovod parallel framework includes:

[0029] Amplifying a preset learning rate according to the number of processing devices to obtain a training learning rate;

[0030] The parasitic parameter extraction neural network is trained in each of the processing devices using the Horovod parallel framework based on the training learning rate and the training sample data.

[0031] In some embodiments, the method of synchronizing all the local gradients using the Horovod parallel framework and determining the model update parameters of the parasitic parameter extraction neural network on each of the processing devices includes:

[0032] Calculate the average of all the local gradients to generate a gradient update parameter;

[0033] Obtaining current model parameters of the parasitic parameter extraction neural network on the first processing device;

[0034] Update the current model parameters according to the gradient update parameters to obtain model update parameters;

[0035] The model update parameters are distributed to each of the processing devices to update the corresponding parasitic parameter extraction neural network.

[0036] To achieve the above-mentioned purpose, the second aspect of the embodiment of the present application proposes a parasitic parameter extraction neural network distributed training device, comprising:

[0037] A data generation module: used to generate at least one training batch according to the point cloud data corresponding to the interconnect conductors in the integrated circuit board;

[0038] Data push module: used to generate at least one data push process, and in each training batch, use the data push process to obtain a local batch corresponding to each processing device, each of the local batches includes at least one training sample data;

[0039] Parallel training module: used to use the Horovod parallel framework to perform parallel training on the parasitic parameter extraction neural network deployed on the processing device to obtain local gradients, obtain local BN layer statistical parameters corresponding to the local batch of each processing device for each BN layer in the parasitic parameter extraction neural network, and then calculate the global BN layer statistical parameters, distribute the global BN layer statistical parameters to each processing device to update the data of the BN layer, and update the local gradients at the same time;

[0040] Network update module: used to synchronize all the local gradients using the Horovod parallel framework, determine the model update parameters of the parasitic parameter extraction neural network on each of the processing devices, continue training until the termination condition is reached, and obtain the trained parasitic parameter extraction neural network.

[0041] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0043] The parasitic parameter extraction neural network distributed training method, device, equipment and storage medium proposed in the embodiment of the present application generate at least one training batch according to the point cloud data corresponding to the interconnect conductor in the integrated circuit board; generate at least one data push process, in each training batch, use the data push process to obtain the local batch corresponding to each processing device, each local batch includes at least one training sample data; use the Horovod parallel framework to parallel train the deployed parasitic parameter extraction neural network on the processing device to obtain local gradients, for each BN layer in the parasitic parameter extraction neural network, obtain the local BN layer statistical parameters corresponding to the local batch of each processing device, and then calculate the global BN layer statistical parameters, distribute the global BN layer statistical parameters to each processing device to update the BN layer data, and update the local gradients at the same time; use the Horovod parallel framework to synchronize all local gradients, determine the model update parameters of the parasitic parameter extraction neural network on each processing device, continue training until the termination condition is reached, and obtain the trained parasitic parameter extraction neural network. The embodiment of the present application first distributes the training process of the parasitic parameter extraction neural network to multiple different processing devices in parallel, and improves the training efficiency by means of parallel training. At the same time, the parasitic parameter extraction neural network is globally updated by converging local gradients to improve the convergence speed and thus improve the training efficiency. And in view of the fact that the training samples corresponding to the parasitic parameter extraction neural network need to predict the capacitance or resistance based on the shape, position, and distribution of the conductor, the amount of data for each training sample data is large, which will make the local batch on each processing device too small. Based on this, the embodiment of the present application obtains the global BN statistical parameters according to the local BN statistical parameters of each processing device, and uses the global BN statistical parameters to carry out batch normalization operations across multiple processing devices to avoid the situation where the training accuracy is low due to the small local batch. Therefore, in the training process of the parasitic parameter extraction neural network, it is possible to maintain the training accuracy, improve the training efficiency, and reduce the training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the parasitic parameter extraction neural network distributed training method provided in the embodiment of the present application.

[0045] Figure 2 It is a flowchart of generating at least one training batch according to the point cloud data corresponding to the interconnect conductors in the integrated circuit board provided by the embodiment of the present application.

[0046] Figure 3 It is a flowchart of using a data push process to obtain a local batch corresponding to each processing device provided by an embodiment of the present application.

[0047] Figure 4It is a flowchart of using a data push process to push training sample data to a processing device through a socket, provided in an embodiment of the present application.

[0048] Figure 5 This is another flow chart of using a data push process to push training sample data to a processing device through a socket, provided in an embodiment of the present application.

[0049] Figure 6 It is a flowchart of the parallel training of the deployed parasitic parameter extraction neural network on the processing device using the Horovod parallel framework provided in the embodiment of the present application.

[0050] Figure 7 It is a flowchart provided in an embodiment of the present application for calculating global BN layer statistical parameters after obtaining local BN layer statistical parameters corresponding to local batches of each processing device.

[0051] Figure 8 It is a flowchart of using the Horovod parallel framework to synchronize all local gradients and determine the model update parameters of the parasitic parameter extraction neural network on each processing device provided by an embodiment of the present application.

[0052] Fig. 9 It is a schematic diagram of training time comparison provided in an embodiment of the present application.

[0053] Fig.10 It is a structural block diagram of a parasitic parameter extraction neural network distributed training device provided in another embodiment of the present application.

[0054] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] First, some nouns involved in this application are analyzed:

[0059] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0060] Processing device: such as a CPU card or a GPU card. "Single machine, single card" means that only one CPU card or one GPU card is installed on one hardware device; while "multiple machines, multiple cards" means that multiple CPU cards or multiple GPU cards are installed on multiple hardware devices.

[0061] In the design process of integrated circuits, it is very important to accurately extract parasitic parameters (such as parasitic capacitance, parasitic resistance, etc.). The circuit simulation after adding the extracted parasitic parameters to the layout is called "post-simulation". Design engineers continue to modify transistor parameters or adjust circuit structures based on the post-simulation results until the simulation results of the post-simulation meet the design indicators and system function requirements. The design of the integrated circuit is considered complete.

[0062] With the continuous development of artificial intelligence, related technologies use neural network models to extract parasitic parameters on integrated circuit boards. In order to improve the prediction accuracy of the model, it is necessary to obtain more sampling points to generate a large-scale data set, so as to accurately characterize the conductor and calculate the area to extract parasitic parameters with high accuracy. For example, for the extraction of parasitic capacitance of large layouts and a large number of conductors, it is necessary to use a large amount of interconnect conductor data to train the neural network model. For example, the unsupervised physical information neural network capacitor extraction model is a neural network model for parasitic parameter extraction. This model includes both the supervised part of data training and the unsupervised physical information part. Specifically, the interconnect conductor structure is modeled using a point cloud neural network, and on this basis, the physical information neural network encoding electric field equation is introduced. The trained neural network model can directly solve the parasitic parameters based on the interconnect conductor structure. When the data set increases, the training of the current neural network model for parasitic parameter extraction is relatively time-consuming. Although the use of Tensorflow in related technologies for parallel training of models can improve training efficiency to a certain extent, Tensorflow usually uses its own API tf.data to process larger data sets. This method still has problems such as excessive memory consumption and lack of support for distributed data loading, resulting in a large loss of time and storage resources in the data set loading process. For the neural network model for parasitic parameter extraction, it is necessary to obtain more sampling points to generate a large-scale data set, which results in a relatively small training batch on a single processing device. When the data set is large, it will affect the convergence effect of the model and reduce the calculation accuracy.

[0063] Based on this, the embodiment of the present application provides a parasitic parameter extraction neural network distributed training method, device, equipment and storage medium. First, the training process of the parasitic parameter extraction neural network is distributed in parallel to multiple different processing devices, and the training efficiency is improved by means of parallel training. At the same time, the parasitic parameter extraction neural network is globally updated by converging local gradients to improve the convergence speed, thereby improving the training efficiency. And in view of the fact that the training samples corresponding to the parasitic parameter extraction neural network need to predict the capacitance or resistance based on the shape, position and distribution of the conductor, the data volume of each training sample data is large, which will make the local batch on each processing device too small. Based on this, the embodiment of the present application obtains the global BN statistical parameters according to the local BN statistical parameters of each processing device, and uses the global BN statistical parameters to carry out batch normalization operations across multiple processing devices to avoid the situation where the training accuracy is low due to the small local batch. Therefore, in the training process of the parasitic parameter extraction neural network, both the training accuracy can be maintained and the training efficiency can be improved, and the training time can be reduced.

[0064] The embodiments of the present application provide a parasitic parameter extraction neural network distributed training method, apparatus, device and storage medium, which are specifically illustrated by the following embodiments. First, the parasitic parameter extraction neural network distributed training method in the embodiments of the present application is described.

[0065] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0066] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0067] The parasitic parameter extraction neural network distributed training method provided in the embodiment of the present application relates to the field of integrated circuit technology. The parasitic parameter extraction neural network distributed training method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be a computer program running in a terminal or a server side. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in an operating system to run, such as a client that supports distributed training of parasitic parameter extraction neural networks, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form. Among them, the terminal communicates with the server via a network. The parasitic parameter extraction neural network distributed training method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0068] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the parasitic parameter extraction neural network distributed training method of this embodiment to provide related services and enhance the driving experience. The server may be an independent server, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, and each service node in the blockchain system forms a peer-to-peer (Peer To Peer, P2P) network, and the P2P protocol is an application layer protocol running on the Transmission Control Protocol (Transmission Control Protocol, TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (Universal Serial Bus, USB) or network communication connection methods, and this embodiment is not limited here.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] The following describes the parasitic parameter extraction neural network distributed training method in the embodiment of the present application.

[0071] Figure 1 is an optional flow chart of the parasitic parameter extraction neural network distributed training method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps 110 to 140. It can also be understood that this embodiment Figure 1The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0072] Step 110: Generate at least one training batch based on the point cloud data corresponding to the interconnect conductors in the integrated circuit board.

[0073] In one embodiment, an integrated circuit board generally includes a plurality of conductors, and different conductors may be connected by wires. Therefore, the connection relationship between the conductors in the integrated circuit board can be obtained by scanning or obtaining by integrated circuit software (such as EDA tools). The connection relationship can be represented as point cloud data. After obtaining the point cloud data, a plurality of training sample data can be constructed based on the point cloud data.

[0074] In one embodiment, referring to Figure 2 , Figure 2 The flowchart of generating at least one training batch according to the point cloud data corresponding to the interconnect conductors in the integrated circuit board provided by the embodiment of the present application specifically includes the following steps:

[0075] Step 210: Obtain the point cloud coordinates, point cloud annotations, number of conductors, and number of Gaussian points of the interconnect conductor as point cloud data.

[0076] In one embodiment, the point cloud data includes one or more of point cloud coordinates, point cloud annotations, the number of conductors, or the number of Gaussian points, because these parameters are closely related to the calculation process of parasitic capacitance. For example, the point cloud coordinates can provide the geometric position information of the interconnect wire, which is the basis for calculating the parasitic capacitance. Through these coordinates, the key parameters such as the length, width, and spacing of the wire can be determined, so as to estimate the parasitic capacitance between the wires. The point cloud annotation provides information on the function and connection relationship of the wire, which helps to distinguish which wires may have parasitic capacitance effects. For example, the parasitic capacitance (Csd) between the source and the drain is different from the parasitic capacitance (Cgs) between the gate and the source, and their effects on circuit performance are also different. And these parasitic capacitances can be more accurately identified and calculated through point cloud annotation. The number of conductors directly affects the total value of parasitic capacitance. In complex integrated circuits, more conductors mean more opportunities for mutual coupling, which may lead to larger parasitic capacitance, and by counting the number of conductors, the total value of parasitic capacitance can be preliminarily estimated. When calculating parasitic capacitance, it may be necessary to perform integral operations on certain areas, so it is necessary to use the number of Gaussian points to implement the Gaussian quadrature method in numerical integration. The Gaussian quadrature method can improve the calculation accuracy by selecting appropriate Gaussian points to approximate these integrals. More Gaussian points usually mean higher calculation accuracy, but it will also increase the amount of calculation. Therefore, the embodiment of the present application needs to weigh the calculation accuracy and calculation efficiency when selecting the number of Gaussian points. It is understandable that other data related to the calculation of parasitic parameters can also be selected as point cloud data, and the embodiment of the present application is not limited to this.

[0077] In one embodiment, the acquired point cloud data is written into a fixed HDF5 (Hierarchical Data Format version 5) file by configuring a standard channel to achieve unified storage. HDF5 is a file format for storing large amounts of data. It supports data compression and parallel I / O and is suitable for storing large amounts of data.

[0078] Step 220: Generate multiple training sample data according to the point cloud data, and divide the training sample data into a preset number of training batches.

[0079] In one embodiment, after obtaining a large amount of point cloud data, multiple training sample data can be generated according to the point cloud data to form a training sample set. However, when the training sample set is very large, loading all the training sample data into the memory at one time for training may cause memory overflow. Therefore, the embodiment of the present application divides the training sample set into a preset number of training batches according to the computing power of the processing device, and part of the training sample data can be used in each training batch to perform the model training process of the parasitic parameter extraction neural network, thereby saving memory resources. It can be understood that the preset number here is set according to actual needs, and this embodiment does not limit this.

[0080] Step 230: Generate a data pipeline object, and store the training sample data in the training batch in the data queue of the data pipeline object in sequence.

[0081] In one embodiment, each training batch not only includes a corresponding plurality of training sample data but also adds an additional dimension: batch size batch_size, which indicates the data range of the training batch. Next, a Python iterator object is established as a data pipeline object, and the training sample data is called through the data pipeline object, which can be understood as arranging all the training batches into a data list in a linear order.

[0082] In one embodiment, the data pipeline object contains two basic methods: iter() and len(). The iter() method is used to return an iterator object, which itself contains a next method. This iterator object reads all the training sample data in the data list in the data pipeline one by one, so that the training sample data is read item by item. The len() method returns the size of the data pipeline, that is, the total number of training sample data read, thereby characterizing the progress of data processing. In the embodiment of the present application, the data pipeline object allows access to the training sample data one by one, rather than loading the entire training sample set into the memory at one time. It can be understood that when the data pipeline object reads the training batches one by one, different training batches can be distinguished according to the batch size parameter.

[0083] Step 120: Generate at least one data push process, and in each training batch, use the data push process to obtain a local batch corresponding to each processing device.

[0084] In one embodiment, in order to improve the training efficiency, the Horovod parallel framework is used for parallel training, and the parasitic parameter extraction neural network is deployed on multiple processing devices at the same time. Then, the data of the same training batch is distributed to these processing devices. Each processing device receives different training sample data, and uses the received training sample data to train the corresponding parasitic parameter extraction neural network, thereby achieving the purpose of parallel training and improving training efficiency.

[0085] In one embodiment, considering that the Horovod parallel framework does not provide a large data set segmentation solution, the embodiment of the present application generates multiple data push processes to distribute data for the training sample set in the data pipeline object, uses the data push process to connect the processing device, and uses the multi-threaded data push process method, combined with ZeroMQ cross-process transmission technology and Python iterator to implement an efficient data loading algorithm, which can process training sample data in parallel, reduce data preparation delays, and save memory. Figure 3 , Figure 3 This is a flowchart of using a data push process to obtain a local batch corresponding to each processing device provided by an embodiment of the present application, which specifically includes the following steps:

[0086] Step 310: Obtain at least one training sample data of the current training batch from the data queue of the data pipeline using the data push process.

[0087] In one embodiment, each time a data push process is used to distribute a training batch of data, the data push process calls iter() from the data queue of the data pipeline to obtain the training sample data of the current training batch. For multiple data push processes, all the training sample data of the current training batch are distributed, which can be evenly distributed or in other distribution methods. At this time, each data push process obtains at least one training sample data.

[0088] Step 320: Use the data push process to push the training sample data to the processing device through the socket, and use all the training sample data received by each processing device as the corresponding local batch.

[0089] In one embodiment, each local batch includes at least one training sample data. If the number of data push processes is consistent with the number of processing devices, each data push process corresponds to a processing device, and all training sample data in the data push process are sent to the corresponding processing device, and the processing device takes all the received training sample data as its corresponding local batch. It is understandable that the sizes of local batches of different processing devices can be different or the same.

[0090] In one embodiment, the data push process uses the ZMQ_PUSH socket of ZeroMQ as the PUSH end to push the training sample data to the downstream processing device. ZeroMQ is a multi-threaded network library based on message queues. It is a new layer in network communication, between the application layer and the transport layer. It is a scalable layer that can run in parallel and is distributed among distributed systems. The data push process of the embodiment of the present application utilizes the pipeline model in the ZeroMQ multi-threaded network library to take the corresponding training sample data from the data pipeline object and send it to the corresponding processing device for subsequent parallel training. In addition, using the Python iterator object to read data can access the training sample data one by one without loading the entire data set into the memory, saving memory and reducing latency. At the same time, the use of the ZeroMQ pipeline model to process data in parallel can quickly realize complex communication modes and efficiently transfer training data, thereby realizing efficient data pipelines and improving the training efficiency of the distributed training process.

[0091] In one embodiment, the same data push process may be connected to more than one processing device. In this case, the ZMQ_PUSH socket needs to push the training sample data to multiple processing devices. Figure 4 , Figure 4 This is a flowchart of using a data push process to push training sample data to a processing device through a socket provided by an embodiment of the present application, which specifically includes the following steps:

[0092] Step 410: Use sockets to load balance the processing devices.

[0093] Step 420: Based on the result of load balancing, select a processing device for receiving training sample data, and send the training sample data to the processing device.

[0094] In one embodiment, when there are multiple processing devices, the ZMQ_PUSH socket will perform load balancing internally, and push the training sample data to the processing device according to the load balancing result. For example, an average distribution algorithm can be used to poll the training sample data among all downstream processing devices, thereby ensuring that all training sample data acquired by the data push process can be evenly distributed to each processing device. This load balancing mechanism helps to avoid the situation where the local batches on some processing devices are too large and the calculation is overloaded, while the local batches of other processing devices are too small and the computing power is idle, resulting in the problem that different processing devices have time-consuming training and large differences in convergence performance during parallel training, thereby improving the training efficiency and training accuracy of the parasitic parameter extraction neural network as a whole.

[0095] In one embodiment, in order to ensure the transmission reliability of the training sample data, the ZMQ_PUSH socket also needs to confirm the availability of the processing device. Figure 5 , Figure 5 This is another flow chart of using a data push process to push training sample data to a processing device through a socket provided by an embodiment of the present application, which specifically includes the following steps:

[0096] Step 510: If the number of training sample data received by each processing device reaches the preset number of samples, or there is no processing device as a downstream node.

[0097] Step 520: Enter a waiting state until a processing device is selected to receive training sample data.

[0098] In one embodiment, when the ZMQ_PUSH socket transmits training sample data to the processing device, the processing device will open a socket buffer to store the training sample data. If the storage in the buffer is full, it is considered that the processing device has reached the high water mark and can no longer receive more training sample data. It can also be considered that the number of training sample data received by each processing device has reached the preset number of samples. Alternatively, there is no processing device as a downstream node, that is, there is no processing device that can receive the training sample data. At this time, in order to avoid data loss, the embodiment of the present application can enter a waiting state until at least one processing device is able to receive the training sample data, at which time data is sent instead of discarding the training sample data.

[0099] Step 130: Use the Horovod parallel framework to perform parallel training on the deployed parasitic parameter extraction neural network on the processing device to obtain local gradients. For each BN layer in the parasitic parameter extraction neural network, obtain the local BN layer statistical parameters corresponding to the local batch of each processing device and then calculate the global BN layer statistical parameters. Distribute the global BN layer statistical parameters to each processing device to update the BN layer data and update the local gradients at the same time.

[0100] In one embodiment, the deployed parasitic parameter extraction neural network is trained in parallel in each processing device to obtain the local gradient corresponding to each training batch and form a local gradient list. Figure 6 , Figure 6 The flowchart of the embodiment of the present application for parallel training of a parasitic parameter extraction neural network deployed on a processing device using the Horovod parallel framework specifically includes the following steps:

[0101] Step 610: Amplify the preset learning rate according to the number of processing devices to obtain a training learning rate.

[0102] In one embodiment, the preset learning rate needs to be amplified according to the number of processing devices during the training process to obtain the training learning rate. This is because in distributed training, the amount of data that each processing device needs to process is reduced, so the preset learning rate needs to be increased to compensate for the increased batch size. For example, if 4 processing devices are used for parallel training, each processing device only processes 1 / 4 of the total data volume, so the training learning rate needs to be multiplied by 4.

[0103] Step 620: Using the Horovod parallel framework, in each processing device, the parasitic parameter extraction neural network is trained based on the training learning rate and the training sample data.

[0104] In one embodiment, a training learning rate is applied in each processing device, and a parasitic parameter extraction neural network is trained according to the training learning rate based on the training sample data corresponding to each processing device. During the training process, an optimizer is used to obtain the gradient information of the parasitic parameter extraction neural network on each processing device. Specifically, hvd.DistributedOptimizer is used to encapsulate the original optimizer of each processing device, and the logic of distributed training is integrated into the optimizer. Then, during the training process, the original optimizer is used to calculate the local gradient, for example, the optimizer.compute_gradients method of TensorFlow is used to calculate the local gradient.

[0105] In one embodiment, the training process of multiple processing devices is constructed based on the Horovod distributed training framework. First, the Horovod framework is initialized through Hvd.init(), and each processing device is bound to a processing process. Through this binding method, each processing device can focus on its own training tasks, improving overall efficiency and stability. At the same time, a unique process number rank is assigned to each processing device for resource allocation identification. During the parallel training process, the orderly execution of the training tasks in each processing device is clarified through the process number. At the same time, through the one-to-one binding of processing devices and processing processes, resource conflicts caused by multiple processing processes accessing the same processing device at the same time can be effectively avoided.

[0106] In one embodiment, during the implementation of the Horovod distributed training framework, it is necessary to modify a training script that only supports a single machine and a single card to a training script that supports multiple machines and multiple cards. Taking the original Tensorflow framework as an example, only about 10 lines of code need to be added, and no cumbersome operations such as device settings of the Parameter Server architecture are involved. The pseudo code is as follows:

[0107] Import tensorflow as tf

[0108] Import Horovod as hvd

[0109] #Initialize Horovod

[0110] Hvd.init()

[0111] #Pin GPU to process local rank

[0112] config=tf.ConfigProto()

[0113] Config.gpu_options.visible_device_list=str(hvd.local_rank())

[0114] #Horovod Distributed Optimizer

[0115] opt=tf.train.AdagradOptimizer(learning_rate*hvd.size())

[0116] opt=hvd.DistributedOptimizer(opt)

[0117] #Add hook to broad cast variable from rank 0to all other processes

[0118] Hooks=[hvd.BroadcastGlobalVariablesHook(0)]

[0119] #Save checkpoints only on worker 0to prevent other workers fromcorrupting them

[0120] Checkpoint_dir=” / tmp / train_logs”if hvd.rank()==0else None

[0121] In the above code, hvd.init(): is used to initialize Horovod. This step must be called before any other Horovod function. Then bind the GPU to the local rank of the process, use tf.ConfigProto() to create a TensorFlow configuration object, and use config.gpu_options.visible_device_list=str(hvd.local_rank()) to set the visible GPU device to the local rank of the current process, which ensures that each process uses a different GPU. Next is to configure the Horovod distributed optimizer and create an Adagrad optimizer using tf.train.AdagradOptimizer(learning_rate*hvd.size()). Learning_rate represents the preset learning rate, and hvd.size() represents the number of processing devices. Next, use hvd.DistributedOptimizer(opt) to wrap the optimizer with Horovod's DistributedOptimizer to handle gradient synchronization between multiple processes. hvd.BroadcastGlobalVariablesHook(0) is used to transfer BN layer information using ring reduction, so that global BN layer statistical parameters are initialized from the process of rank 0 and broadcast to all other processes. checkpoint_dir = " / tmp / train_logs" if hvd.rank() == 0 else None is the checkpoint directory, which is used to set the checkpoint directory only in the process of rank 0 (usually the master node). Other processes do not save checkpoints to avoid damage.

[0122] In one embodiment, the BN layers between different processing devices in the Horovod parallel framework cannot communicate directly. Since the BN layer is widely used in the network structure of the parasitic parameter extraction neural network, if the BN layer needs to be synchronized, it involves communication between a large number of processing devices, so each synchronization process greatly affects the training speed. Therefore, during the training process, the embodiment of the present application uses hvd.all reduce() to implement cross-card batch normalization synchronization operations for each BN layer in the parasitic parameter extraction neural network. Figure 7 , Figure 7 The flowchart of calculating the global BN layer statistical parameters after obtaining the local BN layer statistical parameters corresponding to the local batch of each processing device provided by the embodiment of the present application specifically includes the following steps:

[0123] Step 710: Obtain input data corresponding to each BN layer, and obtain the input square mean according to the mean of the squares of the input data.

[0124] In one embodiment, the BN layer is a method for solving the internal covariate shift problem by normalizing the intermediate layers of the parasitic parameter extraction neural network layer by layer, so that the distribution of each intermediate layer input remains stable. The parasitic parameter extraction neural network includes one or more BN layers, for example, there is a BN layer after each full connection. The BN layer includes two trainable parameters (β, γ) and two local BN statistical parameters, and the local BN statistical parameters include the local mean μ and the local variance σ 2 , and the BN layer is calculated differently during the training process and the inference process.

[0125] During the training process, β, γ and other parameter weights in the parasitic parameter extraction neural network participate in the training process. Then, the mean of the input data is calculated as the batch mean μ1 based on the input data of the BN layer. Next, the batch variance is calculated based on the batch mean μ1 and the input data. Then according to the batch mean μ1 and batch variance Normalize the input data to get the normalized data, and finally scale and translate the normalized data to get the output data of the BN layer. The process is expressed as:

[0126]

[0127] Among them, x i represents the i-th input data, m represents the number of input data, represents the i-th normalized data, y i represents the i-th output data, ∈ represents a very small constant for numerical stability to prevent the denominator from being zero, and β, γ represent two trainable parameters.

[0128] During the training process, as the number of training batches increases, the BN layer will gradually update two local BN statistical parameters as the training process progresses: local mean μ and local variance σ 2 , where the local mean μ is the statistical value corresponding to the mean μ1 of all past batches, and the local variance σ 2 is the variance of all past batches The corresponding statistical values ​​and statistical calculation process are as follows:

[0129] μ=decay×μ+(1-decay)×μ1

[0130]

[0131] Where decay represents the decay factor. In an example, suppose there are three sets of batch means μ1 and batch variances: They are: μ1=10, μ1=12, μ1=11, Assume that the initial local mean μ and local variance σ 2 Both are 0, and the decay factor decay = 0.9.

[0132] At this time, after the first training batch: μ = 0.9 × 0 + 0.1 × 10 = 1, σ 2 =0.9×0+0.1×5=0.5; after the second training batch: μ=0.9×1+0.1×12=1.1, σ 2 =0.9×0.5+0.1×4=0.85; after the third training batch: μ=0.9×1.1+0.1×11=1.29, σ 2 =0.9×0.85+0.1×6=1.37.

[0133] It can be seen that the local mean and local variance can be accumulated in BN, reflecting the statistical characteristics of the data during the entire training process.

[0134] During the inference process, the local mean and local variance are used to update the data instead of the batch mean and batch contrast in the training process. The output data is expressed as:

[0135]

[0136] Based on the above process, the embodiment of the present application requires the local mean and local variance in each BN layer of the parasitic parameter extraction neural network in different processing devices to achieve synchronization across BN. Therefore, the embodiment of the present application obtains the input data corresponding to each BN layer, and obtains the input square mean according to the mean of the square of the input data, which is expressed as:

[0137]

[0138] Among them, m′ represents the total number of all input data, x i Represents the i-th input data.

[0139] Step 720: Calculate the average of the local means to obtain the global mean, and calculate the square of the global mean to obtain the global mean square.

[0140] In one embodiment, the local BN statistical parameters corresponding to the local batch calculated by the BN layer of each processing device are obtained. Specifically, in the Horovod framework, the BN_synchro function is redefined, that is, the BN function that can be synchronized across card batches can replace the original BN function, and hvd.all reduce() is used in the BN_synchro function to count the average of the local means of each training device to obtain the global mean μ′, and the square of the global mean μ′ is calculated to obtain the global mean square μ2 .

[0141] Step 730: Obtain the global variance according to the difference between the input square mean and the global mean square.

[0142] In one embodiment, the global variance σ′ 2 It is expressed as:

[0143]

[0144] Step 740: Obtain global BN statistical parameters according to the global mean and the global variance.

[0145] In one embodiment, the global BN statistical parameters are obtained according to the global mean and the global variance. In this way, the global BN statistical parameters can be calculated by synchronously acquiring the local mean on the processing device only once, which can reduce the traffic overhead and improve the training efficiency.

[0146] In one embodiment, after obtaining the global mean and global variance, the global BN statistical parameters are distributed to each processing device to update the data of the BN layer, wherein the corresponding BN layer can update the input data according to the global mean and global variance, which is expressed as:

[0147]

[0148] In one embodiment, hvd.all reduce() is used in the Horovod framework to count the local mean and local variance of each processing device. The specific steps are: adding a BN_Synchro operation in the parasitic parameter extraction neural network, and adding a call to update_ops through tf.control_dependencies() before calculating the local gradient for parasitic parameter extraction neural network training.

[0149] The specific definition of the BN_Synchro function is given below:

[0150] First, obtain the parameters (β, γ) and two statistical parameters (μ, σ 2). Then, the input data x and the square of the input data are averaged, and hvd.allreduce is used to count the local means of multiple processing devices to obtain the global mean, and the global variance is calculated according to the above formula. The global mean and global variance are used as the local mean and local variance of the BN layer to update the data of the BN layer. After that, the global mean and global variance are updated and added to tf.GraphKey.UPDATE_OPScollection. At this time, the local mean and local variance on each processing device are consistent. Finally, when inferring or not training, the local mean and local variance are used instead of the batch mean and batch variance in the training process to calculate the output number of the BN layer.

[0151] In one embodiment, the pseudo code is as follows:

[0152]

[0153]

[0154] Among them, batch_mean represents the batch mean, batch_mean_square represents the input square mean, mean represents the global mean, var represents the global variance, moving_mean represents the local mean, moving_var represents the local variance, and n_gpu represents the nth processing device.

[0155] In the above code, first, TensorFlow and Horovod are imported, and Horovod is initialized. Then, the parasitic parameter extraction neural network is called on each processing device. It contains several BN layers, represented by the BN_Synchro function. Next, the local gradient is calculated and the update operation set is obtained. Finally, the control dependency is used to ensure that the update operation is performed before the gradient is applied. In this BN_Synchro function, a variable scope is first defined, and four variables, gamma, beta, moving_mean, and moving_var, are created in it. If in training mode, the batch mean and batch variance are calculated, and the global BN statistical parameters are synchronized between multiple processing devices through the all_reduce method of Horovod. Then the global BN statistical parameters are used for batch normalization. And, in training mode, if the current device is GPU:0, moving_mean and moving_var are updated in it, and these update operations are added to the update operation set.

[0156] In one embodiment, in order to achieve the above-mentioned BN synchronization across processing devices, for the training process of multiple processing devices, it is necessary to first load the parasitic parameter extraction neural network in each processing device to obtain the gradient list, so that the local mean and local variance of the BN layer can be obtained during the training process, and after calculating the global mean and global variance, it is loaded into the training through tf.GraphKeys.UPDATE_OPS. The code is as follows:

[0157]

[0158] Among them, AdagradOptimizer represents the optimizer, learning_rate represents the training learning rate, n_gpu represents the nth processing device, and opt.compute_gradients(loss) represents the calculation of the local gradient of the loss function loss with respect to the model parameters. hvd.allreduce(grad,average): Use Horovod's allreduce operation to average the local gradients among all processing devices to obtain the gradient update parameters to support distributed training. averaged_gradients.append((avg_grad,var)): Save the gradient update parameters and the corresponding model update parameters to the list averaged_gradients. update_ops is a collection of all update operations (such as moving averages, etc.). tf.control_dependencies(update_ops) ensures that the operations in update_ops are completed before executing train_op, so that all update operations can be correctly executed in each training iteration. opt.apply_gradients(averaged_gradients,name='train_op'): Apply the calculated gradient update parameters to the model parameters to generate the training operation train_op.

[0159] That is, add a call to update_ops through tf.control_dependencies(). Before using update_ops, you need to add the update_ops of the BN layer to tf.Graphkey.Update_ops. TF's tf.layers.BatchNormalization will automatically add update_ops to the tf.GraphKey.UPDATE_OPS collection (which is only added during training, not during inference or non-training). Therefore, for BN synchronization of multiple processing devices, it is also necessary to add the synchronized global BN statistics parameters to f.GraphKey.UPDATE_OPScollection.

[0160] Step 140: Use the Horovod parallel framework to synchronize all local gradients, determine the model update parameters of the parasitic parameter extraction neural network on each processing device, and continue training until the termination condition is reached to obtain a trained parasitic parameter extraction neural network.

[0161] In one embodiment, referring to Figure 8 , Figure 8 The present invention provides a flowchart of synchronizing all local gradients using the Horovod parallel framework to determine the model update parameters of the parasitic parameter extraction neural network on each processing device, which specifically includes the following steps:

[0162] Step 810: Calculate the average of all local gradients to generate gradient update parameters.

[0163] In one embodiment, after the original optimizer of each processing device calculates the corresponding local gradient, hvd.DistributedOptimizer is used to inherit the original optimizer to obtain all local extractions, and then the obtained local gradients are averaged through hvd.allreduce() to obtain the averaged gradient update parameters of each processing process.

[0164] Step 820: Obtain current model parameters of the parasitic parameter extraction neural network on the first processing device.

[0165] In one embodiment, since parameters need to be shared on all processing devices, the current model parameters of the parasitic parameter extraction neural network on the first processing device (for example, the process number is rank 0) are obtained according to the process number rank, and the subsequent parameter update process is performed based on this.

[0166] Step 830: Update the current model parameters according to the gradient update parameters to obtain the model update parameters.

[0167] In one embodiment, the official optimizer.apply_gradients method of TensorFlow is called to process the current model parameters, and the model update parameters are returned based on the gradient update parameters.

[0168] Step 840: Distribute the model update parameters to each processing device to update the corresponding parasitic parameter extraction neural network.

[0169] In one embodiment, the model update parameters are distributed to each processing device by broadcasting and using the Ring-All-Reduce mechanism, so that each processing device updates the corresponding parasitic parameter extraction neural network according to the model update parameters, thereby achieving synchronization of the model parameters. The embodiment of the present application solves the problem that when the statistical quantity of a single processing device is very small, the local BN statistical parameters in its BN layer are greatly disturbed, causing the BN layer to fail, through the above-mentioned cross-BN layer synchronization process, thereby improving the overall training efficiency and calculation accuracy of the parasitic parameter extraction neural network.

[0170] In one embodiment, training is continued, and multiple training batches are selected one by one to execute the above-mentioned distributed training process until the termination condition is reached, and the final parasitic parameter extraction neural network can be obtained. It can be understood that the termination condition can be that all training batches are trained, or the number of training times reaches a preset number of times, or the performance of the parasitic parameter extraction neural network reaches a preset performance, etc., and the embodiment of the present application does not limit this.

[0171] The embodiment of the present application uses the Horovod distributed training framework to replace the native cluster distributed training method of Tensorflow, and performs all-reduce distributed training with Ring-All-Reduce synchronous gradient update on the unsupervised parasitic parameter extraction neural network. At the same time, it combines ZeroMQ cross-process transmission technology and Python iterators to design an efficient data loading process, which can process training sample data in parallel, reduce the delay caused by data preparation, and save memory. In addition, based on the Horovod distributed training data technology, cross-card synchronization of BN layer statistical parameters is achieved, improving calculation accuracy and efficiency.

[0172] In one embodiment, taking the processing device as a GPU card as an example, the training efficiency and accuracy verification process of the parasitic parameter extraction neural network is performed on a single GPU card, 4 GPU cards and 8 GPU cards respectively. Among them, three data sets are used, data set 1 (Test case 1) contains 5 conductor samples as training sets, each conductor sample contains 12,000 sampling points, and the test set and verification set are the same as the training set; data set 2 (Test case 2) contains 70 randomly generated conductor samples, each conductor sample contains 15,000 sampling points, of which 10 are randomly selected as test sets, 10 are verification sets, and the rest are training sets; data set 3 (Test case 3) contains 5,000 conductor samples, each conductor sample contains 15,000 sampling points, of which 1,000 are randomly selected as test sets, 1,000 are verification sets, and the rest are training sets. The training time and training accuracy of the parasitic parameter extraction neural network under different hardware conditions are as follows. Among them, Table 1 is a comparison of training time on different numbers of processing devices when Dataset 2 and Dataset 3 are selected, Table 2 is a comparison of parallel efficiency on different numbers of processing devices when Dataset 2 and Dataset 3 are selected, and Table 3 is a comparison of the effects of using or not calculating the global BN statistical parameters when Dataset 1 is selected.

[0173] Test case 2 Test case 3 1 GPU card 2h44 mins 21h18 mins 2 GPU cards 1h54 mins 12h58 mins 4 GPU cards 54mins 6h15 mins 8 GPU cards 33mins 3h44 mins

[0174] Table 1 Comparison of training time on different numbers of processing devices

[0175] 2 GPU cards 4 GPU cards 8 GPU cards Test case 2 0.71929825 0.75925926 0.62121212 Test case 3 0.82133676 0.852 0.71316964

[0176] Table 2 Comparison of parallel efficiency on different numbers of processing devices

[0177] Reference Fig. 9 , Fig. 9 is a schematic diagram of training time comparison provided in an embodiment of the present application, Fig. 9 The horizontal axis is the number of GPU cards, and the vertical axis is the speed improvement quantification parameter. Fig. 9 As shown in Table 1 and Table 2, the training process of the parasitic parameter extraction neural network can be significantly improved by parallelizing the training process, especially in the environment of multiple processing devices, where a good acceleration effect can be achieved, and the training time decreases almost linearly with the increase in the number of GPUs. It can also achieve a high parallel efficiency. For example, as shown in Table 2, the parallel efficiency of 8 GPU cards for the large-scale data set Test case 3 is 71%.

[0178]

[0179] Table 3 Comparison of the effects of using or not calculating global BN statistical parameters

[0180] It can be seen from Table 3 that after using cross-card BN synchronization, that is, calculating the global BN statistical parameters based on the local BN statistical parameters, a good convergence accuracy can be achieved. However, when the global BN statistical parameters are not calculated and the single-card Batchsize is small, the training accuracy is poor or even non-convergence occurs.

[0181] The extraction of parasitic parameters is of vital importance in electronic design. The key issue of full-chip capacitance extraction lies in speed. For circuits with a scale of millions, it is usually required to complete the extraction work within one day. Although the existing parasitic parameter extraction model can speed up the calculation speed of the capacitance extraction algorithm, with the increase of circuit conductor data, the scale of neural network parameters under single card data is increasing, and the training cycle is getting longer and longer, so training must be accelerated to improve work efficiency. The network structure of the parasitic parameter extraction neural network covers feature extraction and physical information encoding. How to design an efficient and convergent distributed parallel training method is a problem worthy of research and discussion.

[0182] Therefore, in the embodiments of the present application, the data parallelism and cross-BN layer synchronization strategies are combined to significantly improve the training efficiency while maintaining accuracy. Specifically, the Horovod-based parallel framework is adopted to overcome the problems of cumbersome implementation process of parallel training of TensorFlow native clusters and difficulty in tuning as the cluster scales up; at the same time, Python iterator technology and ZeroMQ cross-process communication technology are used to solve the problems of difficulty in loading large-scale data sets, high memory consumption and calculation delays. When the batchsize of a single card is small, there will be large disturbances in the local BN statistical parameters in the BN layer, which causes the BN layer to fail, which in turn leads to problems such as reduced calculation accuracy, so it is necessary to synchronize the global BN statistical parameters across the BN layer.

[0183] The technical solution provided by the embodiment of the present application generates at least one training batch according to the point cloud data corresponding to the interconnect conductor in the integrated circuit board; generates at least one data push process, in each training batch, uses the data push process to obtain the local batch corresponding to each processing device, and each local batch includes at least one training sample data; uses the Horovod parallel framework to train the deployed parasitic parameter extraction neural network in parallel on the processing device to obtain local gradients, and for each BN layer in the parasitic parameter extraction neural network, obtains the local BN layer statistical parameters corresponding to the local batch of each processing device and then calculates the global BN layer statistical parameters, and distributes the global BN layer statistical parameters to each processing device to update the BN layer data, and updates the local gradients at the same time; uses the Horovod parallel framework to synchronize all local gradients, determines the model update parameters of the parasitic parameter extraction neural network on each processing device, and continues training until the termination condition is reached to obtain a trained parasitic parameter extraction neural network. The embodiment of the present application first distributes the training process of the parasitic parameter extraction neural network in parallel to multiple different processing devices, and improves the training efficiency by means of parallel training. At the same time, the parasitic parameter extraction neural network is globally updated by aggregating local gradients to improve the convergence speed, thereby improving the training efficiency. And given that the training samples corresponding to the parasitic parameter extraction neural network need to predict capacitance or resistance based on information such as the shape, position, and distribution of the conductor, the amount of data for each training sample data is large, which will make the local batch on each processing device too small. Based on this, the embodiment of the present application obtains the global BN layer statistical parameters based on the local BN layer statistical parameters of each processing device, and uses the global BN layer statistical parameters to carry out batch normalization operations across multiple processing devices to avoid the situation where the training accuracy is low due to the local batch being too small. Therefore, in the training process of the parasitic parameter extraction neural network, it is possible to maintain the training accuracy, improve the training efficiency, and reduce the training time.

[0184] The present application also provides a parasitic parameter extraction neural network distributed training device, which can implement the above-mentioned parasitic parameter extraction neural network distributed training method, referring to Fig.10 , the device comprises:

[0185] The data generation module 1010 is used to generate at least one training batch according to the point cloud data corresponding to the interconnect conductors in the integrated circuit board.

[0186] Data push module 1020: used to generate at least one data push process, and in each training batch, use the data push process to obtain a local batch corresponding to each processing device, each local batch includes at least one training sample data.

[0187] Parallel training module 1030: used to use the Horovod parallel framework to perform parallel training on the deployed parasitic parameter extraction neural network on the processing device to obtain local gradients, obtain the local BN layer statistical parameters corresponding to the local batch of each processing device for each BN layer in the parasitic parameter extraction neural network, and then calculate the global BN layer statistical parameters, distribute the global BN layer statistical parameters to each processing device to update the BN layer data, and update the local gradients at the same time.

[0188] The network update module 1040 is used to determine the model update parameters of the parasitic parameter extraction neural network on each processing device according to all local gradients, and continue training until the termination condition is reached to obtain a trained parasitic parameter extraction neural network.

[0189] The specific implementation of the parasitic parameter extraction neural network distributed training device of this embodiment is basically the same as the specific implementation of the parasitic parameter extraction neural network distributed training method described above, and will not be repeated here.

[0190] The present application also provides an electronic device, including:

[0191] at least one memory;

[0192] at least one processor;

[0193] at least one program;

[0194] The program is stored in the memory, and the processor executes the at least one program to implement the above-mentioned parasitic parameter extraction neural network distributed training method implemented in this application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0195] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0196] The processor 1101 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0197] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the parasitic parameter extraction neural network distributed training method of the embodiment of this application;

[0198] Input / output interface 1103, used to implement information input and output;

[0199] The communication interface 1104 is used to realize the communication interaction between the device and other devices, and the communication can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 1105 is used to transmit information between the various components of the device (such as the processor 1101, the memory 1102, the input / output interface 1103 and the communication interface 1104);

[0200] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0201] An embodiment of the present application also provides a storage medium, which is a storage medium that stores a computer program. When the computer program is executed by a processor, the above-mentioned parasitic parameter extraction neural network distributed training method is implemented.

[0202] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0203] The parasitic parameter extraction neural network distributed training method, device, equipment and storage medium proposed in the embodiment of the present application generate at least one training batch according to the point cloud data corresponding to the interconnect conductor in the integrated circuit board; generate at least one data push process, in each training batch, use the data push process to obtain the local batch corresponding to each processing device, each local batch includes at least one training sample data; use the Horovod parallel framework to parallel train the deployed parasitic parameter extraction neural network on the processing device to obtain local gradients, for each BN layer in the parasitic parameter extraction neural network, obtain the local BN layer statistical parameters corresponding to the local batch of each processing device, and then calculate the global BN layer statistical parameters, distribute the global BN layer statistical parameters to each processing device to update the BN layer data, and update the local gradients at the same time; use the Horovod parallel framework to synchronize all local gradients, determine the model update parameters of the parasitic parameter extraction neural network on each processing device, continue training until the termination condition is reached, and obtain the trained parasitic parameter extraction neural network. The embodiment of the present application first distributes the training process of the parasitic parameter extraction neural network to multiple different processing devices in parallel, and improves the training efficiency by means of parallel training. At the same time, the parasitic parameter extraction neural network is globally updated by converging local gradients to improve the convergence speed and thus improve the training efficiency. And in view of the fact that the training samples corresponding to the parasitic parameter extraction neural network need to predict the capacitance or resistance based on the shape, position, and distribution of the conductor, the amount of data for each training sample data is large, which will make the local batch on each processing device too small. Based on this, the embodiment of the present application obtains the global BN layer statistical parameters according to the local BN layer statistical parameters of each processing device, and uses the global BN layer statistical parameters to carry out batch normalization operations across multiple processing devices to avoid the situation where the training accuracy is low due to the small local batch. Therefore, in the training process of the parasitic parameter extraction neural network, it is possible to maintain the training accuracy, improve the training efficiency, and reduce the training time.

[0204] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0205] It will be understood by those skilled in the art that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps. The device embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0206] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0207] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0208] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0213] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A parasitic parameter extraction neural network distributed training method, characterized in that: include: generating at least one training batch based on point cloud data corresponding to interconnect conductors in the integrated circuit board; Generate at least one data push process, and in each training batch, use the data push process to obtain a local batch corresponding to each processing device, each of the local batches including at least one training sample data; The parasitic parameter extraction neural network deployed on the processing device is trained in parallel using the Horovod parallel framework to obtain local gradients. For each BN layer in the parasitic parameter extraction neural network, the local BN layer statistical parameters corresponding to the local batch of each processing device are obtained, and then the global BN layer statistical parameters are calculated. The global BN layer statistical parameters are distributed to each processing device to update the data of the BN layer, and the local gradients are updated at the same time. All the local gradients are synchronized using the Horovod parallel framework, the model update parameters of the parasitic parameter extraction neural network on each of the processing devices are determined, and the training is continued until the termination condition is reached to obtain the trained parasitic parameter extraction neural network.

2. The parasitic parameter extraction neural network distributed training method according to claim 1, characterized in that: The step of generating at least one training batch according to the point cloud data corresponding to the interconnect conductors in the integrated circuit board comprises: Obtaining point cloud coordinates, point cloud annotations, the number of conductors, and the number of Gaussian points of the interconnect conductor as the point cloud data; Generate a plurality of training sample data according to the point cloud data, and divide the training sample data into a preset number of training batches; A data pipeline object is generated, and the training sample data in the training batch are sequentially stored in a data queue of the data pipeline object.

3. The parasitic parameter extraction neural network distributed training method according to claim 2, characterized in that: The method of using the data push process to obtain a local batch corresponding to each processing device includes: Obtaining at least one of the training sample data of the current training batch from the data queue of the data pipeline by using the data push process; The training sample data is pushed to the processing device through a socket by using the data pushing process, and all the training sample data received by each processing device is used as the corresponding local batch.

4. The parasitic parameter extraction neural network distributed training method according to claim 3, characterized in that: The number of the processing devices is multiple, and the using the data pushing process to push the training sample data to the processing devices through a socket includes: load balancing the processing device using the socket; Based on the result of load balancing, the processing device for receiving the training sample data is selected, and the training sample data is sent to the processing device.

5. The parasitic parameter extraction neural network distributed training method according to claim 3, characterized in that: The number of the processing devices is multiple, and the using the data pushing process to push the training sample data to the processing devices through a socket includes: If the number of the training sample data received by each of the processing devices reaches the preset number of samples, or there is no processing device as a downstream node; Entering a waiting state until one of the processing devices is selected to receive the training sample data.

6. The parasitic parameter extraction neural network distributed training method according to claim 1, characterized in that: The local BN statistical parameters include a local mean, and the obtaining of the local BN layer statistical parameters corresponding to the local batch of each processing device and then calculating the global BN layer statistical parameters includes: Obtain input data corresponding to each of the BN layers, and obtain an input square mean according to the mean of the squares of the input data; Calculating the average of the local means to obtain a global mean, and calculating the square of the global mean to obtain a global mean square; Obtaining a global variance according to a difference between the input square mean and the global mean square; The global BN statistical parameter is obtained according to the global mean and the global variance.

7. The parasitic parameter extraction neural network distributed training method according to claim 2, characterized in that: The Horovod parallel framework is used to parallel train the deployed parasitic parameter extraction neural network on the processing device, including: Amplifying a preset learning rate according to the number of processing devices to obtain a training learning rate; The parasitic parameter extraction neural network is trained in each of the processing devices using the Horovod parallel framework based on the training learning rate and the training sample data.

8. The parasitic parameter extraction neural network distributed training method according to claim 1, characterized in that: The method of synchronizing all the local gradients using the Horovod parallel framework and determining the model update parameters of the parasitic parameter extraction neural network on each processing device includes: Calculate the average of all the local gradients to generate a gradient update parameter; Obtaining current model parameters of the parasitic parameter extraction neural network on the first processing device; Update the current model parameters according to the gradient update parameters to obtain model update parameters; The model update parameters are distributed to each of the processing devices to update the corresponding parasitic parameter extraction neural network.

9. A parasitic parameter extraction neural network distributed training device, characterized in that: include: A data generation module: used to generate at least one training batch according to the point cloud data corresponding to the interconnect conductors in the integrated circuit board; Data push module: used to generate at least one data push process, and in each training batch, use the data push process to obtain a local batch corresponding to each processing device, each of the local batches includes at least one training sample data; Parallel training module: used to use the Horovod parallel framework to perform parallel training on the parasitic parameter extraction neural network deployed on the processing device to obtain local gradients, obtain local BN layer statistical parameters corresponding to the local batch of each processing device for each BN layer in the parasitic parameter extraction neural network, and then calculate the global BN layer statistical parameters, distribute the global BN layer statistical parameters to each processing device to update the data of the BN layer, and update the local gradients at the same time; Network update module: used to synchronize all the local gradients using the Horovod parallel framework, determine the model update parameters of the parasitic parameter extraction neural network on each of the processing devices, continue training until the termination condition is reached, and obtain the trained parasitic parameter extraction neural network.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the parasitic parameter extraction neural network distributed training method according to any one of claims 1 to 8 when executing the computer program.

11. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the parasitic parameter extraction neural network distributed training method according to any one of claims 1 to 8 is implemented.

Images