Directly learn neural networks using backpropagation on programmable device blocks

Abstract

An example method for training a neural network includes: defining a hardware building block (HBB), a neuron equivalent (NEQ), and a conversion process from the NEQ to the HBB; defining the neural network using the NEQ in a machine learning framework; training the neural network on a training platform; and converting the trained neural network into a netlist of the HBB using the conversion process to convert the NEQ in the neural network into the HBB in the netlist.

Description

Technical Field

[0001] Examples of the present disclosure relate generally to neural networks, and more particularly to neural networks that directly learn programmable device blocks using backpropagation. Background Art

[0002] As deep neural networks (DNNs) continue to grow in size to improve their predictive power, their memory and computational footprints are also increasing proportionally, making them increasingly difficult to deploy in energy- and resource-constrained hardware environments. DNNs using floating-point arithmetic contain a large amount of redundancy, which can be exploited to reduce computational and memory costs. These techniques include quantization, pruning, and low-rank factorization. Among the proposed quantization techniques, trained quantization methods, in which DNNs are quantized during the training process, have achieved some of the most promising results in reducing the workload while maintaining accuracy in the form of quantized neural networks (QNNs). The most extreme form of quantization is binary neural networks (BNNs), in which weights and activations are constrained to binary values, significantly reducing computational and memory costs while retaining high accuracy.

[0003] Previously, the strategy for creating DNN accelerators for programmable devices such as field-programmable gate arrays (FPGAs) was to map artificial neurons to a fixed hardware architecture by placing the learned parameters in memory and scheduling operations so that the correct computations occur between corresponding weights and activations. In turn, the fixed hardware architecture is implemented using lookup tables (LUTs), digital signal processors (DSPs), and other programmable device resources. Looking at the problem holistically, the fixed architecture acts as an interface between the structural capabilities and the computations performed by the DNN. However, this interface also acts as a barrier to optimizations that would enable the DNN to fully exploit the programmable device structure. Furthermore, the fixed architecture requires a non-trivial optimizing compiler that maps and schedules the DNN computations onto the hardware, which can lead to further performance degradation. Summary of the Invention

[0004] Techniques for learning neural networks directly from programmable device blocks using backpropagation are described. In one example, a method for training a neural network includes defining hardware building blocks (HBBs), neuron equivalents (NEQs), and a conversion process from the NEQs to the HBBs; defining a neural network using the NEQs in a machine learning framework; training the neural network on a training platform; and converting the trained neural network into a netlist of the HBB using a conversion process to convert the NEQs in the neural network into the HBB netlist.

[0005] In another example, a non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform a method for training a neural network, comprising: defining a hardware building block (HBB), a neuron equivalent (NEQ), and a conversion process from the NEQ to the HBB; defining the neural network using the NEQ in a machine learning framework; training the neural network on a training platform; and converting the trained neural network into a netlist of the HBB using the conversion process to convert the NEQ in the neural network into the HBB in the netlist.

[0006] In another example, a computing system includes a system memory configured to store code implementing a design tool; a training platform; and a processor coupled to the system memory and the training platform, the processor configured to execute code to perform training of a neural network by: defining hardware building blocks (HBBs), neuron equivalents (NEQs), and a conversion process from the NEQs to the HBBs; defining the neural network using the NEQs in a machine learning framework; training the neural network on the training platform; and converting the trained neural network into a netlist of the HBB using a conversion process to convert the NEQs in the neural network into the HBB netlist.

[0007] These and other aspects can be understood with reference to the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to be able to understand in detail the manner in which the features set forth above are described, a more particular description, briefly summarized above, may be made by reference to example implementations, some of which are illustrated in the accompanying drawings. It should be noted, however, that the drawings illustrate only typical example implementations and are therefore not to be considered as limiting the scope thereof.

[0009] Figure 1A is a block diagram depicting a hardware acceleration system according to an example.

[0010] Figure 1B is a block diagram depicting an accelerated application according to an example.

[0011] Figure 2 is a block diagram depicting a computing system ("computer") according to an example.

[0012] Figure 3 is a flowchart depicting a method of training a neural network according to an example.

[0013] Figure 4 This describes the execution of the example Figure 3 A flow chart of a method showing a portion of the method is shown.

[0014] Figure 5is a block diagram illustrating an example neural network with a sparse graph topology.

[0015] Figure 6A is a block diagram depicting a multiple integrated circuit (IC) programmable device according to an example.

[0016] Figure 6B is a block diagram depicting a programmable IC according to an example.

[0017] Figure 6C is a block diagram depicting a system-on-chip (SOC) implementation of a programmable IC according to an example.

[0018] Figure 6D A field programmable gate array (FPGA) implementation of a programmable IC according to an example is illustrated.

[0019] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements of one example may be beneficially incorporated in other examples. DETAILED DESCRIPTION

[0020] Various features are described below with reference to the accompanying drawings. It should be noted that the drawings may or may not be drawn to scale, and elements of similar structure or function are represented by the same reference numerals throughout the drawings. It should be noted that these figures are intended only to facilitate the description of the features. They are not intended to be an exhaustive description of the claimed invention or to limit the scope of the claimed invention. In addition, the illustrated examples do not necessarily illustrate all aspects or advantages. Aspects or advantages described in conjunction with a particular example are not necessarily limited to that example and may be practiced in any other example even if not so illustrated or explicitly described.

[0021] Techniques for learning neural networks directly from programmable device blocks using backpropagation are described. These techniques expose the building blocks of programmable device fabric (e.g., lookup tables (LUTs), block random access memories (BRAMs), and combinations of these) directly to machine learning frameworks. These techniques allow deep neural networks (DNNs) to be customized during the training phase in a manner that maps directly to efficient programmable device hardware implementations without the need to define custom overlay architectures or scheduling tools. Essentially, these techniques mirror the capabilities of programmable device fabric in the topology of neural networks by using artificial neurons with quantized inputs, quantized outputs, and arbitrary weights. Once the network is trained, each neuron can be mapped to the truth table of a single building block. The resulting netlist can be placed and routed onto the programmable fabric of a device of sufficient size as a highly pipelined, massively parallel circuit implementing the same neural network. These and other aspects are described below with reference to the accompanying drawings.

[0022] Figure 1A 1 is a block diagram illustrating a hardware acceleration system 100 according to an example. Hardware acceleration system 100 includes a host computing system 102. Host computing system 102 includes a hardware platform ("hardware 104") and a software platform ("software 106") executing on hardware 104. Hardware 104 includes a processing system 110, system memory 116, a storage device ("storage 118"), and a hardware accelerator 122. Software 106 includes an operating system (OS) 144, an acceleration stack 146, a host application 150, and competing threads 139.

[0023] Processing system 110 includes a microprocessor 112, support circuits 114, and a peripheral bus 115. Microprocessor 112 may be any type of general-purpose central processing unit (CPU), such as an x86-based processor, a processor, etc. The microprocessor 112 may include one or more cores and associated circuit systems (e.g., cache memory, memory management unit (MMU), interrupt controller, etc.). The microprocessor 112 is configured to execute program code that performs one or more operations described herein and may be stored in the system memory 116 and / or storage unit 118. The support circuit 114 includes various devices that cooperate with the microprocessor 112 to manage data flow between the microprocessor 112, system memory 116, storage device 118, hardware accelerator 122, or any other peripheral device. For example, the support circuit 114 may include a chipset (e.g., north bridge, south bridge, platform host controller, etc.), a voltage regulator, firmware (e.g., basic input and output system (BIOS)), etc. The support circuit 114 manages data flow between the microprocessor 112 and the peripheral bus 115 to which various peripheral devices such as the hardware accelerator 122 are connected. In some examples, microprocessor 112 may be a system-in-package (SiP), a system-on-chip (SOC), etc., which absorbs all or most of the functions of a chipset (e.g., north bridge, south bridge, etc.). Peripheral bus 115 may implement an expansion bus standard such as Peripheral Component Interconnect Express (PCIe), etc.

[0024] System memory 116 is a device that allows information, such as executable instructions and data, to be stored and retrieved. System memory 116 may include, for example, one or more random access memory (RAM) modules, such as double data rate (DDR) dynamic RAM (DRAM). Storage 118 includes local storage devices (e.g., one or more hard disks, flash memory modules, solid-state drives, and optical disks) and / or storage interfaces that enable computing system 102 to communicate with one or more network data storage systems. Hardware 104 may include various other conventional devices and peripherals of a computing system, such as a graphics card, a universal serial bus (USB) interface, and the like.

[0025] In one example, hardware accelerator 122 includes a programmable device 128 and RAM 126. Hardware accelerator 122 may optionally include non-volatile memory (NVM) 124. Programmable device 128 may be a field programmable gate array (FPGA) or a system-on-chip (SoC) with FPGA programmable logic and other embedded subsystems. NVM 124 may include any type of non-volatile memory, such as flash memory. RAM 126 may include DDR DRAM, etc. RAM 126 may be organized into discrete RAM banks 127, as further described below. Programmable device 128 is coupled to NVM 124 and RAM 126. Programmable device 128 is also coupled to peripheral bus 115 of processing system 110.

[0026] OS 144 may be any commercial operating system known in the art, such as Microsoft Mac Etc. The acceleration stack 146 includes drivers and libraries that provide an application programming interface (API) to the hardware accelerator 122 for its command and control.

[0027] Figure 1B 1 is a block diagram depicting an acceleration application 180 according to an example. The acceleration application 180 includes a host application 150 and an acceleration circuit 130. The acceleration circuit 130 is programmed in the programmable logic (PL) 3 of the programmable device 128 on the hardware accelerator 122. The host application 150 includes software executed on the microprocessor 112, which uses API calls to the acceleration stack 146 to call the acceleration circuit 130 to perform some work. The host application 150 may include a neural network application, a video processing application, a network processing application, or similar types of applications that offload some functions to the hardware accelerator 122.

[0028] Figure 2 2 is a block diagram depicting a computing system ("computer 200") according to an example. Computer 200 includes a software platform 204 executing on a hardware platform 202. Hardware platform 202 includes a central processing unit (CPU) 206, system memory 208, storage devices 210, support circuits 211, and a training platform 212. Software platform 204 includes an operating system (OS) 230 and design tools 235.

[0029] CPU 206 may be any type of general purpose central processing unit (CPU), such as an x86-based processor, a processor, etc. The CPU 206 may include one or more cores and associated circuit systems (e.g., cache memory, memory management unit (MMU), interrupt controller, etc.). The CPU 206 is configured to execute program code that performs one or more operations described herein and may be stored in the system memory 208 and / or storage device 210. The support circuit 211 includes various devices that cooperate with the CPU 206 to manage data flow between the CPU 206, system memory 208, storage device 210, training platform 212, hardware accelerator 214, or any other peripheral device. For example, the support circuit 211 may include a chipset (e.g., north bridge, south bridge, platform host controller, etc.), a voltage regulator, firmware (e.g., BIOS), etc. In some examples, the CPU 206 may be a system-in-package (SiP), a system on chip (SoC), etc., which absorbs all or most of the functions of the chipset (e.g., north bridge, south bridge, etc.).

[0030] System memory 208 is a device that allows information such as executable instructions and data to be stored and retrieved. System memory 208 may include, for example, one or more random access memory (RAM) modules, such as double data rate (DDR) dynamic RAM (DRAM). System memory 208 may store data 226 and program code ("code 228") that is processed and executed by CPU 206 to implement software platform 204. Storage device 210 includes local storage devices (e.g., one or more hard disks, flash memory modules, solid-state drives, and optical disks) and / or storage interfaces that enable computer 200 to communicate with one or more network data storage systems. Hardware platform 202 may include various other conventional devices and peripherals of a computing system, such as a graphics card, a universal serial bus (USB) interface, and the like.

[0031] The training platform 212 includes hardware 216, which may include processor(s), memory, input / output (IO) circuits, etc. In one example, the hardware 216 includes a graphics processing unit (GPU) and related supporting circuitry. In another example, the hardware 216 may include an application-specific integrated circuit (ASIC), a programmable IC, etc., and related supporting circuitry. In one example, the training platform 212 has higher performance than the hardware accelerator 122, but also consumes more energy than the hardware accelerator 122. The training platform 212 can be used to train a neural network.

[0032] OS 230 may be any commercial operating system known in the art, such as Microsoft Mac Etc. Design tools 235 include software that trains a neural network on training platform 212 and implements the neural network on a target programmable device.

[0033] Figure 3 3 is a flow chart depicting a method 300 for training a neural network according to an example. The method 300 can be performed by the design tool 235 on the computer 200 described above. The method 300 begins at step 302, where the design tool 235 defines a hardware building block (HBB), a neuron equivalent (NEQ), and a conversion process.

[0034] HBB is a primitive available in programmable device structures that can implement truth tables. For example, programmable structures can include lookup tables (LUTs), block RAMs (BRAMs), etc. HBB implements truth tables of different sizes. The number of input bits of the HBB truth table is represented by B A The number of output bits (i.e., the number of different output variables) is denoted by B W The contents of the truth table are called the configuration of the HBB. In general, any memory component that can implement a truth table of a certain size can be used as an HBB.

[0035] NEQ is a computational graph represented in a machine learning framework. In one example, a NEQ consists of a multiplication-accumulation operation on the input and learnable parameters, followed by a nonlinear function. NEQ includes quantized inputs and outputs, but can have floating-point parameters. Each NEQ is mathematically equivalent to a single HBB and can be converted to a single HBB via a conversion process. In general, a memory device can handle any form of reduction as long as the following conditions are met: Among them B A is the number of address bits of the memory component, C in is the number of input channels, and B i is the number of bits used for the i-th input channel. NEQ can include any number of element-wise or non-element-wise operations as long as the last operation satisfies the following constraints: Among them C out is the number of output channels, B O is the number of bits of the oth output channel, and B W is the number of output bits (e.g., memory width). Each NEQ includes parameters that can be learned using gradient methods such as standard backpropagation. Therefore, NEQ only includes semi-differentiable functions.

[0036] The conversion process takes a NEQ with trained parameters and outputs an HBB in its configuration. The HBB type to which the NEQ is mapped is statically determined (i.e., each NEQ type maps to only one HBB type). Since the NEQ has quantized inputs and quantized outputs, the configuration of the HBB (i.e., the truth table) can be simply generated by evaluating all permutations of the inputs to the trained NEQ and observing the outputs. Each (input, output) pair becomes a row in the truth table.

[0037] The following describes example HBB and NEQ. For example, consider a 6 to 1 LUT where B A =6 and B W =1. If other types of LUT configurations are available in the target device, these LUT configurations can also be used. In general, each NEQ can include a dot product operation and a nonlinear operation (e.g., a threshold operation). Optionally, the NEQ can include one or more additional operations, such as an element-by-element operation between the dot product operation and the nonlinear operation (e.g., batch normalization). Such NEQs are mapped to LUTs. The conversion includes enumerating each possible input (e.g., 2^6=64 possibilities in this article), evaluating the trained equation for each possible input, and entering the (input, output) pair into the HBB truth table.

[0038] In another example, a BRAM is used as an HBB. Although BRAM is described, any type of RAM element in the structure can be used. BRAM can be configured in a number of different ways, in each case, used as a memory device. For example, consider a BRAM that can be configured as two independent 18k memories or a single 36k memory. Further, each of these configurations can have several modes that can vary the number of bits per word and the capacity of the memory (e.g., for an 18k configuration, 16k 1-bit words, 8k 2-bit words, etc.) and (e.g., for a 36k configuration, 32k 1-bit words, 16k 2-bit words, etc.). In this example, for a 2×18k configuration, the following BRAMs are supported: A / B W Combinations: 14 / 1, 13 / 2, 12 / 4, 11 / 9, 10 / 18, and 9 / 36. Note that in this configuration, there are actually two parallel NEQs, each supporting independent B A / B W Modes, as listed above. For 1×36k configuration, the following B A / B WCombinations: 15 / 1, 14 / 2, 13 / 4, 12 / 9, 11 / 18, 10 / 36, and 9 / 72. NEQs that can be mapped to BRAMs are similar to NEQs that can be mapped to LUTs. However, unlike 6-to-1 LUTs, BRAMs can produce outputs with higher bit widths. Similar to LUT-based NEQs, as long as B W The constraints can contain any number of element-wise operations.

[0039] Figure 4 is a flowchart depicting a method for performing step 302 of method 300 according to an example. At step 402, design tool 235 identifies HBBs for a given programmable device structure. At step 404, for each HBB, design tool 235 defines one or more NEQs in a machine learning framework. Each NEQ can be trained using backpropagation. At step 406, for each NEQ type, design tool 235 identifies the conversion process from the learned neuron to its corresponding HBB.

[0040] Return to Figure 3 At step 304, the user interacts with the design tool 235 to define and train a DNN using NEQs in a machine learning framework. Example machine learning frameworks include PyTorch, TensorFlow, and others. Using the available NEQs identified in step 302, the user constructs a DNN topology by instantiating NEQs and connecting them together. The topology can be trained on a labeled dataset in a machine learning framework using backpropagation.

[0041] From a theoretical point of view, there is no limit to the number and connectivity of NEQs. However, in practice, each NEQ will be mapped back to an HBB, and the capacity and interconnect structure of the programmable devices should be taken into account. For example, Figure 5is a block diagram illustrating an example neural network with a sparse graph topology. The neural network includes LUT layers 502 and 504 and an output layer 508. A data set 501 is applied to LUT layer 502. Data set 501 is sparsely connected to LUT layer 502. LUT layer 502 is sparsely connected to LUT layer 504. LUT layer 504 is fully connected to output layer 508. Each LUT layer 502, 504 includes multiple LUT NEQs 510. Each LUT NEQ 510 includes an AND operator 512 and a nonlinear operator 514 (e.g., a greater than or equal to operator). In this example, each LUT layer is sparsely connected to the next layer, i.e., not every LUT input in level N+1 is directly connected to every LUT output in level N. This enables increasing the fan-in of output nodes without exponentially growing the width of the previous layers and is reminiscent of multi-level logic synthesis. The connection pattern between layers can be chosen randomly or based on a predetermined sparse pattern known to aid place-and-route algorithms. Additionally, in this scheme, intermediate outputs generated in the middle of the hierarchy can be shared to generate new / different neuron outputs.

[0042] In one example, a user can generate a hybrid topology by including non-NEQ building blocks such as standard convolutional layers, fully connected layers, pooling layers, etc. However, in this case, the techniques described herein only apply to the NEQ portion of the topology. That is, the design tool 235 only maps the NEQ in the topology to HBB. The remaining non-NEQ portions of the topology and the connectivity between the NEQ portions and the non-NEQ portions are mapped to hardware using traditional methods (e.g., by defining an overlay architecture and using mapping and scheduling tools). For example, a network can be constructed that starts with regular floating-point convolutional layers and ends with LUT layers, trained by backpropagation, and then the first portion is mapped to a fixed hardware architecture, while the second portion is directly converted to HBB (e.g., LUTs, BRAMs, etc.).

[0043] As long as the NEQ from step 302 is differentiable, the topology constructed by NEQ (and optionally non-NEQ traditional deep learning layers) can be trained in an end-to-end manner using the backpropagation algorithm. Since NEQ has quantized inputs and quantized outputs, the straight-through estimator (STE) technique can be used to propagate gradients through the quantized function, which has been proven to work well in practice for QNNs and BNNs. Since backpropagation is used for training, techniques for improving standard neural network training, such as knowledge distillation and fusion, can be applied.

[0044] Return to Figure 3At step 306, the design tool 235 uses a conversion process to convert the trained network of the NEQ into a netlist of the HBB. Once the topology has been trained and reaches the desired accuracy, the conversion process from step 302 is applied to each NEQ to convert it into its equivalent HBB. Using the trained NEQ parameters obtained in step 304, the conversion process will evaluate each NEQ on every possible input combination to obtain an output. Each (input->output) mapping corresponds to a row of the truth table that the HBB needs to implement. Because of the quantized nature of the NEQ inputs and outputs, the resulting HBB produces the same results as the NEQ for any input. For each NEQ, following this process generates a netlist of configured HBBs, where each HBB mimics the connectivity and functionality of the NEQ from which it was converted.

[0045] At step 308, the design tool 235 optionally performs any post-processing optimizations that may be applied to the netlist and generates an implementation of the target programmable device. For example, a heuristic logic minimizer may be applied to the netlist to use fewer LUTs; pipeline registers may be inserted between layers to increase clock frequency; or the netlist may be split into multiple blocks for mapping one block at a time to smaller programmable devices with dynamic partial reconfiguration. After any optional post-processing is completed, the final netlist is processed using a place-and-route algorithm to generate a bitfile for the programmable device. Notably, since the HBBs correspond to primitives in the target programmable fabric, logic synthesis and technology mapping are not required. The resulting design is highly parallel and identical to the network trained according to step 304. Additionally, unless the netlist is partitioned during optimization, the hardware implementation of the neural network can classify the inputs at the clock rate of the circuit.

[0046] Figure 6A is a block diagram illustrating a programmable device 54 according to an example. The programmable device 54 can be used to implement the programmable device 128 in the hardware accelerator 122. The programmable device 54 includes a plurality of programmable integrated circuits (ICs) 1, for example, programmable ICs 1A, 1B, 1C, and 1D. In one example, each programmable IC 1 is an IC die disposed on an interposer 51. Each programmable IC 1 includes a super logic region (SLR) 53 of the programmable device 54, for example, SLRs 53A, 53B, 53C, and 53D. The programmable ICs 1 are interconnected via conductors on the interposer 51, referred to as super long lines (SLLs) 52.

[0047] Figure 6Bis a block diagram illustrating a programmable IC 1 according to an example. The programmable IC 1 can be used to implement the programmable device 128 or one of the programmable ICs 1A to 1D in the programmable device 54. The programmable IC 1 includes programmable logic 3 (also referred to as programmable fabric), configuration logic 25, and configuration memory 26. The programmable IC 1 can be coupled to external circuits such as non-volatile memory 27, DRAM 28, and other circuits 29. The programmable logic 3 includes logic cells 30, support circuits 31, and programmable interconnects 32. The logic cells 30 include circuits that can be configured to implement general logic functions for multiple inputs. The support circuits 31 include specialized circuits such as transceivers, input / output blocks, digital signal processors, memory, and the like. The logic cells and support circuits 31 can be interconnected using the programmable interconnects 32. Information used to program the logic cells 30, to set parameters for the support circuits 31, and to program the programmable interconnects 32 is stored in the configuration memory 26 via the configuration logic 25. Configuration logic 25 may obtain configuration data from non-volatile memory 27 or any other source (e.g., DRAM 28 or from other circuits 29). In some examples, programmable IC 1 includes a processing system 2. Processing system 2 may include microprocessor(s), memory, support circuits, IO circuits, etc. In some examples, programmable IC 1 includes a network on chip (NOC) 55 and a data processing engine (DPE) array 56. NOC 55 is configured to provide communication between subsystems of programmable IC 1, such as between PS2, PL3, and DPE array 56. DPE array 56 may include a DPE array configured to perform data processing, such as a vector processor array.

[0048] Figure 6C is a block diagram depicting a SOC implementation of a programmable IC 1 according to an example. In this example, programmable IC 1 includes a processing system 2 and programmable logic 3. Processing system 2 includes various processing units, such as a real-time processing unit (RPU) 4, an application processing unit (APU) 5, a graphics processing unit (GPU) 6, a configuration and security unit (CSU) 12, a platform management unit (PMU) 122, and the like. Processing system 2 also includes various support circuits, such as on-chip memory (OCM) 14, a transceiver 7, peripherals 8, an interconnect 16, a DMA circuit 9, a memory controller 10, peripherals 15, and multiplexed IO (MIO) circuit 13. The processing units and support circuits are interconnected via interconnect 16. PL 3 is also coupled to interconnect 16. Transceiver 7 is coupled to external pin 24. PL 3 is coupled to external pin 23. Memory controller 10 is coupled to external pin 22. MIO 13 is coupled to external pin 20. PS 2 is generally coupled to external pin 21. The APU 5 may include a CPU 17 , memory 18 , and support circuits 19 .

[0049] exist Figure 6C In the example of FIG, programmable IC 1 can be used in hardware accelerator 122 and can function as described above. Acceleration circuit 130 can be programmed in PL 3 and function as described above. In another example, the functions of hardware 104 described above can be implemented using PS 2 rather than the hardware of the computing system. In this case, software 106 executes on PS 2 and functions as described above.

[0050] Referring to PS 2, each of the processing units includes one or more central processing units (CPUs) and associated circuitry, such as memory, an interrupt controller, a direct memory access (DMA) controller, a memory management unit (MMU), a floating point unit (FPU), etc. Interconnect 16 includes various switches, buses, communication links, etc., which are configured to interconnect the processing units and to interconnect other components in PS 2 to the processing units.

[0051] OCM 14 includes one or more RAM modules, which can be distributed throughout PS 2. For example, OCM 14 can include battery-backed RAM (BBRAM), tightly coupled memory (TCM), etc. Memory controller 10 can include a DRAM interface for accessing external DRAM. Peripheral devices 8 and 15 can include one or more components that provide an interface to PS 2. For example, peripheral device 15 can include a graphics processing unit (GPU), a display interface (e.g., DisplayPort, High-Definition Multimedia Interface (HDMI) port, etc.), a universal serial bus (USB) port, an Ethernet port, a universal asynchronous receiver / transmitter (UART) port, a serial peripheral interface (SPI) port, a general-purpose IO (GPIO) port, a serial advanced technology attachment (SATA) port, a PCIe port, etc. Peripheral device 15 can be coupled to MIO 13. Peripheral device 8 can be coupled to transceiver 7. Transceiver 7 can include serializer / deserializer (SERDES) circuitry, a multi-gigabit transceiver (MGT), etc.

[0052] Figure 6D A field programmable gate array (FPGA) implementation of a programmable IC1 comprising a PL 3 is illustrated. Figure 6DThe PL 3 shown can be used in any of the examples of programmable devices described herein. The PL 3 includes a number of different programmable blocks, including transceivers 37, configurable logic blocks ("CLBs") 33, random access memory blocks ("BRAMs") 34, input / output blocks ("IOBs") 36, configuration and clock logic ("CONFIG / CLOCKS") 42, digital signal processing blocks ("DSPs") 35, specialized input / output blocks ("I / Os") 41 (e.g., configuration ports and clock ports), and other programmable logic 39, such as digital clock managers, analog-to-digital converters, system monitoring logic, and the like. The PL 3 may also include a PCIe interface 40, an analog-to-digital converter (ADC) 38, and the like.

[0053] In some PLs, each programmable tile may include at least one programmable interconnect element ("INT") 43 having connections to input and output terminals 48 of programmable logic elements within the same tile, such as Figure 6D , as shown in the example included at the top of . Each programmable interconnect element 43 may also include a connection to an interconnect segment 49 of (multiple) adjacent programmable interconnect elements in the same tile or (multiple) other tiles. Each programmable interconnect element 43 may also include a connection to an interconnect segment 50 of a general routing resource between logic blocks (not shown). The general routing resources may include routing channels between logic blocks (not shown), which include tracks for interconnect segments (e.g., interconnect segments 50) and switch blocks (not shown) for connecting the interconnect segments. The interconnect segments (e.g., interconnect segments 50) of the general routing resources may span one or more logic blocks. The programmable interconnect elements 43, together with the general routing resources, implement a programmable interconnect structure ("programmable interconnect") for the illustrated PL.

[0054] In one example implementation, the CLB 33 may include a configurable logic element ("CLE") 44 that can be programmed to implement user logic plus a single programmable interconnect element ("INT") 43. In addition to one or more programmable interconnect elements, the BRAM 34 may also include a BRAM logic element ("BRL") 45. Typically, the number of interconnect elements included in a tile depends on the height of the tile. In the depicted example, the BRAM tile has the same height as five CLBs, but other numbers (e.g., four) may also be used. In addition to an appropriate number of programmable interconnect elements, the DSP tile 35 may also include a DSP logic element ("DSPL") 46. In addition to one instance of the programmable interconnect element 43, the IOB 36 may include, for example, two instances of an input / output logic element ("IOL") 47. It should be clear to those skilled in the art that the actual I / O pads, for example, connected to the I / O logic element 47, are generally not limited to the area of the input / output logic element 47.

[0055] In the depicted example, a horizontal region near the center of the die (e.g. Figure 3 D) is used for configuration, clock and other control logic. Extending from this horizontal area or column are vertical columns 51 that are used to distribute clock and configuration signals across the width of the PL.

[0056] use Figure 6D Some PLs of the illustrated architecture include additional logic blocks that disrupt the regular columnar structure that makes up most of the PL. The additional logic blocks may be programmable blocks and / or dedicated logic.

[0057] Notice, Figure 6D This is intended to illustrate only an exemplary PL architecture. For example, the number of logic blocks in a row, the relative widths of the rows, the number and order of the rows, the types of logic blocks contained in the rows, the relative sizes of the logic blocks, and Figure 6D The interconnect / logic implementation shown at the top is for illustrative purposes only. For example, in a real PL, any location where a CLB appears typically includes more than one adjacent CLB row to facilitate efficient implementation of user logic, but the number of adjacent CLB rows varies with the overall size of the PL.

[0058] Techniques for learning neural networks directly on programmable device blocks using backpropagation have been described. The new techniques described above involve the recognition that implementing a fixed architecture on top of LUTs and other programmable device building blocks creates optimization barriers for running neural networks on programmable devices. These techniques include mapping programmable device primitives to quantized neurons, including individual LUTs and RAM in different modes. The programmable device primitives can be exposed to a machine learning framework and trained using backpropagation. Multiple levels of binary neurons can be stacked and sparsely connected to increase indirect fan-in, trained using backpropagation and STE in a machine learning framework, and then mapped to a network of LUTs for placement on the programmable fabric of the target programmable device.

[0059] While the foregoing is directed to particular examples, other and further examples may be envisaged without the basic scope thereof, and the scope of which is to be determined by the claims that follow.

Claims

1. A method for generating a netlist for a programmable device on a computing system, comprising: defining a hardware building block HBB, a neuron equivalent NEQ, and a conversion process from NEQ to HBB, wherein HBB is a primitive available in the programmable fabric of the programmable device, wherein NEQ is a computational graph expressed in a machine learning framework, and each NEQ is mathematically equivalent to a single HBB and can be converted to the single HBB via the conversion process; defining a neural network using the NEQ in the machine learning framework; training the neural network on a training platform, the training platform comprising hardware; Converting the trained neural network into the netlist using the conversion process to convert each of the NEQs in the neural network into a single HBB in the HBBs of the netlist, the netlist being a netlist of HBBs; and The netlist suitable for programming the programmable device is generated.

2. The method according to claim 1, further comprising: The netlist is post-processed to perform one or more optimizations of the netlist.

3. The method of claim 1 or 2, wherein generating the netlist comprises processing the netlist using a place-and-route algorithm and generating a bit file for the programmable device.

4. The method of claim 1 , wherein the steps of defining the HBB, the NEQ, and the conversion process comprise: identifying primitives in the programmable structure of the programmable device to implement the HBB; For each HBB, defining one or more of the NEQs in the machine learning framework; and For each NEQ, identify the conversion process to the corresponding HBB.

5. The method of claim 1 , wherein each NEQ comprises: quantized inputs and quantized outputs, wherein the sum of the widths of the quantized inputs is less than or equal to the number of address bits of a memory component that implements the HBB to which the corresponding NEQ is converted, and wherein the sum of the widths of the quantized outputs is less than or equal to the width of the memory component.

6. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform a method of generating a netlist for a programmable device on a computing system, the method comprising: defining a hardware building block HBB, a neuron equivalent NEQ, and a conversion process from NEQ to HBB, wherein HBB is a primitive available in the programmable fabric of the programmable device, wherein NEQ is a computational graph expressed in a machine learning framework, and each NEQ is mathematically equivalent to a single HBB and can be converted to the single HBB via the conversion process; defining a neural network using the NEQ in the machine learning framework; training the neural network on a training platform, the training platform comprising hardware; and Converting the trained neural network into the netlist using the conversion process to convert each of the NEQs in the neural network into a single HBB in the HBBs of the netlist, the netlist being a netlist of HBBs; and The netlist suitable for programming the programmable device is generated.

7. The non-transitory computer-readable medium of claim 6, further comprising: The netlist is post-processed to perform one or more optimizations of the netlist.

8. The non-transitory computer-readable medium of claim 6 or 7, wherein generating the netlist comprises processing the netlist using a place-and-route algorithm and generating a bit file for the programmable device.

9. The non-transitory computer readable medium of claim 6, wherein the step of defining the HBB, the NEQ, and the conversion process comprises: identifying primitives in the programmable structure of the programmable device to implement the HBB; For each HBB, defining one or more of the NEQs in the machine learning framework; and For each NEQ, identify the conversion process to the corresponding HBB.

10. The non-transitory computer-readable medium of claim 6, wherein each NEQ comprises a dot product operation and a nonlinear operation, and wherein each NEQ comprises a quantized input and a quantized output, wherein a sum of widths of the quantized inputs is less than or equal to a number of address bits of a memory component that implements an HBB to which the corresponding NEQ is converted, and wherein a sum of widths of the quantized outputs is less than or equal to a width of the memory component.

11. A computing system comprising: a system memory configured to store code implementing a design tool; A training platform, comprising hardware; as well as A processor, coupled to the system memory and the training platform, is configured to execute the code to generate a netlist for a programmable device by: defining a hardware building block HBB, a neuron equivalent NEQ, and a conversion process from NEQ to HBB, wherein HBB is a primitive available in the programmable fabric of the programmable device, wherein NEQ is a computational graph expressed in a machine learning framework, and each NEQ is mathematically equivalent to a single HBB and can be converted to the single HBB via the conversion process; defining the neural network using the NEQ in a machine learning framework; training the neural network on the training platform; Converting the trained neural network into the netlist of HBB using the conversion process to convert the NEQ in the neural network into the HBB of the netlist; and The netlist suitable for programming the programmable device is generated.

12. The computing system of claim 11 , wherein the training of the neural network further comprises: The netlist is post-processed to perform one or more optimizations of the netlist.

13. The computing system of claim 11 or 12, wherein generating the netlist comprises processing the netlist using a place-and-route algorithm and generating a bit file for the programmable device.

14. The computing system of claim 11 , wherein defining the HBB, the NEQ, and the conversion process comprises: identifying primitives in the programmable structure of the programmable device to implement the HBB; For each HBB, defining one or more of the NEQs in the machine learning framework; and For each NEQ, identify the conversion process to the corresponding HBB.

15. The computing system of claim 11 , wherein each NEQ comprises a dot product operation and a nonlinear operation, and wherein each NEQ comprises a quantized input and a quantized output, wherein a sum of widths of the quantized inputs is less than or equal to a number of address bits of a memory component that implements an HBB to which the corresponding NEQ is converted, and wherein a sum of widths of the quantized outputs is less than or equal to a width of the memory component.

Images