Multi-die dot product engines for large-scale machine learning inference applications

Abstract

Systems and methods are provided for providing a multi-die dot product engine (DPE) for providing large-scale machine learning inference applications. The multi-die DPE utilizes a multi-chip architecture. For example, a multi-chip interface can include a plurality of DPE chips, where each DPE chip performs inference computations for performing deep learning operations. During inference operations, a hardware interface between a memory of a host computer and the plurality of DPE chips communicably connects the plurality of DPE chips to the memory of the host computer system such that the deep learning operations span the plurality of DPE chips. Due to the multi-die architecture, inference is allowed using multiple silicon devices, enabling high energy-efficient inference for large-scale machine learning applications and complex deep neural networks. The multi-die DPE can be used to build a multi-device DNN inference system that can perform specific applications, such as object recognition, with high accuracy.

Description

Background Technology

[0001] Deep learning is a method based on the broader concepts of artificial intelligence and machine learning (ML). Deep learning can be described as mimicking the operation of biological systems, such as the human brain, in learning information and recognizing patterns for decision-making. Deep learning typically involves artificial neural networks (ANNs), which are capable of supervised learning from labeled data or unsupervised learning from unstructured or unlabeled data. In examples of deep learning, computer models can learn to perform classification tasks directly from images, text, or sound. With advancements in AI technology, deep learning models—such as those trained using massive amounts of data and multi-layered neural network architectures—can achieve state-of-the-art accuracy, sometimes even surpassing human-level performance. Due to this performance increase, deep learning can have a wide range of practical applications, including function approximation, classification, data processing, image processing, robotics, autonomous vehicles, and computer numerical control. Attached Figure Description

[0002] This disclosure is described in detail with reference to the following accompanying drawings, according to one or more various embodiments. These drawings are provided for illustrative purposes only, and depict only typical or exemplary embodiments.

[0003] Figure 1 Examples of deep learning accelerator systems, including a Deep Learning Accelerator System Interface (DLASI) that connects multiple inference computing units to main memory, are described according to some embodiments.

[0004] Figure 2 An example architecture for a multi-chip interface system, including a multi-die dot product engine (DPE) for providing large-scale machine learning inference applications, is shown according to some embodiments.

[0005] Figure 3 It is shown that, according to some embodiments, it is possible to Figure 2 An example architecture of DPE application-specific integrated circuits (ASICs) implemented in a multi-chip interface system.

[0006] Figure 4A The use of some embodiments is described Figure 2 An example of an object recognition application in a multi-chip interface system.

[0007] Figure 4B Examples of tile-level pipeline schemes according to some embodiments are shown, which allow Figure 2 The multi-chip interface system coordinates memory access for image, inference, and result output.

[0008] Figure 5Depicting according to some embodiments Figure 2 An example of a simulation setup for a multi-chip interface system based on a field-programmable gate array (FPGA).

[0009] Figure 6 The illustration shows that, according to some embodiments, it may include Figure 2 The example computer system shown is a multi-chip interface system.

[0010] These accompanying drawings are not exhaustive and do not limit this disclosure to the precise form disclosed. Detailed Implementation

[0011] The various embodiments described herein relate to multi-die dot product engines that provide large-scale machine learning inference applications. Current dot product engine (DPE) techniques for implementing deep learning hardware accelerators and memristor-based Programmable Ultra-Efficient Accelerator (PUMA) deep neural network (DNN) inference architectures are highly energy-efficient. However, these techniques utilize on-die weight storage methods, which are crucial for these architectures. This on-die weight storage method can limit the size of the DNN, especially when the hardware is implemented on a single silicon device. To address this drawback, the disclosed multi-die dot product engine is explicitly designed to utilize a multi-chip architecture for DPEs. Due to the multi-die architecture, multiple silicon devices can be implemented for inference, enabling energy-efficient inference for large-scale ML applications and complex DNNs. As technology continues to advance, the size and complexity of DNNs may also increase to accommodate a wide range of applications in growing markets such as autonomous driving, natural language processing, cybersecurity, or any number of other applications.

[0012] As will be described herein, the disclosed multi-die dot product engine is part of a multi-device DNN inference system capable of performing object recognition with high accuracy. Furthermore, according to the embodiments described, the system may include an enhanced system interface compliant with scalable IOV, allowing the hyperconverged server to deploy expansion cards as needed. Moreover, the boxes used in the disclosed example systems are designed for integration into application-specific integrated circuits (ASICs).

[0013] As mentioned above, DPE technology can be used to implement deep learning accelerator systems. For example, this paper describes a deep learning accelerator system including a Deep Learning Accelerator System Interface (DLASI). The DLASI is designed to provide a high-bandwidth, low-latency interface between the core (e.g., for inference) and the server, where the core and server might otherwise lack communication compatibility (relative to memory). Designing an accelerator consisting of thousands of small cores can present challenges, such as: coordinating the many cores, maintaining high efficiency of the accelerator despite the vastly different problem sizes associated with DNN inference, and performing these DNN-based computational tasks without consuming excessive power or die area. Generally, coordinating thousands of neural network inference cores can strain a single host interface controller, potentially making it a performance bottleneck.

[0014] Furthermore, the size of different neural networks can vary greatly. Some neural networks may have only a few thousand weights, while others, such as those used for natural language processing or image recognition, may have over 100 million weights. Using a large accelerator for each application seems like a feasible brute-force solution. On the other hand, if a large accelerator is allocated to work on a small neural network, the accelerator may not be fully utilized. Moreover, modern servers host many operating systems and can only accommodate a few expansion cards. For example, each processor socket in a server may have multiple PCIe card slots. Mapping large neural networks onto a single die presents many challenges because there may not be enough on-die storage to accommodate each weight. Therefore, this paper describes a multi-chip interface solution including a multi-die dot product engine (DPE) for providing large-scale machine learning inference applications. As another example, PCIe switches can be used on PCIe form factor cards, with multiple dies interacting with them to map large models. Therefore, even with only a limited number of PCIe lanes, large models can still be mapped onto a single PCIe card by utilizing multiple DPE dies.

[0015] Typically, commodity servers (e.g., Xeon-based), personal computers (PCs), and embedded systems (e.g., Raspberry Pi) run standardized operating systems and contain complex general-purpose CPUs and cacheable memory systems. However, deep learning processors can achieve high performance with simpler instruction sets and memory architectures. Furthermore, the core architecture is optimized for processing smaller numbers, such as handling 8-bit numbers in operation (instead of 32-bit or 64-bit) for neural network inference. Hardware designs for deep learning accelerators can include a large number of processors, such as thousands. Moreover, with the use of these thousands, these deep learning processors may often not require high precision. Therefore, processing small numbers may be optimal for their multi-core design, for example, to alleviate bottlenecks. Conversely, commodity servers can operate very efficiently to process larger numbers, such as 64-bit. Due to these (and other) functional differences, some inconsistencies can arise between the accelerator cores and the server during deep learning processing. As described above, the disclosed DLASI is designed to address such issues. The DLASI implements a multi-die solution that effectively connects different types of processing (processing executed on the accelerator core and host server) to connect entities in the acceleration system, thereby improving compatibility and enhancing the overall performance of the system.

[0016] According to some embodiments, the DLASI includes a fabric protocol, a processor-based host interface, and a bridge that connects the server memory system (treating memory as an array of 64-byte (B) cache lines) to a large number of DNN inference computation units, treating memory as an array of 16-bit words. The fabric protocol can be two virtual channel (VC) protocols, enabling the construction of simple yet efficient switches. The fabric protocol can support large data packets, which in turn support high efficiency. Furthermore, the fabric protocol can be extended to multiple chips through simple sorting rules. In some cases, the fabric protocol can even be layered on top of another protocol (e.g., Ethernet) for server-to-server communication. Additionally, in some examples, the host interface can connect to the server at the "image" level (or input data sequence level) and can "push" smaller workpiece pipelines from larger levels to the multiple cores. This is achieved by applying a synchronization scheme, referred to herein as overlapping interval pipelining. Overlapping interval pipelines can generally be described as a connection of send instructions and barrier instructions. This pipeline approach enables each inference computing unit (e.g., block) to be built using a small amount of on-die memory (e.g., 7nm ASIC, 1 card, 2 dies, 128 blocks) and synchronizes work between blocks in a way that minimizes block idle time (thus optimizing processing speed).

[0017] Figure 1 This is an example block diagram of a deep learning accelerator 100 including a DLASI 105. The deep learning accelerator 100 can be implemented as hardware, such as a field-programmable gate array (FPGA) or other form of integrated circuit (IC) chip (e.g., application-specific integrated circuit - ASIC). As an FPGA, the accelerator 100 can include digital math units (instead of memristor-based analog computing circuitry). The deep learning accelerator 100 can have an architecture that allows various deep learning applications to run on the same silicon chip. Figure 1 As shown, the DLASI (represented by the dashed box) can be a conceptual collection of several components, including: DLI architecture protocol link 108; host interface 121; bridge 111; and switch 107. The deep learning accelerator 100 has an architecture divided into four domains, including: CODI-deep learning inference domain 110; CODI-simple domain 120; AMBA4-AXI domain 130; and peripheral component interconnect fast (PCIe) domain 140. Furthermore, Figure 1This is intended to illustrate that the DLASI 105 can be implemented as an on-die interconnect, thereby allowing the disclosed interface to be a fully integrated on-chip solution (relative to the accelerator chip).

[0018] PCIe domain 140 is shown as including communication connections between server processors 141. PCIe domain 140 may include the PCIe interface 131 as a high-speed interface connecting the DLI inference chip to a host processor, such as a server processor. For example, the server's motherboard may have multiple PCIe slots for receiving add-on cards. The server processor 141 may be implemented in a commodity server communicating with blocks 106a-106n for performing deep learning operations, such as image recognition. The server processor 141 may be implemented as a rack server or blade server, such as a Xeon server, ProLiant server, etc. As described above, by supporting multi-card configurations, the accelerator 100 can support larger DNNs. For a small number of FPGAs or ASICs (e.g., four FPGAs), it is possible to use a PCIe peer-to-peer mechanism. In some cases, the PCIe link may not be able to deliver sufficient bandwidth, thus a dedicated FPGA-to-FPGA (or ASIC-to-ASIC) link may be used.

[0019] In the example shown, the CODI-DLI domain 110 includes a set of tiles 105, multiple blocks 106a-106n, a switch 107, and a bridge 111. As can be seen, the set of tiles 105 consists of multiple blocks 106a-106n that are communicatively interconnected. Each block 106a-106n is configured as a DNN inference computation unit, capable of performing deep learning-related tasks such as computation, inference processing, etc. Therefore, the set of tiles 105 can be considered as an on-chip block network 106a-106n, also referred to herein as a DLI structure. The CODI-DLI domain 110 includes CODI interconnects for connecting the blocks to each other and connecting the blocks to the host interface controller 121.

[0020] Each individual block 106a-106n may further include multiple cores (not shown). For example, a single block 106a may include 16 cores. Furthermore, each core may include a matrix-vector multiplication unit (MVMU). These MVMUs can be implemented using a cross-bar of resistive memory (e.g., memristors) for in-memory analog computation, or using digital circuitry including static random access memory (SRAM) for storing neural network weights and digital multipliers / adders (relative to memristors) for computation. In one embodiment, the core may implement a full set of instructions and employ four 256×256 MVMUs. The cores in the block are connected to block memory. Thus, for example, the block memory for block 106a can be accessed from any core residing in block 106a. Blocks 106a-106n in the block set 105 can communicate with each other by sending datagram packets to other blocks. The block memory has a unique feature for managing flow control—each element in the block memory has a count field that is decremented by reading and set by writing. Furthermore, each of blocks 106a-106n may have a bare-chip architecture interface (not shown) for communicating with the other blocks and the switch 107. The switch 107 can provide block-to-block communication.

[0021] Therefore, on-die interconnects exist, allowing the inference chip to connect to the PCIe domain 140. The CODI-deep learning inference domain 110 is a unique structure that connects many computing units to each other.

[0022] The Deep Learning Inference (DLI) architecture protocol link 108 is configured to provide a communication connection according to the DLI architecture protocol. The DLI architecture protocol can use low-level conventions, such as those proposed by CODI. The DLI architecture protocol can be a 2-Virtual Channel (VC) protocol, which enables the construction of simple and efficient switches. The switch 107 can be a 16-port switch, serving as the basis for the design. The DLI architecture protocol can be implemented as a 2-VC protocol by designing higher-level protocols in a way that minimizes fabrication. The DLI architecture protocol supports a large identifier (ID) space, such as 16 bits, thereby supporting multiple chips that can be controlled by the host interface 121. Furthermore, the DLI architecture protocol can use simple sorting rules, allowing the protocol to be extended to multiple chips.

[0023] The DLASI 105 also includes a bridge 111. In general, the bridge 111 can be an interface that acquires data packets from one physical interface and transparently routes them to another physical interface to facilitate connectivity between them. The bridge 111 is shown as an interface between the host interface 121 in the CODI-Simple Domain 120 and the switch 107 in the CODI-Deep Learning Inference Domain 110, bridging the domains for communication. The bridge 111 can ultimately connect server memory (treating memory as an array of 64B cache lines) to the DLI structure, namely blocks 106a-106n (treating memory as an array of 16-bit words). In an embodiment, the bridge 111 has hardware functions for distributing input data to the blocks 106a-106n, collecting output and performance monitoring data, and switching from processing one image (or a sequence of input data) to processing the next image.

[0024] The host interface 121 provides input data and transmits output data to the host server memory. To implement simple flow control, the host interface can declare when the next interval will occur and be notified when all PUMA cores of a block have reached the halt instruction. When the host interface declares the start of the next interval, each block sends its intermediate data to the next block set to execute the computation for the next interval.

[0025] In one example, when the PCIe card boots, the links in the PCIe domain 140 are trained. For instance, the links in the PCIe domain 140 can complete training, the clock starts timing, and the blocks are retrieved from reset. The blocks in the card can then be initialized. Then, when the DNN is loaded onto the card, matrix weights, core instructions, and / or block instructions can be loaded.

[0026] Now refer to Figure 2This paper describes an example architecture of a multi-chip interface system 200 comprising multiple DPE dies 250a-250l. As background, machine learning (ML) has become an indispensable part of modern life, automating various complex decision-making processes. Due to the data-intensive and computationally intensive nature of ML algorithms (Matrix-Vector Multiplication (MVM), conventional CMOS hardware may offer limited performance at scale. Therefore, employing resistive memory technology (e.g., memristors) to perform computations in parallel (by storing the weights of the memristors) is a viable option. However, as the complexity of ML algorithms increases, the computational intensity and size of the models also continue to grow. The size of neural network models varies considerably depending on the use case and the minimum required accuracy. For example, small, deeply separable models (e.g., Mobile Net-SSD) with fewer than 7 million synaptic weight parameters are sufficient for medium-accuracy pedestrian and vehicle object detection in advanced driver assistance systems (ADAS). Larger models with over 50 million weights (such as YOLO v3 or R-FCN w / ResNet-101) can be used for higher-precision object detection to achieve fully autonomous mobility, medical image analysis, etc. Very large models with 200-400 million weights can be used for natural language machine translation.

[0027] While it is theoretically possible to build an accelerator ASIC with 400 million weights in memristor crossbars on a single silicon die (i.e., 3.2 / 6.4 Gbit at 8-bit / 16-bit weight resolution), the die size would rival some of today's largest chips (e.g., the NVIDIA V100 GPU @ 815 mm²), resulting in low yield, high cost, large footprint, and limited support for only a single copy of the largest model on the die, restricting data and model parallelism and leading to poor performance.

[0028] To overcome these aforementioned challenges, the disclosed multi-chip interface system 200 extends the DPE architecture to support neural network models across multiple silicon DPE dies 250a-250l (also referred to herein as DPE ASICs). The system 200 enables a modular architecture to be adapted to a wide range of markets with a much smaller chip size, supporting a large number of weights at low cost (e.g., using future ASICs supporting up to approximately 100 million weights). In other words, the number of chips used in a design can be tailored to best suit the complexity, constraints, and performance requirements of a specific DNN-based application. For example, a single chip can be used in the embedded market (ADAS, etc.), two chips (50-75W) on a PCIe half-width / half-height form factor accelerator board can be used in high-performance edge systems (including EL-1000 / EL-4000 servers or workstations), and four or more chips on a standard-sized PCIe card or a dedicated accelerator tray can be used for data center applications.

[0029] The multi-chip interface system 200 has a scalable architecture that can support larger models. Furthermore, the system 200 provides a low-power and cost-effective method while supporting a large number of DPE ASICs 250a-250l. In the example shown, the system 200 is shown to include at least twelve DPE ASICs 250a-250l. Moreover, the system 200 utilizes a switch expansion, namely a PCIe switch 260, to support communication links between multiple DPE ASICs 250a-250l. Figure 2 As shown, multiple DPE ASICs can be linked to a single DPE ASIC, which in turn is linked to the PCIe switch 260. For example, DPE ASICs 250d, 250h, and 250l are linked to the PCIe switch 260. DPE ASIC 250d has multiple chips linked to it, namely DPE ASICs 250a, 250b, and 250c. DPE 250h is linked to DPE ASICs 205e, 250f, and 250g. Additionally, DPE 250l is linked to DPE ASICs 205i, 250j, and 250k. Therefore, each of DPE ASICs 250a-250l is communicatively linked to the server processor 241 facing the PCIe switch 260. The off-die connections (indicated by arrows) between each DPEASIC 250a-250l, the PCIe switch 260, and the server processor 241 can be implemented as structural links. In one embodiment, the link is a PCIe Gen3 link.

[0030] Alternatively, a direct link (not shown) between the plurality of DPE ASICs 250a-250l could be implemented as a simple and fast method. However, due to the scale inefficiency of using direct links relative to the cost of greater connectivity, the use of the PCIe switch 260 may be desirable.

[0031] Furthermore, to support off-die interconnects to other DPE ASICs in a multi-chip interconnect system, the DPE ASIC 2501 may include multiple PCIe root port interfaces 252a-252c. The PCIe root port interface (e.g., 252a) can serve as an interface between the DPE ASIC 2501 and another DPE ASIC having a direct interconnect (or structural link) to the DPE ASIC 2501. In some embodiments, each PCIe root port interface 252a-252c corresponds to a link connecting the DPE ASIC 2501 to the corresponding DPE ASIC. Figure 2 In the example, PCIe root port interface 252a is connected to a fabric link to create a chip-to-chip interconnect from DPE ASIC 250l to DPE ASIC 250i. PCIe root port interface 252b is connected to a different fabric link to create a chip-to-chip interconnect from DPE ASIC 250l to DPE ASIC 250j. PCIe root port interface 252c is connected to another fabric link to create a chip-to-chip interconnect from DPE ASIC 250l to DPE ASIC 250k.

[0032] The DPE ASIC 250l also includes an on-die AMBA-AXI switch 207. The AMBA-AXI switch 207 provides chip-to-chip communication between the DPE ASIC 250l and other DPE ASICs directly linked to it. This is in contrast to DLASI-based systems (such as...). Figure 1 Compared to (as shown), the AMBA-AXI switch 207 supports chip-to-chip communication in a manner similar to block-to-block communication supported by its on-chip switch.

[0033] Turning Figure 3 The DPE ASIC 250l may further include a target PCIe interface 231. The target PCIe interface 231 may be for connecting the DPE ASIC 250l to a host processor (e.g., a server processor). Figure 2The high-speed interface shown is used. The DPE ASIC 2501 can be controlled by the host interface 221. Furthermore, the host interface 221 may include a core 222 for processing power. In the example shown, the core 222 is implemented as a RISC-V core.

[0034] Return to the referenced DLASI-based system (e.g.) Figure 1 As shown, the system is a layered design that implements a multi-block communication protocol specifically designed for deep learning inference accelerator cores, along with area-saving on-die interconnects. However, this DLASI-based system relies on block-to-block sideband wires to indicate when a block is ready to receive new data. Modern VLSI designs with fault-tolerant architectures cannot support sideband signals across multiple chips. Therefore, the deep learning inference architecture is extended to have two virtual channels: one for high-priority synchronization packets and the other for data communication packets.

[0035] exist Figure 3 In the image, the multi-chip system 200 is highlighted. Figure 2 A single DPE ASIC 2051 (shown) is illustrated to show the internal components of the ASIC architecture. Specifically, due to the multi-chip design of the system, the DPE ASIC 2051 includes a multi-chip support block 251, which is not used in the previously described DLASI-based systems. The multi-chip support block 251 is configured to bridge the requirements of off-die interconnects to on-die interconnects. Compared to many low-bandwidth on-die links, the off-die interconnects (i.e., PCIe links) of the DPE ASIC have less bandwidth but higher bandwidth. To fully utilize the off-die links, the multi-chip support block 251 is configured to perform buffering on-die packets. To simplify flow control and ensure compatibility with most commodity architectures, flow control in the multi-chip interface system is credit-based. Therefore, the system can efficiently utilize its chip-to-chip interconnects without introducing overly complex packet arbitration. Furthermore, the architecture protocol and instruction set support up to 4096 blocks. In one implementation, multiple chip blocks can support up to 64 blocks, thus allowing scaling to accommodate a full range of popular DNN sizes.

[0036] The multi-chip extension does not complicate higher-level software frameworks. For example, when the DPE compiler maps a model to the interconnect block set 205 on the DPE ASIC 2501, it can use logical identifiers regardless of the hardware topology. This allows for a seamless mapping from the model definition provided by the high-level framework to the DPE hardware. Once the DPE compiler generates an assembly model with logical blocks, the DPE scheduler software provides translations for the physical blocks, such as the interconnect block set 205, which are identified by a combination of chip-identifiers, clump-identifiers, and block-identifiers. Ultimately, the DPE loader can translate the assembly code based on the provided on-the-fly translations as it loads the model into the hardware.

[0037] Now for reference Figure 4A This illustrates execution by computer system 480 and utilization of multi-chip interface system 455 (also shown in...). Figure 2 An example of an object recognition application (shown in the image). The object recognition application 450 can receive an initial image 452 (in the image). Figure 4A Image 452 (referred to as "Preprocessed Image X") refers to image frames, such as those streaming to a host computer in a video format (e.g., 1MB). Therefore, as referred to herein, image 452 refers to preprocessing (e.g., prior to object recognition analysis). Image 452 is then sent for analysis by the multi-chip interface system 455 using DNN inference techniques. This example specifically relates to an implementation based on "You Only LookOnce (yolo)-tiny," a DNN that can be used in video object recognition applications. According to this example, Yolo-tiny can be mapped onto the multi-chip interface system 455. For example, the multi-chip interface system 455 can be implemented in hardware as multiple FPGA chips capable of performing object recognition on the video stream using the Yolo-Tiny framework as a real-time object detection system.

[0038] The OS interface 453 on the host can send a request to analyze data in the work queue 454. Next, a doorbell 459 can be sent as an indication of the request and transmitted to the host interface of the multi-chip interface system 455. As used herein, the term "doorbell" refers to a common signal in interface communication protocols. The doorbell 459 is a signal used to notify the host controller that input data is available for processing in the host system work queue 454. When work related to image analysis is placed in the work queue 454 by the OS interface 453 and the doorbell 459 rings, the host interface can retrieve image data from the queue 454. Furthermore, when analysis results are obtained from the multi-chip interface system 455, the generated objects can be placed in the completion queue 456 and then transferred to the server main memory. The host interface can read the request and then "feed" the image using a bridge and a block (and instructions running therein), which analyzes the image data for object recognition. According to an embodiment, the DLI architecture protocol is a mechanism that allows this "feeding" of the final completion to the block. In other words, the previously described structural protocols and other multi-chip system components link the protocol domain to the hardware domain.

[0039] The result of the object recognition application 450 can be a bounding box and probability associated with the recognized object. Figure 4A A post-processed image 460 (in) is depicted that may be generated due to running the object recognition application 450. Figure 4A (Displayed as "Post-processed Image X"). Therefore, image 452 represents the image before processing by the object recognition application, and image 460 represents the image subsequently processed by the object recognition application, including various recognized objects. Three bounding boxes exist around the objects in image 460, which have been identified as visual representations of "people," each with an association probability displayed as "63%", "61%", and "50%". In image 160, there are also objects identified as "keyboard" with a probability of "50%" and as "TV monitor" with a probability of "51%".

[0040] Figure 4B An example of a block-level pipeline is shown, which allows different images to be processed simultaneously. In detail, Figure 4BA multi-chip interface system for coordinating DMA (Direct Memory Access) transfers of images, inferences, and results is illustrated. As background, computationally, typical DNN algorithms primarily consist of a combination of matrix-vector multiplication and vector operations. DNN layers use nonlinear computation to break input symmetry and achieve linear separability. The core is programmable and can execute instructions to implement the DNN, where each DNN layer is fundamentally expressible in terms of instructions for performing low-level computations. Thus, multiple layers of a DNN are typically mapped to multiple blocks of multiple chips in the multi-chip system for computation execution. Furthermore, in Figure 4B In the example, the layers of the DNN used for image processing are also mapped to blocks 474a-474e of the multi-chip interface system. In this example, the system includes multiple chips, namely chip 0, chip 1, and chip 2, and multiple blocks can be implemented on the respective chips. For example, chip 0 includes blocks 0 474a and 1 474b; chip 1 includes blocks 0 474c and 1 474d; and chip 2 includes block 0 474e. It should be understood that in Figure 4B The partitioning of blocks between chips shown is not intended to be limiting, but rather serves as an example. This partitioning can vary depending on the specific implementation, such as the size of the neural network model.

[0041] As can be seen, at server memory level 471, images 0 472a, 1 472b, and 2 472c are piped as input to be received by multiple blocks 474a-474e (on corresponding chips). In other words, image data may not be sent simultaneously. Instead, the pipelined scheme disclosed herein can stagger the transmission and processing of image data segments (such as images 0 472a, 1 472b, and 2 472c shown). Images 472a-472c are received at host interface level 473 before being received by blocks 474a-474e. Host interface level 473 first transmits image 0 472a to blocks 474a-474e. In this example, the inference work performed by blocks 474a-474e is as follows: Block 0 in chip 0 474a and Block 1 in chip 0 474b are used for the first layer computation of the DNN layer mapping for image 0 472a; Block 0 in chip 1 474c and Block 1 in chip 1 474d are used for the intermediate layer computation of the DNN layer mapping for image 0 472a; and Block 0 in chip 2 474e is used for the final layer computation of the DNN layer mapping for image 0 472a. Then, as the pipeline progresses, after the computation of the final layer is completed, the object detection for image 0 475a can be output to the host interface level 473. In the next interval of the pipeline, the object detection for image 0 475a can be transferred to the server memory 471. Furthermore, according to the pipeline scheme, while the object detection for image 0 475a is being sent to the server memory 475a, the object detection for image 1 475b can be transferred to the host interface level 473.

[0042] More iterations can be performed in the early stages of a Convolutional Neural Network (CNN) than in the later stages of CNN inference. Therefore, in some embodiments, additional resources (blocks or cores) can be allocated to these more iterative stages. Overall, image recognition performance can be determined by the pipeline speed, and in some examples, the pipeline speed can be set by the block that takes the longest time to complete its work. Before each pipeline interval begins, the DNN interface sets the input data and captures the output data.

[0043] Now refer to Figure 5 An example of an FPGA-based multi-chip simulation setup 500 is described, which simulates the multi-chip interface system ( Figure 2The hardware (shown) is used. The FPGA emulator platform can run a large number of commonly used neural network models on the DPE FPGA software development framework. Furthermore, neural networks are considered a first step towards future composability and virtualization, enabling software-defined partitioning of many single / multi-tenant inference workloads across accelerator pools without concern for specific physical hardware dependencies. For example... Figure 5 To replicate the multi-chip setup 500, off-the-shelf Xilinx Alveo FPGA cards can be utilized. Similarly, an extended PCIe interface can be used in the setup 500, which leverages PCIe point-to-point capabilities for host communication for chip-to-chip communication. To enable the interface to scale independently of system-level constraints (e.g., lack of PCIe lanes), a chip-to-chip (C2C) connection between the two FPGA cards is enabled, which extends the multi-chip emulation setup 400 to multiple machines and allows for the mapping of larger models.

[0044] Figure 6 A block diagram of an example computer system 600 is depicted, in which a multi-chip interface system as described herein can be implemented. Figure 2 (As shown). The computer system 600 includes a bus 602 or other communication mechanism for transmitting information, and one or more hardware processors 604 coupled to the bus 602 for processing information. The hardware processor 604 may be, for example, one or more general-purpose microprocessors.

[0045] The computer system 600 also includes a main memory 606, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to a bus 602, for storing information and instructions to be executed by the processor 604. The main memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 604. When these instructions are stored in storage media accessible to the processor 604, the computer system 600 becomes a dedicated machine customized to perform the operations specified in the instructions.

[0046] The computer system 600 further includes a storage device 610, such as a read-only memory (ROM) or other static storage device, coupled to a bus 602 for storing static information and instructions of the processor 604. A storage device 610, such as a disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to the bus 602 for storing information and instructions.

[0047] The computer system 500 can be coupled to a display 612, such as a liquid crystal display (LCD) (or touchscreen), via a bus 602 for displaying information to a computer user. Input devices 614 (including alphanumeric keys and other keys) are coupled to the bus 602 for transmitting information and command selections to the processor 604. Another type of user input device is a cursor control device 616, such as a mouse, trackball, or arrow keys, for transmitting directional information and command selections to the processor 604 and for controlling cursor movement on the display 612. In some embodiments, the same directional information and command selections as with cursor control devices can be implemented by receiving touches on a touchscreen in the absence of a cursor.

[0048] The computing system 600 may include a user interface module for implementing a GUI, which may be stored as executable software code executed by the computing device in a mass storage device. This module and other modules may include, for example, components (e.g., software components, object-oriented software components, class components, and task components), processes, functions, attributes, programs, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0049] Generally, terms such as “component,” “engine,” “system,” “database,” and “data storage” as used herein can refer to logic embodied in hardware or firmware, or to a collection of software instructions written in programming languages ​​such as Java, C, or C++, which may have entry and exit points. Software components can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as BASIC, Perl, or Python. It should be understood that software components can be invoked from other components or from themselves, and / or can be invoked in response to detected events or interrupts. Software components configured to execute on a computing device may be provided on computer-readable media (e.g., optical discs, digital video discs, flash drives, disks, or any other tangible media) or in the form of digital downloads (and may be initially stored in a compressed or installable format, requiring installation, decompression, or decryption before execution). Such software code may be stored, in part or in whole, on a storage device executing the computing device. Software instructions may be embedded in firmware such as EPROM. It should also be understood that hardware components may include connected logic units, such as gates and flip-flops, and / or may include programmable units, such as programmable gate arrays or processors.

[0050] The computer system 600 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, in conjunction with the computer system, enable the computer system 600 or program it as a special-purpose machine. According to one embodiment, the techniques described herein are executed by the computer system 600 in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. These instructions may be read into main memory 606 from another storage medium, such as storage device 610. Execution of the sequence of instructions contained in main memory 606 causes processor 604 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0051] As used herein, the circuit can be implemented using any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be implemented to constitute the circuit. In one embodiment, the various circuits described herein can be implemented as discrete circuits, or the described functions and features can be shared partially or wholly among one or more circuits. Even if various features or functional elements can be described or claimed as separate circuits, these features and functions can be shared among one or more general-purpose circuits, and such description will not require or imply the need for separate circuits to implement such features or functions. When the circuit is implemented wholly or partially using software, the software can be implemented to operate in conjunction with a computing or processing system (e.g., computer system 500) capable of performing functions relative to those described herein.

[0052] As used herein, the term “or” can be interpreted in an inclusive or exclusive sense. Furthermore, descriptions of the singular form of a resource, operation, or structure should not be construed as excluding the plural. Unless otherwise explicitly stated or otherwise understood in the context in which they are used, conditional language (e.g., “can,” “able,” “may,” or “may,” etc.) is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not.

[0053] Unless otherwise expressly stated, the terms and phrases used in this document, and their variations thereof, should be interpreted as open-ended rather than restrictive. Adjectives and similar terms such as “regular,” “traditional,” “normal,” “standard,” and “known” should not be interpreted as limiting the described items to a given time period or items available within a given timeframe, but should be understood to include regular, traditional, normal, or standard techniques that are available or known now or in the future. In some cases, the presence of broad terms and phrases such as “one or more,” “at least,” “but not limited to,” or other similar phrases should not be interpreted as indicating or requiring the use of a narrower definition where such a broad phrase might not exist.

Claims

1. A multi-chip interface system, comprising: Multiple dot product engine (DPE) chips, wherein each of the multiple DPE chips performs inference computation for performing deep learning operations; and A hardware interface is located between the host computer's memory and the plurality of DPE chips, wherein the hardware interface communicatively connects the plurality of DPE chips to the host computer's memory during inference operations, such that the deep learning operations span the plurality of DPE chips.

2. The multi-chip interface system of claim 1, wherein enabling the deep learning operation across the plurality of DPE chips comprises: This enables the neural network model to span the multiple DPE chips.

3. The multi-chip interface system according to claim 1, wherein each of the plurality of DPE chips comprises an application-specific integrated circuit (ASIC).

4. The multi-chip interface system according to claim 1, wherein each of the plurality of DPE chips comprises: Multiple port interfaces, wherein each of the multiple port interfaces facilitates an off-chip communication link to another DPE chip among the multiple DPE chips, thereby enabling chip-to-chip communication between the multiple DPE chips; and Multiple chip uplinks connect one or more on-chip components of the DPE chip.

5. The multi-chip interface system according to claim 4, wherein each of the plurality of DPE chips comprises: A multi-chip support block, wherein the multi-chip support block bridges one or more on-chip components of the DPE chip to another DPE chip among the plurality of DPE chips, and facilitates communication between the plurality of on-chip links and the off-chip communication links to another DPE chip among the plurality of DPE chips.

6. The multi-chip interface system according to claim 5, wherein the multi-chip support block buffers chip-to-chip data packets.

7. The multi-chip interface system according to claim 5, wherein each of the plurality of DPE chips comprises: A switch connected to the plurality of port interfaces, wherein the switch facilitates chip-to-chip communication between the DPE chip and another DPE chip from the plurality of DPE chips via the off-chip communication link.

8. The multi-chip interface system according to claim 1, wherein each of the plurality of DPE chips comprises: An interconnection block set, which includes multiple inference computing units.

9. The multi-chip interface system according to claim 1, wherein each of the plurality of DPE chips comprises: A target interface, which is connected to the hardware interface, facilitates communication between the DPE chip and the host computer's memory via the hardware interface.

10. The multi-chip interface system according to claim 1, wherein the inference operation includes at least one of the following: image recognition application, natural language processing application, or deep learning application.

Images