Multi-die-dot product engine for delivering applications for machine learning on a large scale

Abstract

A multi-chip interface system (200, 455) comprising the following: a plurality of dot product engine (DPE) chips (250a-250l), each of the plurality of DPE chips performing inference calculations to carry out deep learning operations; and a hardware interface between a host computer's memory and the plurality of DPE chips, wherein the hardware interface communicatively connects the plurality of DPE chips to the host computer system's memory during an inference operation, so that the deep learning operations are distributed across the plurality of DPE chips.

Description

State of the art

[0001] Deep learning is an approach based on the broader concepts of artificial intelligence and machine learning (ML). Deep learning can be described as mimicking biological systems, such as the workings of the human brain, in learning information and recognizing patterns for use in decision-making. Deep learning often employs artificial neural networks (ANNs), which are capable of supervised learning from labeled data or unsupervised learning from unstructured or unlabeled data. In one example of deep learning, a computer model can learn to perform classification tasks directly from images, text, or sound. With technological advancements in artificial intelligence, deep learning models (e.g.,Trained with large datasets and neural network architectures containing many layers, deep learning can achieve state-of-the-art accuracy, sometimes surpassing human performance. Because of this increase in performance, deep learning can find a wide variety of practical applications, including function approximation, classification, data processing, image processing, robotics, automated vehicles, and numerical computer control.

[0002] DE 10 2018 129 112 A1 refers generally to the field of network computing and more specifically to a system and method for a system decoder for training accelerators.

[0003] US 2019 / 0 235 889 A1 refers to the virtualization of a memristive Dot Prodict Engine (DPE).

[0004] The present invention is defined by independent claim 1. Embodiments are the subject of the respective dependent claims. Brief description of the drawings

[0005] The present disclosure is described in detail according to one or more different embodiments with reference to the following figures. The figures serve only for illustration and represent only typical or exemplary embodiments. Fig. Figure 1 shows an example of a deep learning accelerator system, including a deep learning accelerator system interface (DLASI) for connecting multiple inference computing units to a host memory, according to some embodiments. Fig. Figure 2 shows an example architecture for a multi-chip interface system with a multi-die dot product (DPE) engine for providing machine learning inference applications at scale according to some embodiments. Fig. Figure 3 illustrates an example architecture of an application-specific integrated circuit (ASIC) used in the multi-chip interface system of Fig. 2 can be implemented according to some embodiments. Fig. Figure 4A shows an example of an object recognition application using the multi-chip interface system of Fig. 2, according to some embodiments. Fig. Figure 4B shows an example of a tile-level pipeline scheme, which allows the multi-chip interface system of Fig. 2 enables the coordination of memory access for images, inferences and the output of results, according to some embodiments. Fig. Figure 5 shows an example of an FPGA (Field Programmable Gate Array) based emulation setup of the multi-chip interface system of Fig. 2 according to some embodiments. Fig. Figure 6 shows an example computer system that performs the function described in Fig. 2 may contain a multi-chip interface system as shown in some embodiments.

[0006] The illustrations do not claim to be exhaustive and do not limit the present disclosure to the exact form shown. Detailed description

[0007] Several embodiments described herein relate to a multi-die dot-product engine for delivering machine learning inference applications at scale. The dot-product engine (DPE) technology and the deep neural network (DNN) inference architecture of the Programmable Ultra-efficient Memristor-based Accelerator (PUMA), currently used in the implementation of deep learning hardware accelerators, are highly energy-efficient. However, these technologies employ an on-die weighted memory approach, which is central to these architectures. The on-die weighted memory approach can limit the DNN size, especially when the hardware is implemented on a single silicon device. To address this drawback, the disclosed multi-die dot-product engine is specifically designed to utilize a multi-chip architecture for DPE.Due to the multi-die architecture, multiple silicon devices can be implemented for inference, enabling energy-efficient inference for large-scale machine learning applications and complex deep neural networks (DNNs). As technology advances, the size and complexity of DNNs can also increase to enable a wide range of applications in emerging markets such as autonomous driving, natural language processing, network security, and countless other applications.

[0008] As described herein, the disclosed multi-die dot-product engine is part of a multi-device deep neural network (DNN) inference system capable of performing high-accuracy object recognition. Furthermore, according to the embodiments, the system can include an extended system interface that is Scalable IOV compliant, allowing hyperconverged servers to employ expansion cards as needed. Additionally, the blocks used in the disclosed example systems are designed for integration into application-specific integrated circuits (ASICs).

[0009] As indicated above, DPE technology can be used to implement a deep learning accelerator system. For example, a deep learning accelerator system with a Deep Learning Accelerator System Interface (DLASI) is described here. The DLASI is designed to provide a high-bandwidth, low-latency interface between cores (e.g., for inference) and servers that might otherwise lack communicative compatibility (in terms of memory). Designing an accelerator consisting of thousands of small cores can present several challenges, such as coordinating the many cores, maintaining high accelerator efficiency despite the radically different problem sizes associated with DNN inference, and completing these DNN-based computational tasks without consuming excessive processing power or die area.In general, coordinating thousands of neural network inference cores can put such a strain on a single host interface controller that the controller can become a performance bottleneck.

[0010] Furthermore, the sizes of different neural networks can vary considerably. Some neural networks may have only a few thousand weights, while others, such as those used in natural language processing or image recognition, can have over 100 million weights. Using large accelerators for each application might seem like a viable brute-force solution. However, if a large accelerator is used to work on a small neural network, the accelerator may not be sufficiently utilized. Additionally, modern servers host many operating systems (OS) and have capacity for only a few expansion cards. For example, a server might have multiple PCIe card slots per processor socket.Mapping large neural networks onto a single die presents several challenges, as there may not be enough on-die memory available to accommodate the increased complexity. Therefore, this paper describes a multi-chip interface solution incorporating a multi-die dot product (DPE) engine for delivering machine learning inference applications at scale. As another example, a PCIe switch on a PCIe form factor card can be used, allowing multiple dies to interact and map large models. Thus, even with a limited number of PCIe lanes, large models can be mapped onto a single PCIe card by using multiple DPE dies.

[0011] Commercial servers (e.g., Xeon-based), personal computers (PCs), and embedded systems like the Raspberry Pi typically run standardized operating systems and contain complex general-purpose CPUs and cache-capable memory systems. Deep learning processors, however, can achieve high performance with a much simpler instruction set and memory architecture. Furthermore, the architecture of a core is optimized for processing smaller numbers, such as 8-bit numbers in operation (as opposed to 32-bit or 64-bit numbers) for neural network inference. The hardware design for a deep learning accelerator can include a substantial number of processors, such as thousands. Because they are deployed in the thousands, these deep learning processors generally do not require high precision.Therefore, processing small numbers can be optimal for their multi-core design, for example, to mitigate bottlenecks. In contrast, commodity servers can operate very efficiently when processing larger numbers, such as 64-bit processing. Due to these (and other) functional differences, some incongruity can occur between the accelerator cores and the servers during deep learning processing. The disclosed DLASI was developed to eliminate such concerns, as indicated above. DLASI implements a multi-die solution that efficiently connects the different types of processing (performed on the accelerator cores and in the host servers) for the interface units in the accelerated system, thereby improving compatibility and increasing the overall system performance.

[0012] According to some embodiments, DLASI comprises a fabric protocol, a processor-based host interface, and a bridge that can connect a server storage system, which views memory as an array of 64-byte (B) cache rows, to a large number of DNN inference processing units, namely the cores (tiles), which view memory as an array of 16-bit words. The fabric protocol can be a two-virtual-channel (VC) protocol, enabling the construction of simple and efficient switches. The fabric protocol can support large packets, which in turn can enable high efficiency. Furthermore, the fabric protocol can be extended to multiple chips using simple ordering rules. In addition, in some cases, the fabric protocol can be layered on top of another protocol, such as Ethernet, for server-to-server communication.Furthermore, in some examples, the host interface can be connected to the server at an "image" layer (or input data sequence layer), directing smaller work segments from the larger layer to the multiple cores. This is achieved by employing a synchronization scheme referred to here as overlapping interval pipelining. Overlapping interval pipelining can be generally described as a combination of send and lock instructions. This pipelining approach allows each of the inference computation units, such as tiles, to be built with a small amount of on-die memory (e.g., a 7nm ASIC, 1 card, 2 dies, 128 tiles), and synchronizes work between the tiles in a way that minimizes tile idle time (thus optimizing processing speed).

[0013] Fig. Figure 1 is an example block diagram of a Deep Learning Accelerator 100, including the DLASI 105. The Deep Learning Accelerator 100 can be implemented as hardware, for example, as a field-programmable gate array (FPGA) or another form of integrated circuit (IC) (such as an application-specific integrated circuit – ASIC). As an FPGA, the Accelerator 100 can contain digital computing units (as opposed to memristor-based analog computing circuits). The Deep Learning Accelerator 100 can have an architecture that allows a variety of deep learning applications to run on the same silicon. As shown in Figure 1, the Deep Learning Accelerator 100 can be implemented as a hardware component. Fig. As shown in Figure 1, the DLASI (indicated by the dashed box) can be a conceptual collective of several components, including: the DLI Fabric protocol connections 108; the host interface 121; the bridge 111; and the switch 107. The deep learning accelerator 100 has an architecture divided into four domains, including: a CODI deep learning inference domain 110, a CODI Simple domain 120, an AMBA4 AXI domain 130, and a Peripheral Component Interconnect Express (PCle) domain 140. Additionally, it serves Fig. Figure 1 illustrates that the DLASI 105 can be implemented as an on-die interconnect, making the disclosed interface a fully integrated and intra-chip solution (with respect to the accelerator chip).

[0014] The PCle domain 140 is depicted as containing a communicative link between a server processor 141 and the DLI inference chip. The PCle domain 140 can include the PCle interface 131 as a high-speed interface for connecting the DLI inference chip to a host processor, such as a server processor. For example, a server motherboard might have several PCle slots for accommodating expansion cards. The server processor 141 can be implemented in a commodity server that communicates with the tiles 106a-106n to perform deep learning operations, such as image recognition. The server processor 141 can be implemented as a rackmount server or a blade server, such as a Xeon server, a ProLiant server, and the like. As indicated above, supporting a multi-card configuration allows the accelerator 100 to support larger DNNs. For a small number of FPGAs or ASICs (e.g.,With four FPGAs, it would be possible to use the PCIe peer-to-peer mechanism. In some cases, a PCIe connection may not provide sufficient bandwidth, and dedicated FPGA-to-FPGA (or ASIC-to-ASIC) connections may be used.

[0015] In the example shown, the CODI Deep Learning Inference Domain 110 comprises the Sea of ​​Tiles 105, a multitude of Tiles 106a-106n, the Switch 107, and the Bridge 111. As can be seen, the Sea of ​​Tiles 10 consists of several Tiles 106a-106n that are communicatively interconnected. Each Tile 106a-106n is configured as a DNN inference processing unit capable of performing deep learning-related tasks such as computations, inference processing, and the like. Thus, the Sea of ​​Tiles 105 can be considered an on-chip network of Tiles 106a-106n, also referred to here as the DLI Fabric. The CODI-DLI domain 110 includes a CODI connection, which is used to connect the tiles to each other and to connect the tiles to a host interface controller 121.

[0016] Each of the individual tiles 106a-106n can also contain multiple cores (not shown). For example, a single tile 106a can contain 16 cores. Furthermore, each core can contain matrix-vector multiplication units (MVMUs). These MVMUs can be implemented with resistor-based crossbars (e.g., memristors) for analog computations in memory, or with digital circuits that include random-access static memory (SRAM) for storing the neural network weights and digital multipliers / adders (as opposed to memristors) for computation. In one embodiment, the core can implement a full set of instructions and uses four 256 x 256 MVMUs. The cores in the tile are connected to a tile memory. Accordingly, for example, the tile memory for tile 106a can be accessed by any of the cores located in tile 106a.Tiles 106a-106n in the sea of ​​tiles 105 can communicate with each other by sending datagram packets to other tiles. The tile memory has a unique flow control management feature: each element in the tile memory has a counter field that is decremented on read operations and set on write operations. Additionally, each of the tiles 106a-106n can have an on-die fabric interface (not shown) for communication with the other tiles as well as with the switch 107. The switch 107 can enable tile-to-tile communication.

[0017] Accordingly, there is an on-die connection that allows the inference chip to communicate with the PCIe domain 140. The CODI deep learning inference domain 110 is a self-contained structure that connects many computing units.

[0018] The Deep Learning Inference (DLI) Fabric protocol connections 108 are configured to provide a communicative link according to the DLI Fabric protocol. The DLI Fabric protocol can use low-level conventions, such as those defined by CODI. The DLI Fabric protocol can be a 2-Virtual Channel (VC) protocol, enabling the construction of simple and efficient switches. The switch 107 can be a 16-port switch, serving as a building block for the design. The DLI Fabric protocol can be implemented as a 2-VC protocol by designing higher-level protocols to minimize fabric downtime. The DLI Fabric protocol supports a large identifier (ID) space, such as 16 bits, which in turn supports multiple chips that can be controlled by the host interface 121.Furthermore, the DLI Fabric protocol can use simple ordering rules, which allows the protocol to be extended to multiple chips.

[0019] The DLASI 105 also includes a bridge 111. In general terms, bridge 111 can be an interface that takes packets from one physical interface and transparently forwards them to another physical interface to enable communication between them. Bridge 111 is represented as the interface between host interface 121 in the CODI Simple domain 120 and switch 107 in the CODI Deep Learning Inference domain 110, bridging the domains for communication. Ultimately, bridge 111 can connect server memory (where memory is considered an array of 64B cache rows) to the DLI fabric, specifically tiles 106a-106n (where memory is considered an array of 16-bit words).In embodiments, the Bridge 111 has hardware functionality for distributing input data to the tiles 106a-106n, for collecting output and power monitoring data, and for switching from processing one image (or input data sequence) to processing the next.

[0020] Host interface 121 provides input data and transmits output data to the host server's memory. To enable simple flow control, the host interface can declare when the next interval will occur and is notified when all PUMA cores of a tile have reached halt instructions. When the host interface declares the start of the next interval, each tile sends its intermediate data to the next set of tiles, which then perform the calculations for the next interval.

[0021] In one example, when a PCIe card boots, a connection in PCIe domain 140 is trained. For instance, the connection in PCIe domain 140 can finish training, start clocking, and retrieve the blocks from reset. Then, the blocks can be initialized on the card. Finally, when a DNN is loaded onto the card, the matrix weights, core instructions, and / or tile instructions can be loaded.

[0022] In Fig. Figure 2 shows an example of the architecture of a multi-chip interface system 200 with multiple DPE dies 250a-250l. Background: Machine learning (ML) has become an integral part of modern life, automating various complex decision-making processes. Due to the data-intensive and matrix-vector multiplication (MVM)-heavy nature of ML algorithms, conventional CMOS hardware can offer limited performance at scale. Therefore, adapting resistive memory technologies, such as memristors, to perform computations in parallel by storing memristor weights has been used as a viable alternative. However, with the increasing complexity of ML algorithms, the computational intensity and size of the models have also grown. The size of neural network models varies greatly depending on the application and the minimum required level of accuracy.For example, small, deep-separable models like Mobile Net-SSD with fewer than 7 million synaptic weight parameters are sufficient for pedestrian and vehicle object detection with modest accuracy for advanced driver assistance systems (ADAS). Significantly larger models like YOLO v3 or R-FCN w / ResNet-101 with over 50 million weights can be used for more accurate object detection for fully autonomous mobility, medical image analysis, and other applications. Even larger models with 200-400 million weights can be used for natural language machine translation.

[0023] Although it would theoretically be possible to build an accelerator ASIC that accommodates 400 million weights in memristor crosspoints on a single silicon die (i.e., 3.2 / 6.4 Gbit at 8-bit / 16-bit weight resolution), the die size would compete with some of today's largest chips (e.g., NVidia V100 GPU @ 815mm2), with low yield, very high cost, large footprint, and support for only a single copy of the largest models on the die, which limits data and model parallelism and leads to suboptimal performance.

[0024] To overcome these aforementioned challenges, the disclosed Multi-Chip Interface System 200 extends the DPE architecture to support the distribution of the neural network model across multiple silicon DPE dies 250a-250l, also referred to here as DPE ASICs. System 200 enables a modular architecture adaptable to a wide range of markets with much smaller chip sizes capable of supporting a large number of weights at low cost (e.g., with a future ASIC supporting up to ~100 million weights). In other words, the number of chips used in a design can be tailored to best meet the complexity, constraints, and performance requirements of the specific DNN-based application. For example, a single chip can be used for embedded markets (ADAS, etc.).) can be used, two chips on a half-width / half-height PCIe accelerator card with 50-75 W form factor for high-performance edge systems (including EL-1000 / EL-4000 servers or workstations) and four or more chips on a standard-size PCIe card or dedicated accelerator tray for data center applications.

[0025] The Multi-Chip Interface System 200 has a scalable architecture that can support larger models. Furthermore, the System 200 offers a power-saving and cost-effective approach while supporting a large number of DPE dies 250a-250l. In the example shown, the System 200 includes at least twelve DPE ASICs 250a-250l. Additionally, the System 200 utilizes switch extensions, specifically the PCle-Switch 260, to support communication links between the multiple DPE ASICs 250a-250l. As shown in Fig. As shown in Figure 2, multiple DPE ASICs can be connected to a single DPE ASIC, which in turn is connected to the PCle-Switch 260. For example, DPE ASIC 250d, DPE 250h, and DPE ASIC 250l are connected to the PCle-Switch 260. Several chips are connected to DPE ASIC 250d, namely DPE ASIC 250a, 250b, and 250c. DPE 250h is linked to DPE ASIC 205e, 250f, and 250g. Furthermore, DPE 250l is connected to DPE ASIC 205i, 250j, and 250k. Accordingly, each of the DPE ASICs 250a-250l communicates with the server processor 241 opposite the PCle-Switch 260. The off-die connections (shown as arrows) between the respective DPE ASICs 250a-250l, the PCle switch 260, and the server processor 241 can be implemented as fabric connections. In one embodiment, the connections are PCle Gen3 connections.

[0026] Alternatively, direct connections (not shown) between the multiple DPE ASICs 250a-250l can be implemented as a simpler and faster approach. However, due to the cost-related scaling inefficiencies associated with increased connectivity when using direct connections, the use of the PCle switch 260 may be preferable.

[0027] To support off-die interconnects to other DPE-ASICs in the multi-chip interconnect system, the DPE-ASIC 250l can also include multiple PCIe root port interfaces 252a-252c. A PCIe root port interface, e.g., 252a, can serve as an interface between the DPE-ASIC 250l and another DPE-ASIC that has a direct interconnect (or fabric connection) to the DPE-ASIC 250l. In some embodiments, each PCIe root port interface 252a-252c corresponds to a connection linking the DPE-ASIC 250l to a corresponding DPE-ASIC. In the example of Fig. 2. PCIe root port interface 252a is connected to a fabric connection to establish a chip-to-chip connection from DPE ASIC 250l to DPE ASIC 250i. PCIe root port interface 252b is connected to another fabric connection, establishing a chip-to-chip connection from DPE ASIC 250l to DPE ASIC 250j. PCIe root port interface 252c is connected to yet another fabric connection, creating a chip-to-chip connection from DPE ASIC 250l to DPE ASIC 250k.

[0028] The DPE-ASIC 250l also includes an on-die AMBA-AXI switch 207. The AMBA-AXI switch 207 enables chip-to-chip communication between the DPE-ASIC 250l and other DPE-ASICs directly connected to it. In comparison to the DLASI-based system (in Fig. (as shown in Figure 1) the AMBA-AXI-Switch 207 supports chip-to-chip communication in a manner similar to the tile-to-tile communication supported by its on-die switch.

[0029] With regard to Fig. 3. The DPE-ASIC 250l can also include a destination PCIe interface 231. The destination PCIe interface 231 can be a high-speed interface for connecting the DPE-ASIC 250l to a host processor, e.g., a server processor (shown in Fig. 2) The DPE-ASIC 250l can be controlled by a host interface 221. Furthermore, the host interface 221 can contain a core 222 for processing functions. In the example shown, the core 222 is implemented as a RISC-V core.

[0030] Back to the DLASI-based system (in Fig. (Figure 1): The system is a hierarchical design that enables multiple tile communication protocols specifically designed for deep learning inference accelerator cores and an area-efficient on-die interconnect. However, this DLASI-based system relies on tile-to-tile sideband lines to indicate when a tile is ready for new data. Modern VLSI designs with fault-tolerant fabrics cannot support sideband signals across multiple chips. Therefore, the deep learning inference fabric is extended to have two virtual channels: one for high-priority synchronization packets and the other for data communication packets.

[0031] In Fig. 3 is a single DPE-ASIC 250l from the Multi-Chip System 200 (shown in Fig. 2) prominently displayed to illustrate the internal components of the ASIC's architecture. In particular, due to the system's multi-chip design, the DPE-ASIC 205l includes a multi-chip support block 251, which is not used in the previously described DLASI-based system. The multi-chip support block 251 is configured to bridge the requirements from off-die interconnect to on-die interconnect. The off-die interconnects of the DPE-ASICs, namely the PCIe connections, are fewer in number and have higher bandwidth compared to the many low-bandwidth on-die connections. To fully utilize the off-die connections, the multi-chip support block 251 is configured to buffer die-to-die packets. To simplify flow control and ensure compatibility with most commercially available fabrics, flow control within the multi-chip interface system is credit-based.Accordingly, the system can efficiently utilize its chip-to-chip connections without introducing overly complex packet arbitration. Furthermore, the fabric protocol and instruction set architecture support up to 4096 tiles. In a single implementation, a multi-chip block can support up to 64 tiles, enabling scaling to accommodate the full range of common DNN sizes.

[0032] Multi-chip scaling does not complicate the higher-level software framework. For example, when a DPE compiler maps a model to the contiguous sea of ​​tiles 205 on the DPE ASIC 250l, it can use logical identifications independent of the hardware topology. This enables seamless mapping from the model definitions provided by a higher-level framework to the DPE hardware. Once the DPE compiler has generated the assembly model with logical tiles, the DPE scheduler software provides the translations into the physical tiles, such as the contiguous sea of ​​tiles 205, which are identified by the combination of chip ID, clump ID, and tile ID. Finally, a DPE loader can transform the assembly code on-the-fly based on the provided translation while loading the model into the hardware.

[0033] In Fig. Figure 4A now shows an example of an object recognition application that is run by a computer system 480 and the multi-chip interface system 455 (also in Fig. 2 shown). The object recognition application 450 can use an output image 452 (in Fig. 4A (represented as "preprocessing image X") is received, such as images streamed to a host computer in a video format (e.g., 1 MB). Therefore, image 452, as mentioned herein, refers to preprocessing (e.g., prior to object detection analysis). Image 452 is then sent to the multi-chip interface system 455 for analysis using DNN inference techniques. This example specifically refers to a You-Only-Look-Once (Yolo) tiny-based implementation, which is a type of DNN that can be used for video object detection applications. According to this example, Yolo-tiny can be mapped to the multi-chip interface system 455. For example, the Multi-Chip Interface System 455 can be implemented in hardware as multiple FPGA chips capable of performing object detection on a video stream using the Yolo Tiny framework as a real-time object detection system.

[0034] An operating system interface 453 on the host can send a request to analyze data in a work queue 454. Next, a doorbell signal 459 can be sent as an indication of the request, which is then transmitted to the host interface of the multi-chip interface system 455. As used here, the term "doorbell" refers to a signal common in interface communication protocols. The doorbell signal 459 is used to inform the host controller that input data is available for processing in the host system's work queue 454. When the work, which relates to image analysis, is placed in the work queue 454 by the operating system interface 453 and the doorbell signal 459 is triggered, the host interface can retrieve the image data from the queue 454.When the analysis results are received from the multi-chip interface system 455, the resulting objects are placed in the completion queue 456 and then transferred to the server's main memory. The host interface can read the request and then prescribe the images using the bridge and the tiles (and the instructions running within them), which analyze the image data for object recognition. According to the embodiments, the DLI fabric protocol is the mechanism that enables this "preparation" of work for the tiles to ultimately be carried out. That is, the fabric protocol and the other previously described multi-chip system components connect the protocol domain with the hardware domain.

[0035] The result of the object recognition application 450 can be a bounding box and a probability associated with a detected object. Fig. 4A shows the post-processing image 460 (in Fig. Figure 4A is shown as "Post-processing image X"), which can result from the execution of the object recognition application 450. Thus, Figure 452 shows the image before it is processed by the object recognition application, and Figure 460 shows the image after processing by the object recognition application, including the various detected objects. There are three bounding boxes around the objects in Figure 460 that have been identified as visual representations of a "person," each with an associated probability shown as "63%," "61%," and "50%." Additionally, there are objects in Figure 160 that are detected as a "keyboard" with a probability of "50%" and as a "TV monitor" with a probability of "51%."

[0036] Figure 4B shows an example of tile-level pipelining, which allows different images to be processed simultaneously. In detail, it shows... Fig. 4B is the multi-chip interface system that coordinates the DMA (Direct Memory Access) transfers of images, inferences, and results. For background: Computationally, typical DNN algorithms largely consist of combinations of matrix-vector multiplication and vector operations. DNN layers use non-linear computations to break the symmetry of the inputs and achieve linear separability. Cores are programmable and can execute instructions to implement DNNs, with each DNN layer being fundamentally expressible as instructions that perform low-level computations. Accordingly, multiple layers of a DNN are typically mapped onto the multiple tiles of the multiple chips of the multi-chip system to perform computations. Additionally, in the example of Fig. 4B The layers of a DNN for image processing are also mapped to tiles 474a-474e of the multi-chip interface system. In the example, the system comprises several chips, namely Chip 0, Chip 1, and Chip 2, and multiple tiles can be implemented on each chip. For example, Chip 0 contains tile 0 474a and tile 1 474b; Chip 1 contains tile 0 474c and tile 1 474d; and Chip 2 contains tile 0 474e. The in Fig. The tile layout shown in Figure 4B is not intended to be restrictive and serves as an example. This layout can vary depending on the implementation, e.g., depending on the size of the neural network model.

[0037] As can be seen, on a server storage level 471, an image 0 472a, an image 1 472b, and an image 2 472c are sent as input to be received by the multiple tiles 474a-474e (on the respective chips) in a pipeline-like manner. In other words, the image data may not be sent simultaneously. Rather, the pipelining scheme, as revealed here, can stagger the transmission and processing of segments of the image data, represented as image 0 472a, image 1 472b, and image 2 472c. Before being received by the tiles 474a-474e, the images 472a-472c are received at the host interface level 473. The host interface level 473 first transmits image 0 472a to the tiles 474a-474e.In this example, the inference work performed by tiles 474a-474e is represented as follows: Tile 0 in chip 0 474a and tile 1 in chip 0 474b are used to map the first layers of the DNN layer computation for image 0 472a; tile 0 in chip 1 474c and tile 1 in chip 1 474d are used to map the middle layers of the DNN layer computation for image 0 472a; and tile 0 in chip 2 474e is used to map the last layers of the DNN layer computation for image 0 472a. Further along the pipeline, and after completion of the final layer computation, the object detection for image 0 475a can be output to the host interface layer 473. At the next interval in the pipeline, this object detection for image 0 475a can be transferred to server memory 471.Furthermore, according to the pipelining scheme, while the object detection for image 0 475a is sent to server memory 475a, the object detection for image 1 475b can be transferred to the host interface level 473.

[0038] In the early stages of a Convolutional Neural Network (CNN), more iterations can be performed than in the later stages of CNN inference. Therefore, in some implementations, additional resources (tiles or cores) can be allocated to these more iterative stages. Overall, image recognition performance can be determined by the pipeline feed rate, and in some examples, the pipeline feed rate can be determined by the tile that takes the longest to complete its work. Before the start of each pipeline interval, the DNN interface sets up the input data and gathers the output data.

[0039] In Fig. Figure 5 is an example of an FPGA-based multi-chip emulation setup 500, which replicates the hardware of the multi-chip interface system (in Fig. 2) emulated. An FPGA emulation platform can enable the execution of some of the commonly used neural network models on the DPE FPGA software development framework. Furthermore, the distribution of neural networks is seen as a first step towards future composability and virtualization, with software-defined partitioning of many single / multi-tenant inference loads across a pool of accelerators without concern for specific physical hardware allocation. As shown in Fig. As shown in Figure 5, commercially available Xilinx Alveo FPGA cards can be used to duplicate the 500 multi-chip setup. Furthermore, an extended PCIe interface can be used in the 500 setup, which was previously used for host communication, for chip-to-chip communication by leveraging PCIe peer-to-peer capability. To enable an interface that scales independently of system-level limitations (e.g., a lack of PCIe lanes), a chip-to-chip (C2C) connection between two FPGA cards is enabled, extending the 400 multi-chip emulation setup across multiple devices and allowing for the mapping of larger models.

[0040] Fig. Figure 6 shows a block diagram of an example computer system 600, in which the (in Fig.2. A multi-chip interface system, as described herein, can be implemented. The Computer System 600 comprises a bus 602 or other communication mechanism for transmitting information and one or more hardware processors 604 coupled to the bus 602 for processing information. The hardware processor(s) 604 may be, for example, one or more general-purpose microprocessors.

[0041] The Computer System 600 also includes main memory 606, such as random access memory (RAM), a cache, and / or other dynamic memory devices connected to bus 602 to store information and instructions to be executed by processor 604. Main memory 606 can also be used to store temporary variables or other intermediate information during the execution of instructions to be carried out by processor 604. Such instructions, when stored in memory media accessible to processor 604, make the Computer System 600 a specialized machine adapted to perform the operations specified in the instructions.

[0042] The Computer System 600 also includes storage devices 610, such as a read-only memory (ROM) or other static storage device, coupled to the bus 602 to store static information and instructions for the processor 604. A storage device 610, such as a magnetic disk, an optical disk, or a USB flash drive, etc., is provided and coupled to the bus 602 to store information and instructions.

[0043] The computer system 500 can be coupled via bus 602 to a display 612, such as a liquid crystal display (LCD) (or a touchscreen), for displaying information to a computer user. An input device 614, including alphanumeric and other keys, is coupled to bus 602 to transmit information and command selections to the processor 604. Another type of user input device is the cursor control 616, such as a mouse, trackball, or cursor directional keys, for transmitting directional information and command selections to the processor 604 and for controlling cursor movement on the display 612. In some embodiments, the same directional information and command selections as with cursor control can be implemented by receiving touch inputs on a touchscreen without a cursor.

[0044] The Computer System 600 can include a user interface module for implementing a graphical user interface, which can be stored on a mass storage device as executable software code that is executed by the computer device(s). This and other modules can contain, for example, components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0045] In general, the terms "component," "engine," "system," "database," "data store," and the like, as used here, can refer to logic embodied in hardware or firmware, or to a collection of software instructions that may have entry and exit points and are written in a programming language such as Java, C, or C++. A software component may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as BASIC, Perl, or Python. Software components may be invoked by other components or by themselves and / or invoked in response to detected events or interrupts. Software components configured to run on computer devices may be stored on a computer-readable medium, such as...Software code can be provided on a compact disc, digital video disc, flash drive, magnetic disk, or other tangible medium, or as a digital download (and may initially be stored in a compressed or installable format that requires installation, decompression, or decryption before execution). Such software code may be stored partially or entirely on a storage device of the executing computer device for execution by that computer device. Software instructions may be embedded in firmware, such as an EPROM. It goes without saying that hardware components may consist of connected logic units, such as gates and flip-flops, and / or may be composed of programmable units, such as programmable gate arrays or processors.

[0046] The computer system 600 can implement the techniques described herein using customer-specific hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, in combination with the computer system, causes or programs the computer system 600 to be a special-purpose machine. According to one embodiment, the techniques described herein are executed by the computer system 600 in response to the processor(s) 604 executing one or more sequences of one or more instructions contained in the main memory 606. Such instructions may be read into the main memory 606 from another storage medium, such as the storage device 610. The execution of the sequences of instructions contained in the main memory 606 causes the processor(s) 604 to perform the process steps described herein.In alternative embodiments, hard-wired circuits can be used instead of, or in combination with, software instructions.

[0047] As used here, a circuit can be implemented in any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be implemented to form a circuit. In implementation, the various circuits described here can be implemented as discrete circuits, or the described functions and features can be partially or completely distributed across one or more circuits.Even if various features or elements of functionality are individually described or claimed as separate circuits, these features and functions may be shared by one or more common circuits, and such a description is not intended to require or imply that separate circuits are necessary to implement these features or functions. If a circuit is implemented wholly or partially in software, such software may be implemented to operate with a computer or processing system capable of performing the functionality described therein, such as the Computer System 500.

[0048] As used herein, the term "or" can be understood in both an inclusive and an exclusive sense. Furthermore, the singular description of resources, processes, or structures is not to be understood as excluding the plural. Conditional expressions such as "may," "could," "might," or "can," unless expressly stated otherwise or understood differently in context, are generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not.

[0049] Terms and expressions used in this document, and variations thereof, should be understood as open and not restrictive unless expressly stated otherwise. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar meaning are not to be understood as limiting the subject matter described to a particular period or to a subject matter available at a particular point in time, but should be understood as encompassing conventional, traditional, normal, or standard technologies that may be available or known now or at any point in the future.The presence of expansive words and phrases such as "one or more", "at least", "but not limited to" or other similar expressions in some cases is not to be understood as implying that the narrower case is intended or required in cases where such expansive phrases may be absent.

Claims

[1] A multi-chip interface system (200, 455) comprising the following: a plurality of dot product engine (DPE) chips (250a-250l), each of the plurality of DPE chips performing inference calculations to carry out deep learning operations; and a hardware interface between a host computer's memory and the plurality of DPE chips, wherein the hardware interface communicatively connects the plurality of DPE chips to the host computer system's memory during an inference operation, so that the deep learning operations are distributed across the plurality of DPE chips. [2] The multi-chip interface system according to claim 1, wherein distributing the deep learning operations over the plurality of DPE chips comprises distributing a neural network model over the plurality of DPE chips. [3] The multi-chip interface system according to claim 1, wherein each of the multiple DPE chips comprises an application-specific integrated circuit (ASIC). [4] The multi-chip interface system according to claim 1, wherein each of the multiple DPE chips comprises: a plurality of port interfaces (252a-252c), wherein each of the plurality of port interfaces enables an off-chip communication link to another DPE chip of the plurality of DPE chips, enabling chip-to-chip communication between the plurality of DPE chips; and a plurality of on-chip connections that connect one or more on-chip components of the DPE chip. [5] The multi-chip interface system according to claim 4, wherein each comprises a plurality of DPE chips: a multi-chip support block (251) wherein the multi-chip support block bridges the one or more on-chip components of the DPE chip to another DPE chip of the plurality of DPE chips and enables communication between the plurality of on-chip connections and the off-chip communication link to another DPE chip of the plurality of DPE chips. [6] The multi-chip interface system according to claim 5, wherein the multi-chip block performs buffering for chip-to-chip packets. [7] The multi-chip interface system according to claim 5, wherein each of the multiple DPE chips comprises: a switch (207) connected to the plurality of port interfaces, wherein the switch enables chip-to-chip communication between the DPE chip and another DPE chip from the plurality of DPE chips via the off-chip communication link. [8] The multi-chip interface system according to claim 1, wherein each of the multiple DPE chips comprises: an interconnected sea of ​​tiles (205) comprising a plurality of inference computation units. [9] The multi-chip interface system according to claim 1, wherein each of the multiple DPE chips comprises: a target interface (231) connected to the hardware interface, wherein the target interface enables communication between the DPE chip and the memory of a host computer via the hardware interface. [10] The multi-chip interface system according to claim 1, wherein the inference operation comprises at least one of the following: an image recognition application, a natural language processing application or a deep learning application.

Images