Edge intelligent computing device and method supporting multi-level storage and computing integrated acceleration

Abstract

The invention belongs to the field of edge computing, and discloses an edge intelligent computing device and method supporting multi-level storage and computing integrated acceleration, and the device comprises a main processor system, a near memory computing module and a near flash memory computing module. The near memory calculation module integrates an NPU (Network Processing Unit) with a distributed SRAM (Static Random Access Memory) and an HBM DRAM (Dynamic Random Access Memory) with an MAC (Media Access Control) array, and realizes high-throughput reasoning acceleration And the near flash memory calculation module is used for tightly coupling the SSD controller and the retrieval acceleration engine to realize the nearest retrieval acceleration of mass data at the SSD level. All the modules are interconnected through a PCIe bus, a three-level storage and calculation integrated framework of SRAM-HBM DRAM-SSD is constructed, cross-level data migration is effectively reduced, and the bottleneck of the PCIe bus is avoided. According to the method, high-throughput video reasoning and TB-level data retrieval tasks on the edge side can be efficiently supported at the same time, and the calculation energy efficiency ratio and the system throughput rate are remarkably increased.

Description

Technical Field

[0001] This invention belongs to the field of edge intelligent computing, and in particular relates to an edge intelligent computing device and method that supports multi-level in-memory computing acceleration. Background Technology

[0002] In the field of edge intelligent computing, with the rapid development of big data applications such as smart transportation, industrial internet, and security monitoring, massive amounts of sensor data need to be processed, stored, and retrieved efficiently at the edge. The traditional "analysis-storage-retrieval" model is no longer sufficient to meet the demands of high concurrency, low latency, and high energy efficiency. To address this, the industry has proposed various computing-in-memory (CIM) or near-memory processing (NMP) architectures to alleviate the "memory wall" problem and improve data processing efficiency.

[0003] The existing technical solutions mainly include the following categories: 1. Near-SRAM in-memory computing architecture: For example, Tensorrent's "Wormhole processor" and Cambricon's DaDianNao architecture achieve high-concurrency data access and computation by tightly coupling distributed SRAM next to the MAC computing unit, making them suitable for efficient reuse of feature maps and weights in inference tasks. However, such architectures primarily optimize data interaction between SRAM and computing units, without fully considering the efficiency of data transfer with external DRAM or SSDs.

[0004] 2. Near-DRAM in-memory computing architecture: Near-DRAM processing techniques, as proposed in studies such as GATE, reduce data transmission paths and improve bandwidth utilization efficiency by deploying computing units near DRAM chips. While these methods have achieved some acceleration in image and graph computing scenarios, they still haven't solved the latency and power consumption issues caused by massive data transfer between SSDs and DRAM.

[0005] 3. Near-flash in-memory computing architecture: Computational Storage Drives (CSDs), such as ScaleFlux CSD5000 and OpenSSD, offload some computational tasks (such as data compression and feature retrieval) to the storage device by integrating a dedicated processing engine next to the SSD controller, reducing the burden on the host CPU and DRAM. However, such solutions are usually not deeply integrated with inference acceleration units, making it difficult to support high-throughput inference and retrieval tasks simultaneously.

[0006] 4. Integrated edge intelligent devices: For example, Dahua's "Tiansuan" series of intelligent analysis all-in-one machines use GPU or NPU accelerator cards to realize inference and retrieval functions and have certain storage and computing integration capabilities. However, its architecture is still mainly based on the traditional "host + accelerator card" and has not achieved systematic storage and computing integration optimization at the storage medium level, resulting in data migration bottlenecks.

[0007] In summary, existing technologies primarily focus on in-memory computing optimization for a single storage tier (such as SRAM, DRAM, or SSD), lacking a systematic design for collaborative computing architectures across multiple storage media (SRAM, HBM DRAM, SSD). Furthermore, existing solutions often only support one type of task in inference or retrieval, making it difficult to simultaneously achieve high-throughput inference and massive data retrieval at the edge. Therefore, there is an urgent need for an edge intelligent computing device that supports multi-tiered in-memory computing acceleration to minimize data migration, maximize bandwidth, and optimize energy efficiency at the system level. Summary of the Invention

[0008] The purpose of this invention is to provide an edge intelligent computing device and method that supports multi-level in-memory computing acceleration, so as to solve the above-mentioned technical problems.

[0009] To address the aforementioned technical problems, the specific technical solution of this invention for an edge intelligent computing device and method supporting multi-level in-memory computing acceleration is as follows: An edge intelligent computing device supporting multi-level in-memory computing acceleration includes: The main processor system includes the main CPU, PCIe root complex controller, host high-capacity DRAM, host SSD storage, Ethernet, and peripheral interfaces; The near-memory computing module includes HBM DRAM, SRAM / MAC array distributed NPU computing units, and a dedicated acceleration computing engine; used to achieve high-throughput intensive inference computing. The near-flash computing module includes a CPU, an SSD controller and NAND flash memory, local DRAM, a retrieval computing acceleration engine, and a DMA controller, wherein the retrieval computing acceleration engine is used to perform feature vector similarity calculations. The main processor system, near-memory computing module, and near-flash computing module are interconnected via a PCIe 4.0 or PCIe 5.0 bus; The device supports data multiplexing and in-memory computing acceleration at the RAM-HBM DRAM level, SSD-local DRAM level, and across levels.

[0010] Furthermore, the NPU chip has multiple MAC computing arrays deployed inside, each MAC array being tightly coupled to a local SRAM, forming a distributed computing and distributed storage structure to achieve high-throughput inference computing.

[0011] Furthermore, the NPU chip is the Muxi N100 chip, which supports the HBM DRAM interface and integrates 16MB of on-chip SRAM, distributed among 8×768 MAC arrays.

[0012] Furthermore, the SSD controller in the near-flash computing module is interconnected with the retrieval computing acceleration engine via a local high-speed bus, supporting direct data transfer to the retrieval acceleration engine via DMA, thus avoiding access to the host DRAM through the PCIe bus.

[0013] Furthermore, the near-flash computing module uses the STAR2000 chip, which integrates a multi-core CPU and NPU, supports the NVMe protocol, and provides an open in-memory computing interface.

[0014] Furthermore, the device supports concurrent decoding of multiple video streams, target detection, feature extraction, and retrieval tasks, wherein: The video stream is decoded and inferred by the near-memory computing module, and the generated feature vector is transmitted to the near-flash computing module via PCIe. The retrieval task completes feature vector similarity calculation and sorting within the near-flash computing module, and only returns the sorted Top-K results to the host DRAM.

[0015] Furthermore, the overall power consumption of the device does not exceed 100W, the storage capacity is not less than 30TB, the peak storage bandwidth is not less than 10GB / s, and the 4KB random read / write IOPS is not less than 1.5 million.

[0016] Furthermore, the device simultaneously supports high-throughput inference and massive data retrieval tasks, achieving in-memory computing acceleration across three storage tiers: SRAM, HBM DRAM, and SSD, thereby improving energy efficiency by more than an order of magnitude.

[0017] The present invention also discloses an edge intelligent computing method for the device, comprising: Video decoding and neural network inference are performed in the near-memory computing module, and data reuse is achieved by utilizing the high bandwidth between HBM DRAM and SRAM; Feature vector retrieval is performed in the near-flash computing module, offloading the retrieval computation to a computing unit closer to the SSD, thereby reducing PCIe data transfer. By employing a multi-layered in-memory computing architecture, high-efficiency concurrent execution of reasoning and retrieval tasks is achieved. The edge intelligent computing device and method supporting multi-layered in-memory computing acceleration of this invention have the following advantages: Compared to storage server + inference GPU / NPU card solutions that only support inference acceleration (such as Dahua DH-IVS-GS8000-GU2-HW), and server + smart flash drive (ScaleFlux CSD, OPENSSD, and SMARTSSD) solutions that support data access / retrieval acceleration, this solution adopts in-memory computing technology that supports both inference and retrieval computing acceleration. It achieves in-memory computing acceleration at three levels simultaneously, alleviating the pain point of redundant data transmission between multiple storage media.

[0018] The high-speed access bandwidth between HBM DRAM and on-chip SRAM can reach 100GB / s-400GB / s, supporting high-throughput inference computing and improving data reuse efficiency between SRAM and DRAM. The NPU deploys 16MB of on-chip SRAM, which is evenly distributed in small blocks among 8*768 tightly coupled MAC computing arrays, achieving efficient data throughput and interaction between computing and storage units through a tightly coupled distributed computing and storage approach. Within the near-flash computing module, the internal bus between the SSD and the vector computing acceleration engine is highly interconnected, enabling data access with bandwidths exceeding 10GB / s. Data from the SSD directly reaches the vector unit computing engine via DMA through the internal bus, bypassing the PCIe bus to reach the CPU's DRAM memory. This avoids PCIe bus contention and allows for local access and computation of the vector data required for high-throughput retrieval, thus accelerating dense vector retrieval and achieving near-flash in-memory computing acceleration. These three layers of in-memory computing acceleration provide a technological guarantee for the improved computing energy efficiency of this device. Attached Figure Description

[0019] Figure 1 A schematic diagram of a near-memory (SRAM / HBM) in-memory computing architecture; Figure 2 A schematic diagram of a near-flash memory computing architecture; Figure 3 A schematic diagram of a device architecture to support multi-tiered in-memory computing acceleration; Figure 4 This is a structural diagram of a multi-level in-memory computing device system; Figure 5 Prototype diagram of hardware device for accelerating multi-level in-memory computing; Figure 6 Optimization model diagram for data flow acceleration in multi-level storage computing. Detailed Implementation

[0020] To better understand the purpose, structure, and function of this invention, the following detailed description, in conjunction with the accompanying drawings, provides an edge intelligent computing device and method that supports multi-level in-memory computing acceleration.

[0021] An edge intelligent device supporting multi-level in-memory computing acceleration according to the present invention includes the following construction steps: Step 1: Build an accelerated computing architecture that supports three-tier in-memory computing optimization: High-speed access bandwidth between HBM DRAM and on-chip SRAM can reach 100GB / s-400GB / s, supporting high-throughput inference computing. This requires efficient data multiplexing within the on-chip SRAM to improve storage and computation efficiency. For example... Figure 1 As shown, the invention proposes a high-efficiency NPU acceleration engine architecture supporting HBM and distributed SRAM. A certain number of MAC array units are concurrently deployed within the NPU. Each MAC array unit is tightly coupled to a local SRAM. The MAC array and SRAM achieve efficient data throughput and interaction between the computing and storage units through a tightly coupled distributed computing and distributed storage approach. Another global SRAM collaborates with these SRAMs to exchange data with external DRAM and achieve efficient data sharing for concurrent computations in different blocks.

[0022] HBM DRAM memory utilizes through-silicon vias (TSVs) or silicon interposers for chip stacking, significantly increasing bandwidth while reducing power consumption. HBM DRAM near-memory computing reduces data transfer frequency by connecting computing units close to the DRAM memory chips, achieving low-latency, high-efficiency processing. This HBM DRAM + distributed SRAM near-memory computing architecture greatly reduces redundant data movement of convolution calculation weights and feature coefficients between DRAM, SRAM, and the MAC array, significantly reducing memory access bandwidth, improving hardware computing unit efficiency, and enhancing hardware computing energy efficiency.

[0023] On the other hand, in order to reduce the massive data transfer from SSD to host DRAM via the PCIe bus for TB-level search objects, this invention proposes, as follows: Figure 2 The near-flash in-memory computing architecture shown includes flash memory chips, a flash NVME controller, a system cache DMA controller, local DRAM, a local CPU, an NPU unit that supports near-flash retrieval acceleration, and a specific function hardware acceleration engine. These modules are interconnected through a local high-speed bus and then communicate with the external host processor system through the host PCIe interface.

[0024] The retrieval process involves calculating feature vector similarity between query record features and tens or even hundreds of millions of candidate search objects stored on SSD flash memory. This calculation is offloaded to the local CPU, NPU, and hardware acceleration engine. The tens or even hundreds of millions of search object data are only stored in local DRAM via the local bus and transferred between DRAM and the CPU / NPU / hardware acceleration engine. After vector distance calculation and sorting, the top K matching search objects are returned. Only this object data needs to be returned to the host CPU and host DRAM via the host interface PCIe bus. This method offloads certain repetitive calculations of feature coefficients for tens or even hundreds of millions of target objects to computing units near the SSD storage for accelerated completion. Large amounts of data access do not need to pass through PCIe to the host DRAM storage, thus significantly reducing retrieval storage access bandwidth consumption and meeting the requirements for high computational efficiency, low latency, and massive data retrieval.

[0025] like Figure 3 As shown, this method presents a device architecture supporting multi-tiered in-memory computing acceleration, including a host processor system, a near-memory computing module, and a near-flash computing module. The host processor system comprises a main CPU, a PCIe root complex controller, host high-capacity DRAM, host SSD storage, Ethernet, and other peripheral interfaces. The near-memory computing module includes HBM DRAM, an SRAM / MAC array distributed NPU computing unit, and other dedicated acceleration computing engines; its main function is to implement high-throughput intensive inference computing. The near-flash computing module includes a CPU, SSD controller and NSD chips, local DRAM, a retrieval computing acceleration engine, and a DMA controller. The main processor system, near-memory computing module, and near-flash computing module are interconnected via a PCIe 4.0 or PCIe 5.0 bus.

[0026] Step 2: Construct a multi-layered in-memory computing optimized device architecture and data flow This invention integrates a hybrid in-memory computing architecture of "HBM near-memory computing + SRAM near-memory computing + near-flash computing". Based on the maturity of industrial products, the device integrates the Muxi NPU computing chip N100, which supports HBM near-memory computing + SRAM near-memory computing, and the Yixin chip STAR2000, which supports near-flash computing + SRAM near-memory computing, to develop an embedded edge-integrated optimized in-memory computing device.

[0027] Specifically, the implementation utilizes the Muxi N100, which supports an HBM DRAM interface and connects to 16GB of external HBM DRAM memory. The chip also features 16MB of on-chip SRAM, distributed evenly in small blocks across 8*768 tightly coupled MAC computing arrays. This distributed computing and distributed storage approach enables efficient data throughput and interaction between the computing and storage units. The entire processor supports 8 inference engines and the simultaneous deployment of 8 network models. Additionally, the Muxi N100 incorporates a 96-channel video decoding and encoder hardware acceleration module.

[0028] The STAR2000 utilizes a PCIe 4.0 interface, integrating a 64-bit multi-core CPU and an NPU with up to 8 TOPS of computing power, and fully supports the latest NVMe 2.0 protocol. By tightly coupling the NPU with the SSD controller chip, it provides customers with an open interface, rapidly unleashing the chip's computing power. This integration of storage and computation significantly improves computing speed and efficiency while also saving energy.

[0029] This invention utilizes domestically produced chips, establishing a complete domestic ecosystem encompassing "main controller + storage chip + firmware + algorithm + module". The device's multi-layered in-memory computing architecture can simultaneously support effective data reuse across different storage media layers such as DRAM->SRAM and SSD->DRAM->SRAM, thereby achieving accelerated in-memory computing that simultaneously supports high-throughput inference and massive data retrieval.

[0030] The research and development simultaneously supports HBM DRAM near-memory computing, SRAM near-memory computing, and near-flash computing technologies. It utilizes two domestically produced chips to develop a multi-level in-memory computing device with a total power consumption not exceeding 100W, a storage capacity of 30TB, a peak storage bandwidth exceeding 10GB / s, and 4KB random read / write I / O of no less than 1.5 million IOPS. The energy efficiency of edge high-throughput inference and TB-level massive data retrieval computing is improved by an order of magnitude compared to traditional computing architectures. The specific device system architecture is as follows... Figure 4 As shown, Figure 5 A prototype system of the device is presented. This device supports accelerated inference and retrieval computing, employing in-memory computing acceleration at three levels, providing a computing architecture technical guarantee for improving the device's computing energy efficiency.

[0031] Figure 6A multi-level in-memory computing acceleration optimization model for data flow is presented. Taking multi-channel video decoding, analysis, feature calculation and retrieval as an example, 40-50 channels of compressed video are input to the host via Ethernet and reach the HBM DRAM memory of the N100 near-memory computing module via PCIe bus. The bit rate of a single channel of compressed video is about 1-4Mbps, totaling 40-160Mbps (① in the figure). The N100 near-memory computing module internally decodes 40-50 channels of compressed video streams. The decoded YUV data resides in HBM DRAM memory, initiating algorithms such as target detection, target tracking, image optimization, and feature calculation. These algorithm modules are concurrently deployed as CNN network models on eight hardware inference engines within the N100. During this process, there is very high-bandwidth data transfer between the HBM DRAM and the 16MB SRAM inside the NPU for reading input / output feature coefficients and weight coefficients (Figure ②), and writing output feature coefficients (Figure ③). The high-speed access bandwidth between the HBM DRAM and the on-chip SRAM can reach 100GB / s-400GB / s, supporting high-throughput inference computing and improving data reuse efficiency between SRAM and DRAM. The target object feature vector obtained from the inference computing is sent to the local DRAM within the near-flash computing module via the PCIe bus, and then stored in non-volatile SSD chips according to certain rules (Figure ④). The target object image is stored on the host SSD (Figure ④). This process is high-throughput intensive inference computing.

[0032] During retrieval, the host program sends the query object features and images to the near-flash computing module's DRAM via the PCIe bus (Figure 5). The near-flash computing module, in coordination with the CPU, controls the reading of tens or even hundreds of millions of object feature vector data. The local hardware acceleration engine calculates and sorts the feature vector distances, selecting the top K records most similar to the query object. This process supports data transfer bandwidths up to 10GB / s between the SSD flash memory chips, DRAM, and the hardware acceleration engine, all occurring entirely within the high-speed local bus of the near-flash computing module (Figure 6). After feature vector similarity calculation and sorting, the feature data of the selected top K records is returned to the host DRAM via the PCIe bus (Figure 7), and the corresponding image data for these records is also returned to the host DRAM via the PCIe bus (Figure 8).

[0033] Data from the SSD flash memory is directly delivered to the vector unit computing engine via the internal bus using DMA, without needing to reach the CPU's DRAM memory through the PCIe bus. This avoids PCIe bus contention and enables the vector data required for high-throughput retrieval to be accessed and computed locally, thereby accelerating intensive vector retrieval.

[0034] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. An edge intelligent computing device supporting multi-level in-memory computing acceleration, characterized in that, include: The main processor system includes the main CPU, PCIe root complex controller, host high-capacity DRAM, host SSD storage, Ethernet, and peripheral interfaces; The near-memory computing module includes HBM DRAM, SRAM / MAC array distributed NPU computing units, and a dedicated acceleration computing engine; used to achieve high-throughput intensive inference computing. The near-flash computing module includes a CPU, an SSD controller and NAND flash memory, local DRAM, a retrieval computing acceleration engine, and a DMA controller, wherein the retrieval computing acceleration engine is used to perform feature vector similarity calculations. The main processor system, near-memory computing module, and near-flash computing module are interconnected via a PCIe 4.0 or PCIe 5.0 bus; The device supports data multiplexing and in-memory computing acceleration at the RAM-HBM DRAM level, SSD-local DRAM level, and across levels.

2. The device according to claim 1, characterized in that, The NPU chip has multiple MAC computing arrays deployed inside, and each MAC array is tightly coupled to a local SRAM to form a distributed computing and distributed storage structure, enabling high-throughput inference computing.

3. The device according to claim 2, characterized in that, The NPU chip is the Muxi N100 chip, which supports the HBMDRAM interface and integrates 16MB of on-chip SRAM, distributed among 8×768 MAC arrays.

4. The device according to claim 1, characterized in that, The SSD controller in the near-flash computing module is interconnected with the retrieval computing acceleration engine through a local high-speed bus, supporting direct data transfer to the retrieval acceleration engine via DMA, thus avoiding access to the host DRAM through the PCIe bus.

5. The device according to claim 4, characterized in that, The near-flash computing module uses the STAR2000 chip, which integrates a multi-core CPU and NPU, supports the NVMe protocol, and provides an open in-memory computing interface.

6. The device according to claim 1, characterized in that, The device supports concurrent decoding of multiple video streams, object detection, feature extraction, and retrieval tasks, wherein: The video stream is decoded and inferred by the near-memory computing module, and the generated feature vector is transmitted to the near-flash computing module via PCIe. The retrieval task completes feature vector similarity calculation and sorting within the near-flash computing module, and only returns the sorted Top-K results to the host DRAM.

7. The device according to claim 1, characterized in that, The overall power consumption of the device shall not exceed 100W, the storage capacity shall not be less than 30TB, the peak storage bandwidth shall not be less than 10GB / s, and the 4KB random read / write IOPS shall not be less than 1.5 million.

8. The device according to claim 1, characterized in that, The device simultaneously supports high-throughput inference and massive data retrieval tasks, achieving in-memory computing acceleration across three storage tiers: SRAM, HBM DRAM, and SSD, improving energy efficiency by more than an order of magnitude.

9. An edge intelligent computing method based on the device according to any one of claims 1-8, characterized in that, include: Video decoding and neural network inference are performed in the near-memory computing module, and data reuse is achieved by utilizing the high bandwidth between HBM DRAM and SRAM; Feature vector retrieval is performed in the near-flash computing module, offloading the retrieval computation to a computing unit closer to the SSD, thereby reducing PCIe data transfer. Through a multi-layered in-memory computing architecture, high-efficiency concurrent execution of reasoning and retrieval tasks is achieved.

Images