AI Inference Acceleration System Based on Unified Computing Architecture (UCS)
The AI inference acceleration system based on the Unified Computing Architecture (UCS) addresses the shortcomings of existing AI hardware designs in terms of flexibility and scalability, achieving efficient and flexible AI inference acceleration to adapt to the rapid iteration of AI technology and changes in application scenarios.
Patent Information
- Application Number
- CN202611130029.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-28
- Publication Date
- 2026-11-17
AI Technical Summary
Existing AI hardware designs, focused on pursuing ultimate performance in specialized designs, struggle to adapt to the rapid iteration of AI technology and changes in application scenarios. This results in insufficient flexibility and scalability, slow product iteration, and an inability to quickly respond to market demands.
The AI inference acceleration system adopts a unified computing architecture (UCS), which includes a basic framework layer, an AI-dedicated computing layer, an AI-specific interface adaptation layer, and an AI hardware acceleration unit. Through fully hard-wired control logic and dedicated AI operator hard core acceleration, combined with a multi-queue engine and a DMA acceleration engine, it achieves high computing power utilization and low latency acceleration.
It achieves high computing power utilization and low-latency AI inference acceleration, has stronger scalability and scenario adaptability, can quickly respond to changes in AI algorithms and application scenarios, and improves hardware acceleration efficiency and flexibility.
Smart Images

Figure CN123117320A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to an AI inference acceleration system based on a unified computing architecture (UCS). Background Technology
[0002] In recent years, with the rapid development of deep learning technology, artificial intelligence (AI) has permeated many fields such as autonomous driving, intelligent security, industrial automation, and consumer electronics. In particular, the growing demand for deploying AI inference applications on edge computing and terminal devices has posed unprecedented challenges to the performance, power consumption, cost, and flexibility of AI acceleration chips.
[0003] Currently, mainstream AI inference acceleration hardware solutions can be mainly divided into the following categories:
[0004] (1) Solution based on general-purpose processors (CPU / GPU): AI operations are performed using the vector instruction set of the CPU or the massively parallel computing units of the general-purpose GPU. This type of solution is highly flexible and has a mature ecosystem, but because it is not designed for specific computing modes of AI inference (such as tensor operations), its energy efficiency and performance density are relatively low, making it difficult to meet the stringent power consumption and cost constraints.
[0005] (2) Application-Specific Integrated Circuit (ASIC) Solution: This solution involves deeply customized hardware design for specific AI algorithms (such as a mainstream convolutional neural network CNN), completely solidifying the algorithm logic. This solution can achieve extremely high energy efficiency and performance. However, its disadvantages are also very obvious: once the AI algorithm evolves or the application scenario changes, the solidified hardware is difficult to adapt, resulting in limited chip functionality or even obsolescence. It also has a long R&D cycle, extremely high iteration costs, and lacks scalability and reusability.
[0006] (3) Programmable Logic Device (FPGA) Solution: AI acceleration is achieved through reconfigurable logic units. FPGA provides hardware-level flexibility, but its development threshold is high and the cost is expensive. In addition, it is usually inferior to ASIC solutions with equivalent technology in terms of performance and power consumption. It is mostly used for prototype verification or small-batch specific applications.
[0007] Existing technologies employ hardened, specialized designs in pursuit of ultimate performance in AI hardware design. This conflicts with the flexibility and versatility required for the rapid iteration of AI technology. Developers often face a difficult choice between "investing heavily in developing a dedicated chip" and "using a general-purpose but inefficient hardware platform." When new algorithms emerge or application requirements change, systems based on dedicated hardware often need to undergo expensive re-tape-out, resulting in slow product iteration and an inability to respond quickly to the market. Therefore, existing technologies urgently need a new AI inference acceleration architecture that can improve hardware acceleration efficiency, provide sufficient flexibility and scalability, and adapt to constantly changing AI algorithms and application scenarios. Summary of the Invention
[0008] This invention provides an AI inference acceleration system based on a unified computing architecture (UCS). The main objective of this invention is to solve the technical problems mentioned in the background section of the prior art.
[0009] This invention provides an AI inference acceleration system based on a unified computing architecture (UCS), comprising: Basic framework layer: used to provide general hardware support, with all control logic hardwired, including a global data packet multi-queue engine and a DMA acceleration engine; AI-dedicated computing layer: includes multiple unified computing microcore clusters (UCCs). Each UCC consists of multiple RISC-V microcores with RVV extension features and an internal message engine, serving as a carrier for AI inference and local message interaction. AI-specific interface adaptation layer: used to provide a dedicated transmission path for AI data, realize the isolation and verification of data stream and message stream, and connect to global and local message paths; AI Hardware Acceleration Unit: Fully integrated within each of the Unified Computing Microcore Clusters (UCCs), dedicated to accelerating AI operator hardware cores and receiving scheduling instructions from the UCC's internal message engine.
[0010] In an optional embodiment of the present invention, the data packet multi-queue engine in the basic framework layer is equipped with an AI-dedicated broadcast / multicast message pipeline: the AI-dedicated broadcast / multicast message pipeline includes four independent multicast / broadcast dedicated queues, which are physically isolated from the ordinary AI task queues and independently arbitrated; the multicast / broadcast dedicated queues are configured with a 512-level FIFO buffer to carry multiple UCC global synchronization instructions; the ordinary AI task queues are configured with a 256-level FIFO buffer to carry point-to-point inference task messages.
[0011] In an optional embodiment of the present invention, the data packet multi-queue engine includes a hardwired replication circuit: the hardwired replication circuit is used to generate multiple message copies in real time based on the 16-bit target UCC bitmap field in the received unified 32-byte fixed-length message, and synchronously distribute the global configuration information to multiple specified target unified computing microcore clusters UCC without real-time software overhead through the AI-dedicated broadcast / multicast dedicated message pipeline.
[0012] In an optional embodiment of the present invention, the data packet multi-queue engine adopts a four-level hard-wired priority congestion control mechanism optimized for AI hybrid inference. The priority attributes of different message queues in the four-level hard-wired priority congestion control mechanism can be configured according to scenario requirements. In the AI inference scenario, the four-level hard-wired priority congestion control mechanism includes: priority 0 for global broadcast / multicast synchronization messages, priority 1 for cross-scenario direct pipeline messages, and priorities 2 and 3 for point-to-point messages and batch inference messages; when the queue occupancy rate reaches the statically configured threshold, the enqueueing of low-priority messages is automatically suspended; when the queue is full, the lowest priority message is discarded.
[0013] In an optional embodiment of the present invention, the AI hardware acceleration unit adopts a tensor computing core TCU: the tensor computing core TCU includes a configurable multiply-accumulate MAC array, and 1 to 8 independent multiply-accumulate MAC arrays are statically configured in a single unified computing micro-core cluster UCC; the hardware of the multiply-accumulate MAC array is completely fixed, only the operation parameters are programmable, and the number of arrays cannot be dynamically adjusted during operation.
[0014] In one optional embodiment of the present invention, the tensor computing core TCU is configured with dedicated high-speed on-chip storage: the high-speed on-chip storage is physically independent of the shared data RAM of the unified computing microcore cluster UCC; the high-speed on-chip storage is directly read and written by the tensor computing core TCU and the internal message engine of the UCC; the RISC-V microcore can only indirectly read the data in the high-speed on-chip storage for task verification through a dedicated read-only channel, and cannot directly write computation data, thereby isolating the core memory access and computation path between the microcore and the TCU at the hardware level.
[0015] In an optional embodiment of the present invention, the AI-specific interface adaptation layer includes a unified cross switch within the unified computing microcore cluster (UCC): the unified cross switch enables non-blocking interconnection of the tensor computing core (TCU), the RISC-V microcore, the shared instruction RAM, and the shared data RAM, supporting multiple microcores to simultaneously issue task instructions to the tensor computing core (TCU); data migration and TCU data preloading in AI scenarios are completed by the local DMA of the message engine within the UCC to hide communication latency; while the DMA acceleration engine of the basic framework layer is only responsible for cross-UCC message pass-through and addressing, and does not participate in AI preprocessing.
[0016] In an optional embodiment of the present invention, the microcore cluster configuration within each Unified Computing Microcore Cluster (UCC) in the AI-dedicated computing layer includes: each UCC is configured with 4 to 8 RISC-V microcores, supporting SIMD+MIMD hybrid hard-wired scheduling; each RISC-V microcore is configured with an exclusive 8KB tightly coupled memory (TCM), and each RISC-V microcore is independently addressed without shared cache; the interconnection network adopts a cacheless consistency protocol design.
[0017] In an optional embodiment of the present invention, multiple unified computing microcore clusters (UCCs) in the AI inference acceleration system work collaboratively with the packet multi-queue engine through a mesh network to support heterogeneous hybrid inference scenarios: the unified computing microcore clusters (UCCs) of different application scenarios are statically mapped; wherein, for intra-cluster communication requiring high bandwidth, the high-bandwidth path of the mesh network is fixedly used; for small-batch communication requiring low latency or cross-group cross-scenario communication, the mesh network is bypassed, and the cross-scenario direct-through pipeline or unicast queue of the packet multi-queue engine is used for physical isolation transmission.
[0018] In an optional embodiment of the present invention, in the communication of the cross-scenario direct communication channel, an asynchronous interactive confirmation mechanism is implemented by combining the message ID field in the unified 32-byte message: message synchronization and response confirmation between the message generating end and the consumer end are completely completed by the two ends themselves.
[0019] Beneficial Effects: This invention provides an AI inference acceleration system based on a Unified Computing Architecture (UCS). It includes a basic framework layer providing general hardware support and fully hardwired control logic, including a global data packet multi-queue engine and a DMA acceleration engine; an AI-dedicated computing layer serving as the carrier for AI inference and local message interaction, comprising a unified computing microcore cluster (UCC) consisting of multiple RISC-V microcores with RVV extension capabilities and an internal message engine within the UCC; an AI-specific interface adaptation layer providing a dedicated transmission path for AI data, achieving isolation and verification between data and message streams, and connecting to global and local message paths; and an AI hardware acceleration unit integrated within each UCC cluster, dedicated to accelerating AI operator hardware and receiving scheduling instructions from the UCC's internal message engine. Through hardware collaboration at each level, this invention achieves high computing power utilization and low-latency AI inference acceleration, while also possessing stronger scalability and scenario adaptability. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of an embodiment of an AI inference acceleration system based on a unified computing architecture (UCS) according to the present invention. Figure 2 This is a schematic diagram of an embodiment of a multi-core processor inter-core communication architecture with a 4-core fully interconnected architecture according to the present invention; Figure 3 This is a schematic diagram of an embodiment of a message interface timing according to the present invention; Figure 4 This is a schematic diagram of an embodiment of an inter-core communication system for a multi-core processor according to the present invention; Figure 5 This is a schematic diagram of an embodiment of a network communication device according to the present invention. Detailed Implementation
[0021] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0022] This invention provides an AI inference acceleration system based on a unified computing architecture (UCS). Adhering to the design principle of "unchanged framework, replaceable core," the system employs a layered hardware architecture. Specifically, it includes a basic framework layer, an AI-dedicated computing layer, an AI-specific interface adaptation layer, and an AI hardware acceleration unit. The overall architecture hierarchy and functional positioning of this system can be summarized as follows: Basic Framework Layer: Supported by general-purpose hardware, with all circuits hardwired and programmable only for parameters; it does not participate in AI computation or preprocessing. AI-Dedicated Computing Layer: Composed of a RISC-V RVV microcore cluster and the UCC internal message engine, serving as the core of AI inference and the local message interaction carrier. It adapts to multiple scenarios and relies on a multi-queue engine to complete global message interaction. AI-Dedicated Interface Adaptation Layer: Adds a dedicated AI data transmission path, achieving data stream and message stream isolation and verification, and connecting to global and local message paths. AI Hardware Acceleration Unit: Includes general-purpose and LLM-specific acceleration modules, dedicated to high-frequency AI operator hard-core acceleration, receiving scheduling instructions from the UCC internal message engine. The general-purpose AI operator acceleration module adopts a tensor core architecture.
[0023] Taking a cluster of 8 unified computing microcores (UCC) as an example, the overall framework of the AI inference acceleration system based on the unified computing architecture (UCS) can be described as follows: Figure 1 Eight UCCs constitute the core of the computing layer, employing a mesh network topology. The DMA acceleration engine acts as a message broker for all UCCs, providing message interaction channels across UCCs and between UCCs and the external UCS. When assembling messages, it can actively perform data migration. All eight UCCs are directly connected to the DMA engine. The packet multi-queue engine provides multiple message queues for message interaction between UCCs and the big core, and has the following access interfaces: a fast interface for big cores to access the internal UCS; a high-bandwidth memory access interface to on-chip memory; and an access interface to DDR via the on-chip network.
[0024] More specifically, in this invention, the AI inference acceleration system based on the Unified Computing Architecture (UCS) includes:
[0025] (1) Basic framework layer: used to provide general hardware support, with all control logic hardwired, including a global data packet multi-queue engine and a DMA acceleration engine. In this invention, the basic framework layer serves as the general hardware support base, and its circuit adopts a hardwired design. Only the parameters are programmable, and it does not participate in the specific calculations and data preprocessing of AI. The core of this layer includes a DMA acceleration engine and a multi-queue engine. DMA acceleration engine: connected to all unified computing clusters (Uniform ComputeCluster, UCC), specializing in high-speed data transfer and message pass-through. It is only responsible for message pass-through, static UCC-ID addressing on the sending side, and the transfer of message-related data, and does not participate in any AI preprocessing or message arbitration. Multi-queue engine: as the core of global cross-subject message interaction, it is responsible for the reception, caching, and priority arbitration of global messages.
[0026] To address the latency jitter and software overhead issues arising from peer-to-peer (P2P) message transmission in traditional architectures during multi-cluster synchronization, this invention provides a specialized extension to the multi-queue engine.
[0027] See Figure 2 The multi-queue engine adds four independent multicast / broadcast dedicated queues to the existing conventional point-to-point queues; physical isolation: the four multicast / broadcast dedicated queues are physically isolated from the ordinary AI task queues and are equipped with independent hard-wired arbitration logic (that is, the data packet multi-queue engine in the basic framework layer has an AI-dedicated broadcast / multicast dedicated message pipeline: the AI-dedicated broadcast / multicast dedicated message pipeline includes four independent multicast / broadcast dedicated queues, which are physically isolated from the ordinary AI task queues and arbitrate independently).
[0028] Asymmetric FIFO configuration: The dedicated multicast / broadcast queue is equipped with a FIFO buffer with a depth of 512 levels, specifically for carrying core AI synchronization messages such as multi-UCC global synchronization instructions; the ordinary AI task queue is equipped with a FIFO buffer with a depth of 256 levels, used to carry point-to-point regular inference task messages. This physical isolation design prevents high-frequency multicast synchronization messages from being blocked by large volumes of ordinary point-to-point messages, ensuring that the synchronization latency remains stable at ≤6ns (i.e., the dedicated multicast / broadcast queue is equipped with a 512-level FIFO buffer for carrying multi-UCC global synchronization instructions; the ordinary AI task queue is equipped with a 256-level FIFO buffer for carrying point-to-point inference task messages).
[0029] Hard-wire replication circuit: The multi-queue engine integrates a hard-wire replication circuit. When the multicast / broadcast dedicated queue receives a message that meets the multicast or broadcast conditions, the hard-wire replication circuit generates multiple completely identical message copies in real time at the hardware layer without software intervention. These copies are then synchronously distributed to the target UCCs through a dedicated pipeline. To support fast hard-wired parsing, this system uniformly adopts a 32-byte (256-bit) fixed-length message format. The specific field definitions are shown in Table 1 below (i.e., the data packet multi-queue engine includes a hard-wire replication circuit: the hard-wire replication circuit is used to generate multiple message copies in real time based on the 16-bit target UCC bitmap field in the received uniform 32-byte fixed-length message, and synchronously distribute the global configuration information to multiple specified target unified computing microcore cluster UCCs without real-time software overhead through the AI-dedicated broadcast / multicast dedicated message pipeline). Table 1. Message Format Definition Table of this Invention
[0030] In this invention, the AI task message format is a unified 32-byte fixed message. Broadcast / multicast messages are given new broadcast / multicast identifiers and dedicated fields for synchronization timing requirements. Global messages are arbitrated by a multi-queue engine and moved with the help of DMA. Local messages are flexibly parsed and processed by the CPU core of UCC.
[0031] Asynchronous interactive confirmation mechanism: After the UCC message receiver (consumer) finishes processing the received message, it will send an acknowledgment signal to the message generator (sender) through the reverse path according to the "message ID" in the message. The generator clears the pending confirmation state according to the acknowledgment signal, thus ensuring the reliability of message transmission between multiple UCCs through pure hardware handshake without software intervention (that is, in the communication of the cross-scenario direct channel, the asynchronous interactive confirmation mechanism is implemented by combining the message ID field in the unified 32-byte message: message synchronization and acknowledgment between the message generator and the consumer are completely completed by the two ends themselves).
[0032] In this invention, the data packet multi-queue engine adopts a four-level hard-wired priority congestion control mechanism. The priority attributes of different message queues in the congestion control mechanism can be configured according to scenario requirements. In an optional embodiment of this invention, to ensure the determinism of time-sensitive services in AI hybrid inference scenarios, the four-level hard-wired priority congestion control mechanism can, for example, be: Priority 0 (highest): Global broadcast / multicast synchronization messages, using a dedicated broadcast / multicast channel for rapid parameter synchronization and timing alignment; Priority 1: Cross-scenario direct-access pipeline messages, used for direct data interaction between different inference groups (such as CNN groups and LLM groups); Priority... Level 2: LLM decoding point-to-point messages; Priority 3 (lowest): CNN batch inference messages and system operation and maintenance messages (i.e., the data packet multi-queue engine adopts a four-level hard-wired priority congestion control mechanism optimized for AI hybrid inference. The priority attributes of different message queues in the four-level hard-wired priority congestion control mechanism can be configured according to the scenario requirements. In the AI inference scenario, the four-level hard-wired priority congestion control mechanism includes: priority 0 for global broadcast / multicast synchronization messages, priority 1 for cross-scenario pass-through pipeline messages, and priority 2 and priority 3 for point-to-point messages and batch inference messages).
[0033] Congestion control logic: The multi-queue engine has an internal state monitor. When the occupancy rate of any queue in the multi-queue engine reaches the statically configured threshold (set to 70% in this embodiment), the hardwired control logic automatically suspends the enqueueing of low-priority (e.g., priority 3) messages. If extreme congestion causes the queue to be full, the hardwired control circuit directly discards non-core messages with the lowest priority (priority 3), thereby freeing up FIFO space and absolutely ensuring the real-time performance and low jitter of high-priority inference tasks (i.e., when the queue occupancy rate reaches the statically configured threshold, the enqueueing of low-priority messages is automatically suspended; when the queue is full, the lowest priority messages are discarded).
[0034] In this invention, the basic framework layer only handles AI scene parameter and timing adaptation. The core enhancements include DMA, multi-queue engine hardware implementation details, and UCC internal message engine adaptation mechanisms. Control logic is entirely hardwired, with only parameters programmable. The DMA acceleration engine is dedicated to high-speed data transfer and message pass-through hardware within the basic framework layer. It is only responsible for message pass-through, static UCC-ID addressing (on the message sending side), and message-related data transfer, without participating in AI preprocessing or message arbitration, and connects to all UCCs. Hardware configuration: N sets of direct message read / write interfaces with UCCs (N being the number of UCCs); 64-bit message bus data width; multi-cycle transmission is used when the message length exceeds 64 bits; each message interface is equipped with a 32-depth FIFO, and one set of message interfaces with the multi-queue engine is equipped with a 128-depth FIFO; 1GHz main frequency; hardwired polling arbitration; bottleneck avoidance through multi-path routing, FIFO overflow protection, and pipeline overlapping execution; the timing diagram of the message interface is shown below. Figure 3 As shown, the timing is as follows: a 32-byte (256-bit) message is transmitted with a 64-bit width, which takes 4 cycles to complete (the illustration is only one example). The EOP signal is valid in the 4th cycle.
[0035] (2) AI-Dedicated Computing Layer: This layer comprises multiple Unified Computing Microcore Clusters (UCCs). Each UCC consists of multiple RISC-V microcores with RVV extension capabilities and an internal message engine, serving as the carrier for AI inference and local message interaction. In this invention, the AI-dedicated computing layer uses multiple peer-to-peer UCCs as the computing core for AI inference. The UCCs are interconnected using a 2D Mesh network topology. Each UCC integrates an internal message engine and a RISC-V RVV microcore cluster. The internal message engine, as the local message interaction carrier of the UCC, is responsible for basic task scheduling and topology routing, as well as synchronization between microcores within the UCC, task distribution, and AI tensor computation preprocessing. It also collaborates with a multi-queue engine to complete global message interaction.
[0036] In the AI-dedicated computing layer, each UCC is configured with 4 to 8 microcores with RISC-V RVV (VectorExtension) vector extension characteristics.
[0037] Hybrid Scheduling and Register Configuration: The microcore cluster supports hybrid hard-threaded scheduling of Single Instruction Multiple Data (SIMD) and Multiple Instruction Multiple Data (MIMD). Each microcore is internally configured with 32 64-bit scalar registers and 32 128-bit vector registers (with a 512-level deep vector operation cache) to support efficient pipelined operations on multi-length vectors.
[0038] See Figure 4Near-core storage and no inconsistency overhead design: Each microcore is equipped with an 8KB exclusive tightly coupled memory (TCM).
[0039] Independent addressing and no shared cache: Each microcore addresses its own TCM independently. There is no shared cache at any level between microcore clusters or between microcores and the external UCC.
[0040] The interconnect network adopts a non-consistent design: By eliminating the shared cache, the system hardware design completely eliminates the hardware overhead and protocol handshake latency of maintaining multi-core global cache coherency. The interconnect network adopts a non-consistent protocol design, and the microcore TCM storage interface directly connects to the internal message engine using the AXI4-Lite protocol. While ensuring low data transmission latency, this greatly reduces the hardware complexity and gate circuit area of RTL (i.e., the microcore cluster configuration in each unified computing microcore cluster UCC in the AI dedicated computing layer includes: each unified computing microcore cluster UCC is configured with 4 to 8 RISC-V microcores, supporting SIMD+MIMD hybrid hard-wired scheduling; each RISC-V microcore is configured with a dedicated 8KB tightly coupled memory TCM, and each RISC-V microcore is independently addressed and has no shared cache; the interconnect network adopts a cacheless coherent protocol design).
[0041] (3) AI-dedicated interface adaptation layer: This layer provides a dedicated transmission path for AI data, enables isolation and verification between data streams and message streams, and connects to global and local message paths. In this invention, the AI-dedicated interface adaptation layer adds a dedicated transmission path for AI data to achieve isolation and verification between AI business data streams and control message streams. This layer connects to the global Mesh network path and the UCC local message path to avoid bus congestion during high-concurrency data transmission.
[0042] See Figure 4In this invention, the AI-specific interface adaptation layer is manifested as a unified crossbar within the UCC. This crossbar enables non-blocking interconnection between the TCU, RISC-V microcore, shared instruction RAM, and shared data RAM, allowing multiple microcores to simultaneously issue task instructions to the TCU. In terms of workflow division, large-scale data transfers in AI scenarios (such as feature map and weight loading) are completed by the local DMA of the UCC's internal message engine to hide communication latency. Meanwhile, the global DMA acceleration engine of the basic framework layer is only responsible for cross-UCC message pass-through and addressing, and does not participate in AI preprocessing operations. (That is, the AI-specific interface adaptation layer includes a unified cross switch within the unified computing microcore cluster UCC: the unified cross switch enables non-blocking interconnection of the tensor computing core TCU, the RISC-V microcore, the shared instruction RAM and the shared data RAM, supporting multiple microcores to simultaneously issue task instructions to the tensor computing core TCU; data migration and TCU data preloading in AI scenarios are completed by the local DMA of the message engine within the UCC to hide communication latency; while the DMA acceleration engine of the basic framework layer is only responsible for cross-UCC message pass-through and addressing, and does not participate in AI preprocessing).
[0043] (4) AI Hardware Acceleration Unit: Fully integrated within each of the Unified Computing Microcore Clusters (UCCs), dedicated to accelerating AI operator hardware cores and receiving scheduling instructions from the UCC's internal message engine. See also Figure 4 In this invention, the AI hardware acceleration unit is specifically manifested as a Tensor Core Unit (TCU) integrated inside each UCC. The TCU is a dedicated AI computing hardware for the UCC, not shared with other UCCs. It is dedicated to hard-core acceleration of high-frequency AI operators and directly receives scheduling instructions from the internal message engine of the UCC.
[0044] For example, in this invention, each UCC independently integrates a TCU as its dedicated AI hardware acceleration unit, and the TCUs of each UCC are physically isolated and do not share with each other.
[0045] TCU Hardware Configuration: TCU Tensor Computation Array: Composed of 2D Multiply-Accumulate (MAC) arrays. A single UCC can be statically configured with 1, 2, 4, or 8 independent MAC arrays at the factory, depending on computing power requirements. Each standard array has 512 INT8 MAC units, providing a peak computing power of 1.024 TOPS at a 1GHz operating frequency. All array hardware is fully fixed, and the number of arrays cannot be dynamically changed during operation (i.e., the AI hardware acceleration unit uses a tensor computation core TCU: the tensor computation core TCU includes configurable multiply-accumulate MAC arrays, and a single unified computing micro-core cluster UCC is statically configured with 1 to 8 independent multiply-accumulate MAC arrays; the multiply-accumulate MAC array hardware is fully fixed, only the computation parameters are programmable, and the number of arrays cannot be dynamically adjusted during operation).
[0046] TCU Dedicated Local Memory: The capacity is configured at 256KB (linearly matchable between 32KB and 512KB depending on the MAC size). This memory is physically independent of the shared data RAM within the UCC, employs a single-cycle read / write latency design, and features a 512-bit data bus width, providing extremely high memory access bandwidth (the tensor computing core TCU is equipped with dedicated high-speed on-chip storage: this high-speed on-chip storage is physically independent of the shared data RAM of the unified computing microcore cluster UCC).
[0047] Crossbar interconnection interface: Adopting the AXI4 protocol, the TCU, RVV microcore cluster, shared instruction RAM and shared data RAM are physically interconnected without blocking through a high-bandwidth crossbar, supporting four microcores to issue task instructions to the TCU at the same time without bus conflict.
[0048] To completely avoid bandwidth contention between the CPU (microcore) and the hardware accelerator (TCU) in high-concurrency scenarios (i.e., the memory wall problem), this invention designs an asymmetric memory access path at the hardware level: Write path isolation: The TCU-dedicated local memory only allows direct read and write access from the TCU itself and the UCC internal message engine (which integrates a local DMA controller). The RISC-V microcore inside the UCC cannot directly write computation data to this dedicated memory; Read-only verification channel: Only a dedicated hardware read-only channel is established between the microcore and the TCU-dedicated local memory; the microcore can only indirectly read the computation results in the TCU local memory for task verification through this channel (that is, the high-speed on-chip storage is directly read and written by the tensor computation core TCU and the UCC internal message engine; the RISC-V microcore can only indirectly read the data in the high-speed on-chip storage for task verification through the dedicated read-only channel, and cannot directly write computation data, thus isolating the core memory access and computation path between the microcore and the TCU at the hardware level).
[0049] Collaborative Workflow: Before performing tensor calculations, the UCC internal message engine, based on RVV instructions, uses the local DMA controller to preload input data from external or shared data RAM into the TCU's dedicated local memory. Since this process is autonomously controlled by the hardware message engine and supports multi-command batch caching and pipelined overlapping execution, communication latency is completely hidden in the calculation process.
[0050] In this invention, in addition to message reception and transmission, task scheduling, general DMA, and topology routing processing as defined by the UCS, the AI computing layer is also responsible for inter-microcore synchronization within the UCC, task distribution within the UCC, and AI tensor computation preprocessing. In terms of hardware configuration, each microcore is equipped with a 64-level local message FIFO, a 1GHz main frequency, hard-wired AI data preprocessing circuitry, global synchronization message reception and processing, a dedicated high-speed access interface with local memory, and a data bit width of 1024 bits. The microcore TCM storage interface uses the AXI4-Lite protocol to ensure low latency data transmission and support local message interaction and data preprocessing collaboration in high-concurrency scenarios.
[0051] To better illustrate the technical solution of the present invention, an exemplary deployment scenario is provided as follows: See Figure 5 This embodiment demonstrates a hardware system with a total peak computing power of 96 TOPS (composed of 12 standard UCC peer clusters of approximately 8 TOPS each) for simultaneously performing large-scale CNN (such as YOLOv8x object detection) and LLM (such as Llama3-8B large language model) hybrid inference tasks. The system achieves efficient matching of hardware and software through the collaboration of a multi-queue engine and an on-chip 2D Mesh network.
[0052] Basic framework layer extended configuration: For clusters with 12 UCCs and the collaborative needs of hybrid inference scenarios, only the reserved extended resources of the basic framework layer are activated, without modifying the core RTL code:
[0053] 1. Multi-queue engine: Use the default configuration directly.
[0054] 2. DMA Acceleration Engine Expansion: The number of message channels directly connected to UCCs is expanded to 12 groups as the number of UCCs increases.
[0055] 3. Global Interconnection Network: The 12 UCCs are fully interconnected through a 3×4 regular 2D mesh network in the basic framework layer. The link data width is 128 bits, 32 bytes. The total bandwidth of the mesh network = 17 effective links × 32GB / s, and the bidirectional bandwidth of a single link = 544GB / s.
[0056] 4. TCU configuration: A tensor array with 4096 INT8 MAC units, with a peak computing power of approximately 8 TOPS@1GHz, and hardware-optimized CNN-specific operations such as convolution kernel sliding, pooling, and non-maximum suppression (NMS).
[0057] 5. Storage Configuration: CNN: 512KB shared data RAM for UCC, 256KB dedicated local memory for TCU, 8KB / core TCM; LLM: 1MB shared data RAM for UCC, 512KB dedicated local memory for TCU, integrated KVMU cache management engine and real-time decompression engine, KV cache capacity 512KB.
[0058] 6. Interconnection Configuration: CNN: Prioritizes access to the high-bandwidth mesh network path to optimize batch data transmission performance, supplemented by a multi-queue engine message path; LLM: Prioritizes access to the low-latency multi-queue engine path to optimize small-batch, high-frequency message transmission performance, supplemented by a mesh network.
[0059] The interconnection strategy of this invention can be summarized as shown in Table 2 below: Table 2. Summary of an Interconnection Strategy of the Invention
[0060] This embodiment extends the functionality of the multi-queue engine in the basic framework layer to achieve efficient collaboration between CNN and LLM tasks. All collaboration mechanisms are hardwired and fixed without real-time software involvement, ensuring low latency and high determinism.
[0061] Global instruction broadcast synchronization mechanism: Global configuration information such as model weights, inference thresholds, and system parameters are distributed to 12 UCCs at once through a dedicated broadcast queue for mixed scenarios using a multi-queue engine; the hard-wired replication circuit of the multi-queue engine synchronously generates 12 message copies; messages sent only to the CNN group / LLM group are precisely distributed to the target UCC by specifying the target UCC through a multicast bitmap to avoid bandwidth waste.
[0062] Cross-Scene Direct Pipeline Collaboration Mechanism: Addressing the high-frequency interaction needs between CNN and LLM (e.g., sending CNN detection results to LLM for semantic analysis), a multi-queue engine reuses the basic UCC inter-pathway, optimizes priorities for cross-scene interactions, and forms a cross-scene direct pipeline that eliminates the need for mesh network interaction. After the CNN UCC completes object detection, it sends the results directly to the designated LLM UCC via a straight-through pipeline, without the need for the main control UCC to relay them. Straight-through channels are dedicated hardware pathways, physically isolated from ordinary message pathways, resulting in low end-to-end transmission latency. Supports batch result transmission, with a maximum of 64 detection target information items transmitted at a time, meeting high concurrency requirements. Hardline priority congestion control mechanism: The multi-queue engine employs a four-level hard-wired priority arbitration mechanism, specifically optimized for the latency-sensitive characteristics of hybrid inference services (priority 0 is the highest priority). Its core purpose is to prioritize the stable operation of global synchronization and time-sensitive inference services, preventing core tasks from being blocked by non-core messages. Priority 0: Global broadcast / multicast synchronization messages Priority 1: Cross-scenario direct pipeline messages Priority 2: LLM-Decode peer-to-peer messages Priority 3: CNN batch inference messages and operation and maintenance messages When the queue occupancy rate reaches the statically configured threshold (typically 70%), low-priority message enqueueing is automatically paused; when the queue is full, only the lowest priority messages are discarded to ensure that core inference tasks are not affected.
[0063] In this embodiment, all physical links have the same specifications: 128-bit width, 1GHz main frequency, full-duplex bidirectional transmission, and a single-link bidirectional bandwidth of 32GB / s; internal nodes (such as UCC5): 4 neighbors (UCC1, UCC4, UCC6, UCC9); corner nodes (such as UCC0): 2 neighbors (UCC1, UCC4); edge nodes (such as UCC1): 3 neighbors (UCC0, UCC2, UCC5).
[0064] Static partitioning deployment: The system statically divides the 12 UCCs into two peer groups in the physical topology: The CNN-dedicated UCC group consists of UCC0, UCC1, UCC4, UCC5, UCC8, and UCC9 in the left two columns, and is used to perform 6-channel 4K@30fps object detection tasks (YOLOv8x). Its UCC shared data RAM is configured with 512KB, and the TCU's dedicated local memory is configured with 256KB.
[0065] LLM Dedicated UCC Group: Composed of UCC2, UCC3, UCC6, UCC7, UCC10, and UCC11 in the right two columns, it is used to handle the inference of the large language model (Llama 3-8B). Its UCC shared data RAM is configured with 1MB, TCU dedicated local memory is configured with 512KB, and it integrates the KV cache management engine (KVMU).
[0066] In this invention, the core principle of the mapping strategy is: dense communication within a group and sparse communication across groups: within a CNN group, feature map slice transmission and batch parameter synchronization are performed, resulting in large communication volume and low latency; within an LLM group, KV cache synchronization and pipeline inter-layer data transmission are performed, resulting in frequent communication and high latency; and across groups, only a small number of small data packets are exchanged.
[0067] Partition mapping minimizes the number of communication hops within a group: CNN / LLM groups each have 2×3 sub-mesh, with a maximum communication hop count of 3; hybrid mapping increases the maximum hop count to 5, resulting in a 67% increase in latency.
[0068] Cross-group communication bypasses the mesh network and uses a direct pipeline: Cross-CNN-LLM business interaction bypasses the mesh network and is achieved through a direct path between UCCs using a multi-queue engine, with end-to-end latency ≤6ns, which is much lower than the cross-group hop latency of the mesh network.
[0069] The core features of the direct path are: reuse existing hardware functions of the basic framework and optimize only the priority configuration; directly connect two UCCs via the DMA acceleration engine and multi-queue engine without going through mesh routing; physically isolated from the ordinary message path, configured with high priority to ensure low latency.
[0070] In summary, the division of labor and complementary mechanism of this invention can be summarized in Table 3 below: Table 3. Division of Labor and Complementary Mechanism of the Invention
[0071] For a specific example, the differentiated interconnection strategy adopted by this invention can be as follows: Considering the differences in service transmission characteristics between CNN and LLM, the system statically configures different interconnection strategies in hardware scheduling, as shown in Table 4 below (i.e., multiple unified computing micro-core clusters (UCCs) in the AI inference acceleration system work collaboratively with the packet multi-queue engine through a Mesh network to support heterogeneous hybrid inference scenarios: the unified computing micro-core clusters (UCCs) for different application scenarios are statically mapped; wherein, for intra-cluster communication requiring high bandwidth, a fixed high-bandwidth Mesh network path is used; for small-batch communication requiring low latency or cross-group cross-scenario communication, the Mesh network is bypassed, and the cross-scenario direct-through pipeline or unicast queue of the packet multi-queue engine is used for physically isolated transmission): Table 4. A Differentiated Interconnection Strategy Table of the Present Invention
[0072] 3. Cross-scenario direct access pipeline mechanism: In the hybrid inference business flow, the target results detected by the CNN group (such as vehicle and face locations) need to be sent to the LLM group at high frequency for semantic analysis.
[0073] To avoid this cross-group communication blocking the regular Mesh network, the multi-queue engine reused and optimized the direct physical path between UCCs, constructing a cross-scenario direct communication pipeline: Physical isolation and high priority: This pass-through is physically independent of the regular Mesh routing path and is given high hardwire priority (priority 1).
[0074] Straight-through transmission process: After the CNN UCC (such as UCC5) completes target detection, it does not need to be relayed through the main control core, nor does it need to be forwarded by hop-by-hop routing through the Mesh network. Instead, it directly writes the detection results into the receive FIFO of the LLM UCC (such as UCC6) in batches through the cooperation of the multi-queue engine and the DMA acceleration engine.
[0075] Technical Results: The cross-group end-to-end transmission latency is ≤6ns (supporting batch transmission of up to 64 detection target information items per batch), achieving efficient parallel collaboration between CNN and LLM tasks at the hardware layer. This embodiment verifies the significant advantages of the AI inference acceleration system architecture of this invention in high-computing-power hybrid inference scenarios. Through UCC differentiated static configuration and a fully hard-wired collaborative mechanism, it achieves efficient parallelism of CNN and LLM tasks, maintaining high computing power utilization, low latency, and high determinism, meeting the needs of high-end edge computing scenarios.
[0076] In summary, the AI inference acceleration system based on the Unified Computing Architecture (UCS) of this invention has the following three core innovations:
[0077] Core Innovation Point 1: Multi-queue engine with dedicated AI broadcast / multicast message pipelines This invention adds four independent multicast / broadcast queues to the multi-queue engine, physically isolated from and independently arbitrated by the ordinary AI task queues. The multicast / broadcast queues are configured with a 512-level FIFO buffer, specifically for carrying core AI synchronization messages such as multi-UCC global synchronization instructions; the ordinary AI task queues are configured with a 256-level FIFO buffer, used to carry point-to-point inference task messages. This clear division of labor prevents multicast synchronization messages from being blocked by ordinary messages, ensuring a stable synchronization latency of ≤6ns, while reducing the complexity of hardwired replication circuitry and improving system reliability.
[0078] This innovation, one of the core technologies of this invention, is essentially an extension of the dedicated message pipeline for a multi-queue engine. It addresses the pain point of the basic patent's point-to-point transmission being unable to adapt to AI multi-cluster collaboration, complementing the mesh network in a division of labor, and eliminating real-time software scheduling overhead. Software running in the UCC can choose to have messages travel via the mesh network or directly reach the target UCC through the unicast queue of the multi-queue engine. The hardware framework is as follows... Figure 2 As shown, the multi-queue engine is represented by the gray area.
[0079] Each message is fixed at 32 bytes in length and includes the queue ID, a synchronization timing requirement field (used for synchronization between UCCs), and custom information; multicast / broadcast specifies the target queue ID in bitmap form, and the message format is defined as shown in Table 1 above.
[0080] The enqueue message interface receives messages from the big core and UCC (via the DMA acceleration engine), and routes them to the specified destination queue according to the information carried in the message. In multicast / broadcast scenarios, messages are synchronously distributed to the specified queue according to the bitmap information. At the same time, an asynchronous interactive confirmation mechanism is implemented based on the message ID. After the message receiver completes its processing, it sends back an acknowledgment signal through the corresponding message ID, ensuring the reliability and integrity of multi-UCC message transmission.
[0081] Queue number configuration reference: The number of queues is not directly related to the number of UCCs or the number of large CPU cores, but is determined by the needs of the business scenario; the default setting is 32 ordinary queues + 4 special queues, for a total of 36 queues.
[0082] Core positioning: Belongs to the multi-queue engine of the basic framework layer, relying on the original foundation for expansion without changing the core architecture; the addition of dedicated queues, hard-wired replication circuits and arbitration logic, together with conventional pipelines and direct paths, constitutes the three core capabilities, which are the core support for AI multi-cluster collaboration.
[0083] Hardware implementation: All hardwired and fixed, the core consists of 4 dedicated broadcast / multicast queues (512-level FIFO), hardwired replication circuits, and dedicated message fields; priority is configurable, the hardwired replication circuits can generate multiple message copies in real time, and dedicated fields are used to specify the distribution type and target.
[0084] Workflow: The message generator can set a dedicated field to identify distribution requirements; The multi-queue engine routes messages to dedicated queues and arbitrates them according to priority; Hardwired replication generates copies, which are then distributed to the target UCC via a dedicated pipeline. Asynchronous interaction, message synchronization and confirmation are handled automatically by the generating and consuming ends; It shares the workload with the mesh network, mainly handling the synchronization of core AI messages across multiple consumer terminals, and providing a message transmission channel that bypasses the mesh network.
[0085] This innovation achieves efficient distribution of a single message across multiple devices through a combination of dedicated queue physical isolation, hard-wired replication, AI-specific fields, and mesh-based task allocation. Priority, FIFO depth, and dedicated fields are all custom-designed for AI.
[0086] Application Adaptation: In CNN scenarios, it is used to distribute multi-UCC synchronization instructions and global parameters; in LLM scenarios, it is used to distribute weight configurations in the Prefill stage and KV cache synchronization messages in the Decode stage, serving as the core support for multi-cluster collaboration.
[0087] Core Innovation Point Two: Design of Supporting AI Hardware Acceleration Unit The general-purpose AI operator acceleration module adopts a heterogeneous collaborative architecture with a shared TCU (tensor core unit) based on RISC-V RVV microcores within the UCC. It is fully integrated within a single UCC cluster and is dedicated AI computing hardware for the UCC, not shared with other UCCs. (See [link to relevant documentation]). Figure 4 .
[0088] To implement efficient tensor operations in this invention, the following components need to work closely together and coordinate efficiently: the TCU tensor computation array (i.e., the TCU itself), the TCU dedicated local memory, the crossbar interconnect interface, and the data transfer mechanism of the message engine. TCU Tensor Computation Array: The core of general-purpose AI operator computing power, adopting a configurable multiply-accumulate (MAC) array architecture; a single UCC can be statically configured with 1 / 2 / 4 / 8 independent MAC arrays according to computing power requirements. Each standard group has 512 INT8 MAC units, corresponding to 1.024 / 2.048 / 4.096 / 8.192 TOPS@1GHz (approximately 1 / 2 / 4 / 8 TOPS) of INT8 peak computing power; all array hardware is fixed, only the operation parameters are programmable, and the number of arrays cannot be dynamically adjusted during operation.
[0089] The formula for calculating computing power is: INT8 peak computing power (TOPS) = number of MAC units × operating frequency (GHz) × 2 ÷ 1000; each group has 512 MAC units @ 1GHz: 512 × 1 × 2 ÷ 1000 = 1.024 TOPS ≈ 1 TOPS; TCU Dedicated Local Memory: High-speed on-chip storage dedicated to the TCU, physically independent of the UCC shared data RAM, with a capacity configuration of 32KB-512KB (linearly matched to the TCU's computing power specifications), single-cycle access design, and single-cycle read / write latency; the data bus width is 512bit, and it has a high-speed memory access interface; this memory is directly accessed by the TCU and the UCC's internal message engine (locally integrated DMA controller), and the microcore can only indirectly read data through a dedicated read-only channel for task verification, and cannot directly write computation data, thus isolating the core memory access and computation path between the microcore and the TCU from the hardware level, completely avoiding bandwidth contention issues.
[0090] Crossbar interconnect interface: Non-blocking interconnection between TCU and RVV microcore, shared instruction RAM, and shared data RAM is achieved through a unified crossbar switch inside the UCC. The data path adopts the AXI4 protocol, which supports four microcores to send task instructions to TCU at the same time without bus conflict.
[0091] The message engine performs data migration: The message engine preloads the data required by the TCU according to the RVV instructions, supports batch caching and processing of multiple commands, and hides communication latency; while the DMA acceleration engine is only responsible for message pass-through and addressing, and the data migration in the AI scenario is completed by the local DMA of the UCC internal message engine.
[0092] Core Innovation Point 3: Specialized Customization of AI-Specific Computing Layer This innovation is specifically customized for AI. Each UCC is configured with 4 to 8 RISC-V microcores with RVV characteristics, supporting SIMD+MIMD hybrid hard-wired scheduling; it is configured with 32 64-bit scalar registers and 32 128-bit vector registers (with 512-level deep vector operation cache), supporting multi-length vector operations; the core modules (operation unit, register group, local storage controller) have fully fixed RTL; each microcore is equipped with 8KB TCM dedicated storage, each microcore is independently addressed and there is no shared cache, so there is no global cache consistency maintenance overhead, and the interconnection network adopts a non-consistency protocol design, which greatly reduces hardware complexity and logical latency.
[0093] Support for instruction extension interface: After integrating the TCU, the RISC-V core instruction pipeline sends commands to the TCU through the instruction extension interface, which can receive feedback results from the TCU.
[0094] Instruction RAM: UCC configuration instruction RAM, which is shared by the RISC-V microcore inside the UCC. The default configuration is 16KB, which can be changed according to actual needs.
[0095] Synchronization between microcores: The UCC internal message engine is responsible for synchronization between microcores and within the UCC; global / inter-UCC synchronization is achieved by a dedicated broadcast / multicast pipeline of the multi-queue engine.
[0096] Single UCC computing power options: The computing power is mainly provided by the Tensor Core. The computing power depends on the Tensor Core specifications and bandwidth matching. The four specifications do not require any changes to the overall UCC structure. They are achieved only by the number and specifications of the Tensor Cores, data bandwidth, and precision configuration. Expansion is achieved by stacking multiple UCCs.
[0097] Computing power statistics: 1 INT8 MAC = 2 OPs; a single UCC can be selected with approximately 1 / 2 / 4 / 8 TOPS specifications. Different computing power requirements can be adapted by horizontally stacking UCC clusters by adding or removing them, and a unified mesh interconnection + multi-queue engine collaboration is adopted.
[0098] Tiered configuration rules: INT8 computing power requirements of 8 TOPS or less are met by a single UCC; scenarios below 0.5 TOPS adopt a half-size MAC array design, achieving lightweight configuration by hardware shielding half of the MAC units and matching corresponding bandwidth and timing parameters, without modifying the main UCC architecture; 0.5-1 TOPS uses one approximately 1 TOPS UCC; 1-2 TOPS uses one approximately 2 TOPS or two approximately 1 TOPS UCCs; 2-4 TOPS uses one approximately 4 TOPS or multiple low-size UCCs; 4-8 TOPS uses one approximately 8 TOPS or multiple low-size UCCs; higher computing power is achieved by horizontally stacking multiple UCCs, strictly adhering to the core design principle of unchanged framework and replaceable core.
[0099] Cluster interconnection design: Multiple UCCs collaborate through a mesh network (regular data / messages), a direct path through multiple queue engines (to compensate for speed shortcomings), and a dedicated broadcast / multicast pipeline (AI multi-terminal synchronized messages); each UCC is managed by its own message engine, and the interconnection interface and synchronization mechanism are unified.
[0100] Based on the above-mentioned innovations, the AI inference acceleration system based on the Unified Computing Architecture (UCS) of this invention, through the synergy of the three core innovations mentioned above, has at least the following significant advantages compared to existing technologies:
[0101] (1) Strong scene adaptability and low R&D iteration cost: The basic framework layer is fully reused, and only the AI-related modules are statically adapted to single / multiple scenarios to solve the problem of architecture fragmentation; relying on the mature foundation of basic patents, the R&D cost is greatly reduced and the iteration cycle is shortened.
[0102] (2) High message interaction efficiency and strong operational determinism: a dual-level hardware message interaction mechanism and a dedicated broadcast / multicast pipeline to solve the pain point of multi-terminal synchronous message distribution; all hard-wired solidification, no real-time software scheduling intervention, low latency jitter and stable operation.
[0103] (3) Strong scalability and good industrial adaptability: single UCC computing power is optional and multiple UCCs can be horizontally stacked to adapt to computing power requirements of 1TOPS-100TOPS+. The interconnection interface and synchronization mechanism are unified. The hardware meets industrial-grade standards and can be widely used in high-end edge AI devices, with strong market competitiveness.
[0104] (4) High computing power utilization and low memory access pressure: Multi-scenario micro-core cluster customization and optimization, combined with AI acceleration unit to achieve hard core acceleration, computing power utilization is more than 30% higher than traditional architecture; near-core TCM storage and dedicated local memory independent memory access port effectively alleviate memory wall problem.
[0105] (5) Clear protection boundaries and strong protection: There is no overlap in the protection boundaries. All three core innovations are dedicated hardware designs for AI inference, constructing a complete AI computing protection matrix to support industrialization.
[0106] In summary, this invention provides an AI inference acceleration system based on a Unified Computing Architecture (UCS). It includes a basic framework layer providing general hardware support and fully hardwired control logic, including a global data packet multi-queue engine and a DMA acceleration engine; an AI-dedicated computing layer serving as the carrier for AI inference and local message interaction, comprising a unified computing microcore cluster (UCC) consisting of multiple RISC-V microcores with RVV extension capabilities and an internal message engine within the UCC; an AI-specific interface adaptation layer providing a dedicated transmission path for AI data, achieving isolation and verification between data streams and message streams, and connecting to global and local message paths; and an AI hardware acceleration unit integrated within each UCC cluster, dedicated to accelerating AI operator hardware and receiving scheduling instructions from the UCC's internal message engine. This invention achieves high computing power utilization and low-latency AI inference acceleration through hardware collaboration at each layer, and possesses stronger scalability and scenario adaptability.
[0107] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An AI inference acceleration system based on a unified computing architecture (UCS), characterized in that, include: Basic framework layer: used to provide general hardware support, with all control logic hardwired, including a global data packet multi-queue engine and a DMA acceleration engine; AI-dedicated computing layer: includes multiple unified computing microcore clusters (UCCs). Each UCC consists of multiple RISC-V microcores with RVV extension features and an internal message engine, serving as a carrier for AI inference and local message interaction. AI-specific interface adaptation layer: used to provide a dedicated transmission path for AI data, realize the isolation and verification of data stream and message stream, and connect to global and local message paths; AI Hardware Acceleration Unit: Fully integrated within each of the Unified Computing Microcore Clusters (UCCs), dedicated to accelerating AI operator hardware cores and receiving scheduling instructions from the UCC's internal message engine.
2. The AI inference acceleration system based on Unified Computing Architecture (UCS) according to claim 1, characterized in that, The data packet multi-queue engine in the basic framework layer is equipped with a dedicated AI broadcast / multicast message pipeline: the dedicated AI broadcast / multicast message pipeline includes 4 independent multicast / broadcast dedicated queues, which are physically isolated from the ordinary AI task queues and independently arbitrated; the multicast / broadcast dedicated queues are configured with a 512-level FIFO buffer to carry multiple UCC global synchronization instructions; the ordinary AI task queues are configured with a 256-level FIFO buffer to carry point-to-point inference task messages.
3. The AI inference acceleration system based on the Unified Computing Architecture (UCS) according to claim 2, characterized in that, The data packet multi-queue engine includes a hardwired replication circuit: the hardwired replication circuit is used to generate multiple message copies in real time based on the 16-bit target UCC bitmap field in the received unified 32-byte fixed-length message, and distribute the global configuration information synchronously to multiple specified target unified computing microcore clusters UCC without real-time software overhead through the AI-dedicated broadcast / multicast dedicated message pipeline.
4. The AI inference acceleration system based on the Unified Computing Architecture (UCS) according to claim 2, characterized in that, The data packet multi-queue engine employs a four-level hard-wired priority congestion control mechanism optimized for AI hybrid inference. The priority attributes of different message queues in this mechanism can be configured according to scenario requirements. In AI inference scenarios, the four-level hard-wired priority congestion control mechanism includes: priority 0 for global broadcast / multicast synchronization messages, priority 1 for cross-scenario direct-access pipeline messages, and priorities 2 and 3 for point-to-point messages and batch inference messages. When the queue occupancy rate reaches the statically configured threshold, low-priority message enqueueing is automatically paused; when the queue is full, the lowest priority message is discarded.
5. The AI inference acceleration system based on the Unified Computing Architecture (UCS) according to claim 1, characterized in that, The AI hardware acceleration unit adopts a tensor computing core TCU: the tensor computing core TCU includes a configurable multiply-accumulate MAC array, and 1 to 8 independent multiply-accumulate MAC arrays are statically configured in a single unified computing micro-core cluster UCC; the hardware of the multiply-accumulate MAC array is completely fixed, only the operation parameters are programmable, and the number of arrays cannot be dynamically adjusted during operation.
6. The AI inference acceleration system based on the Unified Computing Architecture (UCS) according to claim 5, characterized in that, The tensor computing core TCU is equipped with dedicated high-speed on-chip storage: the high-speed on-chip storage is physically independent of the shared data RAM of the unified computing microcore cluster UCC; The high-speed on-chip storage is directly read and written by the Tensor Computation Core (TCU) and the internal message engine of the UCC; the RISC-V microcore can only indirectly read data in the high-speed on-chip storage through a dedicated read-only channel for task verification, and cannot directly write computation data, thereby isolating the core memory access and computation path between the microcore and the TCU at the hardware level.
7. The AI inference acceleration system based on the Unified Computing Architecture (UCS) according to claim 6, characterized in that, The AI-specific interface adaptation layer includes a unified cross switch within the Unified Computing Microcore Cluster (UCC): the unified cross switch enables non-blocking interconnection of the Tensor Computing Core (TCU), the RISC-V microcore, the shared instruction RAM, and the shared data RAM, supporting multiple microcores to simultaneously issue task instructions to the Tensor Computing Core (TCU); data migration and TCU data preloading in AI scenarios are completed by the local DMA of the message engine within the UCC to hide communication latency; while the DMA acceleration engine of the basic framework layer is only responsible for cross-UCC message pass-through and addressing, and does not participate in AI preprocessing.
8. The AI inference acceleration system based on the Unified Computing Architecture (UCS) according to claim 1, characterized in that, The microcore cluster configuration within each Unified Computing Microcore Cluster (UCC) in the AI-dedicated computing layer includes: each UCC is configured with 4 to 8 RISC-V microcores, supporting SIMD+MIMD hybrid hard-wired scheduling; each RISC-V microcore is configured with a dedicated 8KB tightly coupled memory (TCM), and each RISC-V microcore is independently addressed without shared cache; the interconnect network adopts a cacheless consistency protocol design.
9. The AI inference acceleration system based on the Unified Computing Architecture (UCS) according to claim 1, characterized in that, The multiple Unified Computing Microcore Clusters (UCCs) in the AI inference acceleration system work together with the packet multi-queue engine through a Mesh network to support heterogeneous hybrid inference scenarios: the Unified Computing Microcore Clusters (UCCs) for different application scenarios are statically mapped; wherein, for intra-cluster communication requiring high bandwidth, the Mesh network high-bandwidth path is fixed; for small-batch communication or cross-group cross-scenario communication requiring low latency, the Mesh network is bypassed, and the cross-scenario direct-through pipeline or unicast queue of the packet multi-queue engine is used for physical isolation transmission.
10. The AI inference acceleration system based on the Unified Computing Architecture (UCS) according to claim 9, characterized in that, In the cross-scenario direct communication pipeline, an asynchronous interactive confirmation mechanism is implemented by combining the message ID field in the unified 32-byte message: message synchronization and response confirmation between the message generating end and the consumer end are completely completed by the two ends themselves.
