AI processor based on storage and calculation integration, three-dimensional integration and PCIe remapping

Abstract

The invention relates to an AI processor based on storage and calculation integration, three-dimensional integration and PCIe remapping, and the AI processor comprises at least one storage unit which comprises a storage circuit composed of a plurality of storage arrays and is configured to be used for storing input feature data of an input neural network model, neural network weight data, feature data and an intermediate calculation result; the at least one calculation unit is in communication coupling with the at least one storage unit through a three-dimensional integration technology, and the calculation unit comprises a neural network processing unit based on a storage and calculation integrated array and is configured to be used for executing neural network correlation calculation by using data of the storage unit; and the PCIe interface device is at least compatible with one PCIe memory standard protocol and is configured to be capable of analyzing the information of the head of the transaction layer data packet and remapping the information into a calculation instruction which can be identified by the calculation unit based on a preset remapping mechanism. Through a remapping mechanism of a PCIe standard interface, neural network calculation with low overhead, high integration level and high efficiency is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an AI processor based on in-memory computing, three-dimensional integration, and PCIe remapping. Background Technology

[0002] With the rapid development of artificial intelligence technology, computational models represented by deep neural networks (especially large models based on the Transformer structure, such as Llama2-7B) have been widely used in fields such as image recognition and natural language processing. The self-attention mechanism and encoder-decoder structure of these models place extremely high demands on the computing power and data bandwidth of the hardware—a single large model inference may require trillions of multiply-accumulate operations, a challenge that traditional hardware architectures struggle to meet.

[0003] Traditional Feng Because the Von Neumann architecture processors physically separate computing and storage units, data needs to be frequently moved via the bus, creating a "memory wall" bottleneck due to limitations in bus bandwidth. For neural network computing, the movement of large amounts of weight and feature data not only consumes energy but also severely limits hardware performance, making computing density and bandwidth co-constraining factors.

[0004] To overcome these bottlenecks, Compute-in-Memory (CIM) technology has emerged. Its core is the integration of memory and processing unit, pre-storing weighted data in a storage array. After the input feature vector is broadcast to the array, matrix-vector multiplication is performed directly within the storage, fundamentally reducing data movement and improving computing density and energy efficiency (TOPS / W). However, CIM technology still requires external storage to provide feature data; insufficient inter-chip communication bandwidth will still limit the overall system performance.

[0005] 3D integration technology achieves vertical chip stacking and interconnection through through-silicon vias (TSVs) and hybrid bonding, offering advantages such as short paths and high interconnect density. It provides bandwidth far exceeding that of traditional 2D packaging, effectively alleviating the "memory wall" between chips—High-bandwidth memory (HBM) is a commercial prime example of this technology, providing ample bandwidth support for high-performance computing. However, 3D integration only solves the "data transmission pipeline" problem; if the computing engine remains an inefficient architecture, the value of the high-bandwidth link will be wasted.

[0006] Therefore, the collaborative architecture of in-memory computing technology and 3D integration technology complement each other. By simultaneously applying in-memory computing technology and 3D integration technology, the bottlenecks of traditional architectures can be broken through, and the performance of chip systems can be greatly improved.

[0007] However, interface interconnect flexibility is another core pain point: neural network processors need to communicate with external devices (such as CPUs, GPUs, and mobile phone main chips) through interfaces to form a complete system, but existing solutions mostly use custom communication interfaces. Such interfaces require co-design of interconnected devices, which greatly limits the choice of devices—for example, an AI accelerator with a custom interface can only be adapted to the main control chip of a specific manufacturer and cannot be compatible with mainstream GPUs or mobile phone main chips, resulting in poor system integration flexibility and difficulty in optimizing overall efficiency.

[0008] Therefore, it is necessary to improve existing neural network processors. Summary of the Invention

[0009] To address the above issues, this application provides an AI processor (hereinafter also referred to as a neural network processor) based on in-memory computing, 3D integration, and PCIe remapping, comprising: At least one storage unit, including a storage circuit composed of multiple storage arrays, is configured to store input feature data, neural network weight data, feature data, and intermediate calculation results of an input neural network model; At least one computing unit is communicatively coupled to the at least one storage unit via three-dimensional integration technology. The computing unit includes a neural network processing unit based on a memory-computing array and is configured to perform neural network-related calculations using data from the storage unit. The PCIe interface device is compatible with at least one PCIe memory standard protocol. The PCIe can parse the information in the transaction layer data packet header and remap it into computing instructions that the computing unit can recognize based on a preset remapping mechanism.

[0010] Optionally, the PCIe interface device is configured to: declare a contiguous reserved address space as an instruction space through the base address register of the PCIe configuration space; monitor the received transaction layer data packets conforming to the PCIe specification in real time, and compare the address field in the header of the transaction layer data packet with the address range of the instruction space; when it is detected that the address field in the header of the transaction layer data packet falls within the instruction space, intercept the transaction layer data packet, and extract the content of its data segment as a calculation control instruction.

[0011] Optionally, the computing unit further includes an instruction parsing unit, wherein the PCIe interface device is configured to send the extracted computing control instructions to the instruction parsing unit, and the instruction parsing unit is used to parse control information including instruction type, computing parameters and data address, and the control information is used to drive the computing unit of the AI ​​processor to perform corresponding neural network operations.

[0012] Optionally, the PCIe interface device is further configured to: monitor the format field and type field in the header of the transaction layer data packet; when the format field is detected to be a first preset value and the type field is a second preset value, identify the monitored transaction layer data packet as a custom computation control instruction; wherein, the combination of the first preset value and the second preset value belongs to an undefined field of the PCIe standard protocol, and the number of second preset values ​​is multiple, with different second preset values ​​respectively mapped to computation task issuance instructions, computation status query instructions, computation result reading instructions, and device configuration instructions.

[0013] Optionally, the PCIe interface device is further configured to: extract calculation parameters from the custom control parameter area of ​​the transaction layer data packet header; the calculation parameters include: calculation type identifier, calculation precision, matrix dimension information, physical address of input data in the storage unit, and output data address; the monitoring transaction layer data packet header accommodates the custom control parameter area by extending the bit width.

[0014] Optionally, the protocol remapping mechanism includes: mapping the first stored data to computation instructions that the computing unit can recognize, wherein the first stored data is stored data read and written to a storage circuit in a specific address space of the storage unit.

[0015] By mapping memory access operations to a specific address space to instructions executed by the computing unit, a computing mechanism triggered by data access is implemented, reducing master control interaction latency and improving computing response speed.

[0016] Optionally, the protocol remapping mechanism includes: defining reserved fields in the standard PCIe instruction set as instructions that the computing unit can recognize; or reusing some instructions from the standard PCIe as synchronous instructions that the computing unit can recognize.

[0017] Implement instruction extensions that are compatible with the PCIe standard, enabling computing control to be embedded in existing protocol architectures without modifying external host hardware.

[0018] Optionally, the PCIe interface device is used to monitor the format field (fmt) and type field (type) in the transaction layer packet (TLP) header; when the format field (fmt) is detected to be a first preset value and the type field (type) is detected to be a second preset value, the monitored transaction layer packet is identified as a custom computation control instruction; wherein, the combination of the first preset value and the second preset value belongs to an undefined field of the PCIe standard protocol, and there are multiple second preset values; different second preset values ​​are respectively mapped to computation task issuance instructions, computation status query instructions, computation result reading instructions, and device configuration instructions.

[0019] This application utilizes the "Reserved" fields (e.g., fmt=11 and its type combinations) explicitly defined in the PCIe standard specification to embed custom instructions without modifying the standard PCIe physical layer and link layer hardware. This allows standard PCIe switches to forward these packets normally, while only this AI processor will recognize them as special instructions, greatly improving system integration compatibility.

[0020] Optionally, the PCIe interface device is further configured to: extract computation parameters from the custom control parameter area of ​​the transaction layer packet header, the computation parameters including: computation type identifier, computation precision, matrix dimension information, physical address of input data in the storage unit, and output data address; the monitoring transaction layer packet header accommodates the custom control parameter area by extending the bit width. This application, by directly embedding key computation parameters (such as matrix dimension and address) into the TLP header instead of placing them in the data payload, enables the PCIe interface device to obtain complete control information within the first clock cycle of parsing the packet header, without waiting for subsequent data transmission to complete, significantly reducing the control latency from instruction issuance to computation initiation.

[0021] Optionally, the "specific address space" is a reserved non-storage area declared through the Base Address Register (BAR) in the PCIe configuration space; the PCIe interface device includes an address matching and instruction parsing module, configured to: compare the address field in the header of the received monitoring transaction layer data packet in real time; when the address field is detected to fall within the specific address space range, intercept the monitoring transaction layer data packet and prohibit writing data to the storage circuit; and extract the data segment content of the monitoring transaction layer data packet and parse it into control signals to drive the computing unit.

[0022] This application declares via the BAR register that the operating system kernel can directly map the "instruction space" to virtual memory addresses. Host-side drivers only need to perform standard memory write operations to issue instructions, without needing to go through a complex I / O instruction stack. Simultaneously, a hardware-level address interception mechanism ensures that instruction data will not erroneously overwrite the actual data in the memory location, guaranteeing high system reliability.

[0023] Optionally, the neural network processing unit based on the in-memory computing array is configured as follows: Receives input feature data from the neural network and weight data from the storage unit; Perform neural network calculations based on the weight data and the input feature data; and The calculation results are output to the storage unit or external device.

[0024] The computing unit can directly read weight and feature data from the storage layer, perform core calculations for neural network inference or training, and output the results directly to storage or an external host, forming a complete neural network computing closed loop.

[0025] Optionally, the PCIe interface device is compatible with the PCIe memory standard protocol and can be recognized as a standard storage device and accessed in memory format by the host system or a memory controller that supports the PCIe standard.

[0026] By making the PCIe interface compatible with the standard PCIe storage protocol, it can be recognized by the host as a standard storage device, improving the system's plug-and-play capability and ecosystem compatibility, and facilitating deployment in general-purpose servers or edge systems.

[0027] Optionally, the three-dimensional integration technology includes at least one of the following: through-silicon via (TSV) technology, flip-chip technology, hybrid bonding technology, and microbump connection technology.

[0028] By selecting different three-dimensional integration processes (such as TSV, hybrid bonding, etc.), high-bandwidth, low-latency vertical interconnect channels are formed between the storage, computing and transmission layers, improving inter-layer communication speed and energy efficiency.

[0029] Optionally, the PCIe memory standard protocol supported by the PCIe interface device is selected from at least one of PCIe 4.0, PCIe 5.0, PCIe 6.0, and PCIe 7.0 to adapt to different bandwidths.

[0030] By supporting multiple protocol standards such as PCIe 4.0 to 7.0, it adapts to application scenarios with different bandwidth requirements, ensuring that the neural network processor has scalability and forward compatibility across different platforms.

[0031] Optionally, the storage unit and the computing unit can be stacked interchangeably to adapt to different heat dissipation and stress requirements.

[0032] By allowing the stacking order of the storage and compute layers to vary, it flexibly adapts to different heat dissipation and mechanical stress requirements, thereby improving the freedom of packaging design and thermal management performance. Optionally, it also includes: An interface verification circuit is used to detect and correct errors in instructions and data information exchanged between the PCIe interface device and the external main control chip, to prevent erroneous data from entering the storage unit or the computing unit; and An on-chip verification circuit is used to detect and correct errors in the data stored in the memory unit and the intermediate calculation results in the computing unit, so as to ensure the reliability of data transmission and calculation within the neural network processor. Error detection and correction in cross-layer data transmission and calculation results are achieved through interface verification and on-chip verification circuitry, ensuring the reliability of the neural network data processing process and the system stability.

[0033] Optionally, if the PCIe interface device is located in the storage unit, then the interface verification circuit and the on-chip verification circuit are located in the storage unit. When the PCIe interface is located in the storage layer, the verification circuit is co-located with the storage layer, which can shorten the verification path and reduce latency, thereby improving transmission reliability and interface layer efficiency.

[0034] Optionally, if the PCIe interface device is located in the computing unit, the interface verification circuit is located in the computing unit, and the on-chip verification circuit is located in the storage unit or the computing unit. When the PCIe interface is located in the computing layer, the interface verification circuit can directly monitor the data exchanged between the computing layer and the external host, improving the transmission correctness and data consistency of the critical path.

[0035] Optionally, the in-memory computing array is implemented based on at least one of SRAM, ReRAM, MRAM, or FeFET technologies. By supporting multiple in-memory computing implementation technologies (SRAM, ReRAM, MRAM, FeFET, etc.), the implementation forms and process compatibility of computing units are expanded, improving design flexibility and process adaptability.

[0036] Optionally, the computation results output by the neural network processing unit based on the in-memory computing array are transmitted through at least one of the following paths: directly written to a designated storage area of ​​the storage unit, transmitted to an external main control chip through the PCIe interface device, or transmitted to other extended storage units through a three-dimensional integrated connection structure.

[0037] This application provides a multi-path result output mechanism, supporting write-back to storage, external transmission to the host, or transmission to an extension module, thereby improving the flexibility of computation result scheduling and system parallelism.

[0038] Optionally, the "reusing some instructions from the standard PCIe" specifically involves selecting low-frequency or reserved instructions from the PCIe memory standard protocol and redefining them as control instructions for the computing unit. This allows the instruction to simultaneously possess both the original memory unit control function and the computing unit control function. By redefining low-frequency or reserved instructions in the PCIe protocol, instruction reuse and expansion are achieved, enabling the PCIe interface to perform both memory access and computing control functions, thus reducing protocol modification costs.

[0039] Optionally, the neural network processor further includes a transmission unit, which is stacked with the storage unit and the computing unit using a three-dimensional integration technique, and the PCIe interface device is disposed within the transmission unit. By introducing an independent transmission layer, the PCIe interface and protocol processing are separated from the computing / storage layer, reducing the load on the main computing path and improving system modularity and scalability.

[0040] Optionally, a PCIe protocol parser is located in the transport unit. Deploying the PCIe protocol parser at the transport layer allows protocol parsing and inter-layer data interaction to be completed at the same layer, reducing cross-layer communication latency and improving interface processing efficiency.

[0041] Optionally, it also includes a data buffer module for temporarily storing data transmitted between the external main control chip and the storage and computing units, avoiding congestion during data transmission and improving the stability of data interaction. By temporarily storing data between the host and the chip, the data buffer module prevents bus congestion and sudden data loss, thereby improving system throughput and communication stability.

[0042] Optionally, the instruction parsing unit can be implemented as a fully custom circuit, a CPU, a RISC-V instruction set, or an extended instruction parser thereof. By using a RISC-V or custom CPU as the instruction parsing core, instruction control becomes programmable and expandable, facilitating subsequent protocol upgrades or custom operator expansions.

[0043] The neural network processor provided in this application achieves "computing power-bandwidth-interface" collaboration, thereby releasing system performance.

[0044] In-memory computing provides high computing power density (reducing data movement), 3D integration provides high-bandwidth inter-layer transmission (matching CIM computing power requirements), and PCIe interface provides universal high-bandwidth external interconnection to avoid interface bottlenecks.

[0045] Enhance connectivity flexibility and reduce integration costs: The PCIe interface is a universal standard in the market. Most devices on the market (CPU, GPU, mobile phone main chip) have a PCIe interface. External main control chip can recognize this device as a standard PCIe device (such as PCIe accelerator card or PCIe memory) without custom hardware or drivers, realizing "plug and play". For example, if a mobile phone manufacturer adopts this device, it can be directly integrated without modifying the PCIe controller of MediaTek main chip, shortening the integration cycle by 60% and reducing the cost by 40%.

[0046] Adaptable to various scenarios and highly scalable: Scenario adaptation: By adjusting the PCIe version (PCIe 5.0 for mobile devices and PCIe 7.0 for servers), the number of storage layers (2 layers for edge devices and 4 layers for data centers), and the stacking order (heat dissipation priority or thin and light priority), it can cover multiple scenarios such as smartphones, IoT terminals, PCs, and high-performance computing devices. Process upgrade: When CIM process nodes advance, only the number of storage layers needs to be increased to match the computing power, without modifying the overall architecture. This reduces chip iteration costs by 50% and shortens version iteration cycles by 30%.

[0047] Optimize transmission efficiency and reduce power consumption and latency: The optional transport layer centrally handles PCIe protocol conversion and data buffering, avoiding additional protocol overhead for storage / computing units; the three-dimensional integrated vertical connection shortens the transmission path between the PCIe interface and the storage-computing unit, reducing latency by 60% and power consumption by 35% compared to traditional two-dimensional packaged PCIe interfaces. For example, in a certain end-side device, PCIe data transmission latency was reduced from 50ns to 20ns, and standby power consumption was reduced from 1.2W to 0.8W.

[0048] This application uses a general PCIe interface, without relying on a specific manufacturer's custom interface ecosystem (such as Qualcomm's proprietary interconnect protocol); at the same time, CIM technology breaks through the computing power density limitation of three-dimensional stacking, providing an independent and controllable technical path for high-end neural network processors. Attached Figure Description

[0049] Figure 1 A schematic diagram of the structure of a neural network processor provided for an embodiment of this application; Figure 2 This is a schematic diagram of the neural network processor in another embodiment of this application; Figure 3 This is a schematic diagram of the neural network processor in another embodiment of this application; Figure 4 This is a schematic diagram of the structure of a neural network processor when an instruction space is provided in the embodiments of this application; Figure 5 Here is a schematic diagram of the neural network processor structure when a transmission unit is provided in the embodiments of this application: Figure 6 This is a schematic diagram of the neural network processor in another embodiment of this application; Figure 7 This is a schematic diagram of the structure of the neural network processing unit in the embodiments of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] like Figure 1 As shown, this embodiment provides an AI processor based on in-memory computing, three-dimensional integration, and PCIe remapping, including: at least one storage unit 100 for storing input feature data, neural network weight data, feature data, intermediate calculation results, and some system data of the input neural network model; at least one computing unit 200, which is communicatively coupled to at least one storage unit 100 through three-dimensional integration technology; and a PCIe interface device 300, which is compatible with at least one PCIe memory standard protocol and configured to parse the information in the transaction layer data packet header and remap it into computing instructions recognizable by the computing unit 200 based on a preset remapping mechanism.

[0052] In some implementations, such as Figure 2 As shown, the PCIe interface device 300 can also be located in the computing unit 200, which will not be described in detail here.

[0053] Optionally, such as Figure 3 As shown, the computing unit 200 includes a neural network processing unit 210 based on a memory-computing array, which is configured to perform neural network-related calculations using data from the storage unit 100.

[0054] Optionally, such as Figure 7 As shown, the specific structure of the neural network processing unit 210 mainly includes a DCIM array based on 6T SRAM, a word line decoder (WL DEC), a word line buffer (WL DRV), read / write circuitry, and a controller. The DCIM array consists of C DCIM columns, each containing R subarrays and an adder tree. Each subarray integrates 6T SRAM bit units and a Local Readout and Computation Unit (LRCC), storing N bits of weight per row. Functionally, the DCIM macro can perform matrix-vector multiplication (MVM) multiplication-accumulation (MAC) calculations: the operating mode is switched via the MEM_EN signal. When MEM_EN is low, the N bits of weight in the subarray are multiplied by a 1-bit input, read and latched by the LRCC, accumulated by the adder tree, and then sent to a shift-and-accumulator to extend the input precision in the time dimension, ultimately completing a full MVM operation with K-bit input and N-bit weights, while also supporting the storage and efficient computation of multi-weight matrices.

[0055] For interface devices conforming to the PCIe protocol specification, in traditional memory application scenarios, their functions are limited to controlling memory chips (i.e., memory units 100 mentioned in this embodiment). However, this embodiment, through an innovatively designed PCIe interface device control mechanism, enables the PCIe interface device to not only achieve conventional control of memory units after receiving external instructions and data from external host devices, but also to flexibly regulate the computing behavior of computing units. This breaks through the functional limitations of traditional interface devices and solves the problems of low computing power density, limited bandwidth, and poor interface versatility of existing neural network processors.

[0056] Specifically, the external control device can be a host device that uses a CPU, GPU, or NPU.

[0057] Optionally, when the computing unit 200 is started, the "feature data" stored in the storage unit 100 includes the input feature map data and output feature map data of each layer of the neural network; the "intermediate calculation results" include the partial accumulation results after matrix-vector multiplication operations and the intermediate data before activation function processing.

[0058] Optionally, external commands can be issued by a main control chip connected to the neural network processing unit 210.

[0059] The PCIe interface device 300 is used to: declare a continuous reserved address space as an instruction space through the base address register of the PCIe configuration space; and monitor the received transaction layer data packets conforming to the PCIe specification in real time, and compare the address field in the header of the transaction layer data packet with the address range of the instruction space; when it is detected that the address field in the header of the transaction layer data packet falls into the instruction space, intercept the transaction layer data packet, and extract the content of its data segment as a calculation control instruction.

[0060] Optionally, such as Figure 3 As shown, the computing unit 200 also includes an instruction parsing unit 220. After the computing control instruction is sent to the instruction parsing unit 220, the instruction parsing unit 220 parses the control information, including the instruction type, computing parameters and data address. The control information is used to drive the computing unit 200 of the AI ​​processor to execute the corresponding neural network operation.

[0061] Optionally, the PCIe interface device 300 is also used to: receive memory cell access instructions from an external host control chip; parse the instructions and control the computing unit 200 to perform neural network operations, or control the data read / write of the memory unit 100; realize the multiplexing of storage instructions and computing instructions based on a preset protocol remapping mechanism; or as follows: Figure 4As shown, data is transmitted to an instruction space located in the storage circuit via the PCIe interface device 300. The computing unit 200 reads instructions from the instruction space and parses the instructions. After parsing the instructions, it can trigger the computing layer to perform neural network operations or control the data reading and writing of the storage layer, realizing the integration of "data transmission - computing control".

[0062] Optionally, the protocol remapping mechanism includes: mapping the first stored data to a computation instruction that the computing unit 200 can recognize, wherein the first stored data is stored data read and written to the storage circuit 110 in a specific address space of the storage unit 100.

[0063] For example, predefined regions in the PCIe address space (such as 0x1000-0x2000) are mapped to computing layer control addresses. When an external device sends a "write TLP" to this region, the data is automatically used as computing layer instructions (such as NPU start signal, neural network layer configuration). The computing layer instruction parser accesses this address and starts the corresponding computing task.

[0064] Optionally, the protocol remapping mechanism includes: defining reserved fields in the standard PCIe instruction set as instructions that the computing unit 200 can recognize; or reusing some instructions from the standard PCIe as synchronous instructions that the computing unit 200 can recognize.

[0065] For example, reserved fields of PCIe TLP (such as the 2 free bits in the "attribute field") can be reused to define computation control functions (such as 01 for "start computation", 10 for "pause computation", and 11 for "read intermediate results"), thereby realizing the dual function of PCIe instructions: "data transfer + computation control".

[0066] Optionally, “reusing some instructions from the standard PCIe” specifically means: selecting instructions or reserved instructions that are used less frequently in the PCIe memory standard protocol, redefining them as control instructions for the computing unit 200, so that the instructions have both the original control functions of the storage unit 100 and the control functions of the computing unit 200.

[0067] Optionally, the PCIe interface device 300 is configured to monitor the format field (fmt) and type field (type) in the transaction layer packet (TLP) header; when the format field (fmt) is detected to be a first preset value and the type field (type) is detected to be a second preset value, the monitored transaction layer packet is identified as a custom computation control command; wherein, the combination of the first preset value and the second preset value belongs to an undefined field of the PCIe standard protocol, and there are multiple second preset values, which are respectively mapped to computation task issuance commands, computation status query commands, computation result reading commands, and device configuration commands.

[0068] This implementation utilizes the "Reserved" fields (e.g., fmt=11 and its type combination) explicitly defined in the PCIe standard specification to embed custom instructions without modifying the standard PCIe physical layer and link layer hardware. This allows standard PCIe switches to forward these packets normally, while only this AI processor will recognize them as special instructions, greatly improving system integration compatibility.

[0069] Optionally, when a command is issued for a computation task, the PCIe interface device 300 is configured to extract computation parameters from the custom control parameter area in the header of the monitoring transaction layer data packet. The header of the monitoring transaction layer data packet accommodates the custom control parameter area by extending the bit width. The computation parameters include: computation type identifier, computation precision, matrix dimension information, physical address of input data in the storage unit, and output data address.

[0070] This implementation embeds key computational parameters (such as matrix dimensions and addresses) directly into the TLP header instead of placing them in the data payload. This allows the PCIe interface device to obtain complete control information within the first clock cycle of parsing the packet header, without waiting for subsequent data transmission to complete, significantly reducing the control latency from instruction issuance to computation initiation.

[0071] Optionally, the PCIe interface device 300 is configured to declare a contiguous reserved address space as an instruction space through the base address register of the PCIe configuration space; monitor the received transaction layer data packets conforming to the PCIe specification in real time, and compare the address field in the header of the transaction layer data packet with the address range of the instruction space; when it is detected that the address field in the header of the transaction layer data packet falls within the instruction space, the transaction layer data packet is intercepted, and the content of its data segment is extracted as a calculation control instruction.

[0072] This implementation uses the BAR register declaration, allowing the operating system kernel to directly map the "instruction space" to virtual memory addresses. Host-side drivers only need to perform standard memory write operations to issue instructions, eliminating the need for complex I / O instruction stacks. Simultaneously, a hardware-level address interception mechanism ensures that instruction data does not erroneously overwrite actual data in memory locations, guaranteeing high system reliability.

[0073] The following explains the implementation of PCIe instruction remapping. The core carrier of PCIe TLP data transmission is the TLP packet, whose header contains multiple functional fields. Among them, the fmt and type fields are key to distinguishing TLP types and functions. Taking a common 32-bit TLP header as an example, the fmt field occupies 2 bits and is mainly used to distinguish the basic format type of the TLP; the type field occupies 5 bits and further subdivides the specific functions of the TLP based on the fmt field, such as memory read / write requests, I / O requests, etc. In addition to these two core fields, the header also includes a 3-bit Traffic Class (TC) field, a 1-bit Tag (TD) bit, a 1-bit Error Handling (EP) bit, a 3-bit Attr bit, a 10-bit Length field, and a 16-bit Requester ID (using BDF encoding to identify the device initiating the TLP request), as well as other auxiliary fields. According to the PCIe standard specification, some combinations of fmt and type have been defined for standard functions, while some combinations remain undefined. These undefined combinations provide usable space for custom device control.

[0074] To achieve precise computational control of the AI ​​processor, this implementation selects the fmt=11 (binary) format segment (i.e., the first preset value) which is not fully defined in the PCIe standard. Under this format segment, there are multiple type reserved values ​​that are not occupied by the standard (i.e., the second preset value). Based on this, the mapping definition of the core control function of the 3D-CIM device and the fmt+type combination is completed.

[0075] In this embodiment, the AI ​​processor is equipped with a PCIe interface device 300 capable of parsing TLP packets to recognize and extract parameters from custom TLPs. Under the control of clock and reset signals, the parsing module receives the transmitted 64-bit TLP header data. It first extracts the `fmt` field from the highest two bits of the header and the `type` field from the next five bits. It identifies the computation task instruction by determining whether these two fields match a custom combination (`fmt=11`, `type=11000`). When a matching TLP is detected, the module enables the CIM calculation function and extracts key information such as the corresponding calculation operation type, input data address, and output data address from the parameter area of ​​the TLP header. This information is then transmitted to the calculation core of the 3D-CIM device, triggering the start of the calculation process. If no matching `fmt+type` combination is detected, the calculation enable signal remains off, ensuring that the device only responds to valid custom control commands.

[0076] The remapping scheme provided in this implementation has significant advantages: First, it has strong compatibility, being developed based on the fmt+type combination retained by the PCIe standard, which will not interfere with the normal interaction of the existing standard TLP protocol and can seamlessly adapt to the mainstream PCIe ecosystem; second, it is lightweight, directly reusing the native header fields of PCIe TLP without the need for a new dedicated hardware bus or protocol stack, greatly reducing the design complexity and hardware cost of the AI ​​processor; third, it has good scalability, allowing for further expansion of control functions based on the remaining undefined type values, such as adding instructions for power consumption adjustment, data verification, and firmware upgrades, to meet the needs of subsequent device function iterations; and fourth, it has low transmission latency, relying on the native transmission mechanism of PCIe TLP, reducing multiple layers of protocol parsing compared to a custom protocol stack, and can adapt to the real-time computing needs of the AI ​​processor.

[0077] The following describes the implementation of PCIe address space remapping. The core carrier of PCIe TLP data transmission is the TLP packet, whose header contains multiple functional fields. Among them, the address field is crucial for locating the data transmission target. Taking a common 32-bit TLP header as an example, the address field occupies 4 bytes (offset 4-7 bytes) and is used to identify the target address for data writing / reading. If a larger address space is required, a 64-bit extended TLP header can be used, in which case the address field is extended to 8 bytes (offset 4-11 bytes). In addition to the address field, the TLP header also includes a 2-bit format field (fmt, used only to distinguish basic formats such as address width), a 3-bit traffic level field (TC), a 1-bit flag bit (TD), a 1-bit error handling bit (EP), a 3-bit attribute bit (Attr), a 10-bit data length field (Length), and a 16-bit requester ID (using BDF encoding to identify the device initiating the request), among other auxiliary fields. This implementation does not rely on any custom extensions of fields; it only utilizes the native address location function of TLP to achieve instruction mapping by specifying the address space.

[0078] First, a dedicated "instruction space" is allocated for the AI ​​processor. This space is a reserved contiguous address segment within the PCIe address space, declared and mapped using the BAR (Base Address Register) of the PCIe configuration space to ensure no conflict with the address spaces of other devices in the system. In this example, the 3D-CIM instruction space is defined as a contiguous address segment starting at address 0x80000000 and with a length of 0x00010000 (this can be flexibly adjusted according to the number of instructions). This address segment is specifically used to receive control instruction data transmitted via PCIe and does not handle ordinary data storage.

[0079] It's important to clarify that the address field in the TLP header carries the instruction space address: If a 32-bit TLP is used (suitable for small to medium-sized address spaces), the address field is located 4-7 bytes off the TLP header, directly storing the 32-bit physical address of the 3D-CIM instruction space; if a 64-bit TLP is used (suitable for larger address space expansion), the address field is extended to 8 bytes (offset 4-11 bytes) by extending the header, storing the complete 64-bit instruction space address. Once the address field of the TLP packet matches the instruction space range of the AI ​​processor, the AI ​​processor will recognize the data segment of the TLP packet as an instruction, not ordinary data.

[0080] Taking the issuance of a "matrix multiplication calculation instruction" to an AI processor as an example, the core of the TLP packet construction is to point the header address field to the corresponding offset address in the AI ​​processor's instruction space and encapsulate the instruction content in the data segment. Different calculation instructions are distinguished based on different offset addresses and data segment encapsulation of instruction content.

[0081] In this embodiment, the PCIe interface device 300 configured in the AI ​​processor can monitor TLP packets on the PCIe bus in real time and continuously compare the value of the address field in the header of each TLP packet with the address range of its own instruction space. When the address field of a TLP packet is detected to fall within the instruction space of the AI ​​processor, the module will immediately intercept the TLP packet, extract its data segment content, and send it to the instruction parsing unit; if the address field does not match, the TLP packet will be allowed to pass directly without any processing. The instruction parsing unit parses the extracted data according to a preset format, distinguishes the instruction type (calculation / memory access, etc.), and passes the corresponding parameters to the calculation unit of the AI ​​processor to trigger the corresponding workflow (such as matrix multiplication calculation); at the same time, the parsing module will return a response signal to the host indicating successful instruction reception to ensure the reliability of the interaction.

[0082] This implementation method, based on "specific address space mapping," achieves AI processor instruction control and boasts significant advantages: First, it offers exceptional compatibility, fully reusing the PCIe standard's memory write transfer mechanism without modifying any TLP header field definitions. It seamlessly adapts to all PCIe-compliant hosts and devices, eliminating concerns about conflicts with standard protocols. Second, it is simple to implement; neither software nor hardware design requires complex custom field parsing logic. The software driver only needs to handle address mapping and data writing, while the hardware only needs to implement address matching and instruction parsing, significantly reducing development difficulty and cost. Third, it offers excellent scalability. By expanding the number of address segments in the AI ​​processor's instruction space and adding various control instructions, only corresponding entries need to be added to the address mapping table, without modifying the core transmission logic. Fourth, it offers high reliability. Relying on PCIe's native address verification and error handling mechanisms, it effectively ensures the integrity and accuracy of instruction transmission, meeting the high-reliability computing requirements of AI processor devices. Optionally, the neural network processing unit 210 based on the in-memory computing array is configured to: receive input feature data of the neural network and weight data from the storage unit 100; perform neural network calculations based on the weight data and input feature data; and output the calculation results to the storage unit 100 or an external device.

[0083] Optionally, the PCIe interface device 300 is compatible with the PCIe memory standard protocol and can be recognized as a standard storage device and accessed in memory format by the host system or a memory controller that supports the PCIe standard.

[0084] Optionally, the three-dimensional integration technology includes at least one of the following: through-silicon via (TSV) technology, flip-chip technology, hybrid bonding technology, and microbump connection technology.

[0085] Optionally, the PCIe interface device 300 supports PCIe memory standard protocols selected from PCIe 4.0, PCIe 5.0, PCIe 6.0, and PCIe 7.0.

[0086] Optionally, the storage unit 100 and the computing unit 200 can be interchanged in stacking order to adapt to different heat dissipation and stress requirements. Optionally, the neural network processor further includes: an interface verification circuit (not shown) for detecting and correcting errors in instructions and data information interacting with an external main control chip through the PCIe interface device 300, so as to prevent erroneous data from entering the storage unit or computing unit inside the device; and an on-chip verification circuit (not shown) for detecting and correcting errors in the data stored inside the storage unit 100 and the intermediate calculation results in the computing unit 200, so as to ensure the reliability of data inside the neural network processor during cross-layer transmission and calculation. Optionally, if the PCIe interface device 300 is located in the storage unit 100, then the interface verification circuit and the on-chip verification circuit are located in the storage unit 100.

[0087] Optionally, the PCIe interface device 300 is disposed in the computing unit 100, the interface verification circuit is disposed in the computing unit 100, and the on-chip verification circuit is disposed in the storage unit 100 or the computing unit 200.

[0088] Optionally, the in-memory computing array is implemented based on at least one of the following technologies: SRAM (Static Random Access Memory), ReRAM (Resistive Random Access Memory), MRAM (Magnetic Random Access Memory), or FeFET (Ferroelectric Field Effect Transistor).

[0089] Optionally, the computation results output by the neural network processing unit 210 based on the in-memory computing array are transmitted through at least one of the following paths: directly written to a designated storage area of ​​the storage unit 100, transmitted to an external main control chip via the PCIe interface device 300, and transmitted to other extended storage units via a three-dimensional integrated connection structure.

[0090] Other extended storage units are extended storage modules that have the same function as storage unit 100 and are connected to storage unit 100 using three-dimensional integration technology.

[0091] Optionally, the neural network processor can be adapted to the needs of different application scenarios such as mobile devices (including smartphones and IoT terminals), PCs, servers, and high-performance computing devices by adjusting the protocol configuration, number of storage units, and number of computing units (NPUs) of the PCIe interface device 300.

[0092] Optionally, the design constraint that the area of ​​the computing unit 200 and the storage unit 100 is basically the same is used to adapt to the three-dimensional integrated stacking architecture, avoid the computing power bottleneck caused by the limited area of ​​the computing unit, and improve the computing power density per unit area.

[0093] Optionally, the PCIe interface device 300 is also equipped with a data buffer module (not shown) for temporarily storing data transmitted between the external main control chip and the storage unit 100 and the computing unit 200, so as to avoid congestion during data transmission and improve the stability of data interaction.

[0094] Optionally, such as Figure 5 As shown, the neural network processor also includes a transmission unit 500, which has some or all of the functions of the storage unit 100 or the computing unit 200. It is integrated with the storage unit 100 or the computing unit 200 through three-dimensional stacking technology. Specifically, it can be arranged adjacent to the storage unit 100 or the computing unit 200, or arranged between the storage unit 100 and the computing unit 200. This arrangement realizes the expansion of the capacity or computing power of the storage unit 100 or the computing unit 200.

[0095] Optionally, the PCIe interface device 300 is disposed in the transmission unit 500.

[0096] In this embodiment, the PCIe interface device 300 can be flexibly set in the storage unit 100, the computing unit 200 and the transmission unit 500. For example, in a server scenario, the PCIe interface is set in the transmission unit 500 to optimize protocol processing; in a mobile scenario, the PCIe interface is set in the storage unit 100 to reduce power consumption.

[0097] Optionally, the instruction parsing unit 220 can be implemented as a full-custom circuit, a central processing unit, a RISC-V instruction set, or an extended instruction parser. RISC-V (Reduced Instruction Set Computer V) is an open-source instruction set architecture. It employs the principles of reduced instruction set computing, featuring a simple, efficient, and modular design that supports various data widths (such as 32-bit, 64-bit, and 128-bit). RISC-V's openness and flexibility make it widely applicable in academic research, industry, and embedded systems, and it can meet a wide range of needs, from microcontrollers to supercomputers.

[0098] Optionally, such as Figure 6 As shown, in some embodiments, the PCIe interface device 300 can also be configured independently and connected to multiple packages formed by three-dimensional stacking of computing units 200 and storage units 100. The host connects to the multiple packages through the PCIe interface device 300 to control the multiple packages.

[0099] The technical solution of the present invention has now been described in conjunction with the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to the specific embodiments described above. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An AI processor based on in-memory computing, three-dimensional integration, and PCIe remapping, characterized in that, include: At least one storage unit includes a storage circuit composed of multiple storage arrays for storing input feature data, neural network weight data, feature data, and intermediate calculation results of the input neural network model; At least one computing unit is communicatively coupled to the at least one storage unit via three-dimensional integration technology. The computing unit includes a neural network processing unit based on a storage-computing array, used to perform neural network-related calculations using data from the storage unit. as well as The PCIe interface device is compatible with at least one PCIe memory standard protocol. The PCIe interface device is able to parse the information in the transaction layer data packet header and remap it into computing instructions that the computing unit can recognize based on a preset remapping mechanism.

2. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping as described in claim 1, characterized in that, The PCIe interface device is used for: A contiguous reserved address space is declared as the instruction space by using the base address register of the PCIe configuration space; Real-time monitoring of received PCIe-compliant transaction layer data packets, and comparison of the address field in the header of the transaction layer data packet with the address range of the instruction space; When it is detected that the address field in the header of the transaction layer data packet falls within the instruction space, the transaction layer data packet is intercepted, and the content of its data segment is extracted as a calculation control instruction.

3. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping as described in claim 2, characterized in that, The computing unit also includes an instruction parsing unit. The PCIe interface device is used to send the extracted computing control instructions to the instruction parsing unit. The instruction parsing unit is used to parse out control information including instruction type, calculation parameters and data address. The control information is used to drive the calculation unit of the AI ​​processor to perform the corresponding neural network operation.

4. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping as described in claim 1, characterized in that, The PCIe interface device is also used for: Monitor the format and type fields in the transaction layer data packet header; When the format field is detected to be a first preset value and the type field is detected to be a second preset value, the monitoring transaction layer data packet is identified as a custom calculation control instruction; The combination of the first preset value and the second preset value belongs to the undefined field of the PCIe standard protocol. There are multiple second preset values, and different second preset values ​​are respectively mapped to computing task issuance instructions, computing status query instructions, computing result reading instructions, and device configuration instructions.

5. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping as described in claim 1, characterized in that, The PCIe interface device is also used for: The calculation parameters are extracted from the custom control parameter area in the header of the transaction layer data packet. The calculation parameters include: calculation type identifier, calculation precision, matrix dimension information, physical address of input data in the storage unit, and output data address. The monitoring transaction layer data packet header accommodates the custom control parameter area by extending the bit width.

6. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping as described in claim 1, characterized in that, The protocol remapping mechanism includes: mapping the first stored data to computation instructions that the computing unit can recognize, wherein the first stored data is stored data read and written to the storage circuit of a specific address space of the storage unit.

7. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping as described in claim 1, characterized in that, The protocol remapping mechanism includes: Define reserved fields in the standard PCIe instruction set as instructions that the computing unit can recognize; or, Some instructions from the standard PCIe are reused and synchronized as instructions that the computing unit can recognize.

8. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, The neural network processing unit based on the in-memory computing array is used for: Receives input feature data from the neural network and weight data from the storage unit; Perform neural network calculations based on the weight data and the input feature data; The calculation results are output to the storage unit or external device.

9. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, The PCIe interface device is compatible with the PCIe memory standard protocol and can be recognized as a standard storage device by the host system or a memory controller that supports the PCIe standard and accessed in memory format.

10. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, The three-dimensional integration technology includes at least one of the following: through-silicon via (TSV) technology, flip-chip technology, hybrid bonding technology, and microbump connection technology.

11. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, The PCIe interface device supports at least one of PCIe memory standard protocols, namely PCIe 4.0, PCIe 5.0, PCIe 6.0, and PCIe 7.0, to adapt to different bandwidths.

12. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, The storage unit and the computing unit can be stacked interchangeably to adapt to different heat dissipation and stress requirements.

13. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, Also includes: An interface verification circuit is used to detect and correct errors in instructions and data information exchanged between the PCIe interface device and the external main control chip, so as to prevent erroneous data from entering the storage unit or the computing unit. as well as An on-chip verification circuit is used to detect and correct errors in the data stored in the storage unit and the intermediate calculation results in the computing unit, so as to ensure the reliability of data in the AI ​​processor during cross-layer transmission and calculation.

14. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping as described in claim 13, characterized in that, The PCIe interface device is disposed in the storage unit, and the interface verification circuit and the on-chip verification circuit are disposed in the storage unit.

15. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to claim 13, characterized in that, The PCIe interface device is disposed in the computing unit, the interface verification circuit is disposed in the computing unit, and the on-chip verification circuit is disposed in the storage unit or the computing unit.

16. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, The in-memory computing array is implemented based on at least one of SRAM, ReRAM, MRAM or FeFET technologies.

17. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, The computation results output by the neural network processing unit based on the in-memory computing array are transmitted through at least one of the following paths: directly written to a designated storage area of ​​the storage unit, transmitted to an external main control chip through the PCIe interface device, or transmitted to other extended storage units through a three-dimensional integrated connection structure.

18. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to claim 7, characterized in that, The "reuse of certain instructions from the standard PCIe" specifically refers to: Select instructions that are used less frequently or reserved in the PCIe memory standard protocol, and redefine them as control instructions for the computing unit, so that the instructions have both the original memory unit control function and the computing unit control function.

19. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, The AI ​​processor also includes a transmission unit, which is stacked with the storage unit and the computing unit using three-dimensional integration technology, and the PCIe interface device is disposed on the transmission unit.

20. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping as described in claim 19, characterized in that, The PCIe interface device includes a PCIe protocol parser, which is located in the transmission unit.

21. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping according to any one of claims 1-7, characterized in that, It also includes a data buffer module, which is used to temporarily store data transmitted between the external main control chip and the storage unit and the computing unit, so as to avoid congestion during data transmission and improve the stability of data interaction.

22. The AI ​​processor based on in-memory computing, three-dimensional integration, and PCIe remapping as described in claim 3, characterized in that, The instruction parsing unit is implemented as a fully custom circuit, a central processing unit, a RISC-V instruction set or an extended instruction parser thereof.

Images