High-efficiency GPNPU (General Purpose Network Processing Unit) chip architecture integrating memory and calculation

Abstract

The invention discloses a high-efficiency GPNPU (General Purpose Network Processing Unit) chip architecture with integrated memory and calculation, which belongs to the technical field of NPU chip architectures and comprises a plurality of array PE (Provider Edge) units, each PE unit comprises a local memory and a calculation unit and can independently execute vector and tensor calculation tasks, and the plurality of PE units are connected through data channels to realize direct exchange and sharing of data. Therefore, dependence on an external memory is reduced, and data processing efficiency is improved. According to the method, various existing AI network models can be supported, particularly, hardware, large-scale and instruction processing is carried out on operators in a recently popular LLM (Large Language Model), the AI calculation requirement of a large data volume is met, parallel calculation is achieved through specific instruction combination, the AI calculation efficiency and speed are improved, and complex AI tasks can be rapidly processed.

Description

Technical Field

[0001] This invention belongs to the field of NPU chip architecture technology, specifically referring to a high-efficiency GPNPU chip architecture that integrates memory and computing. Background Technology

[0002] Modern applications of artificial intelligence, especially machine learning, are driving innovation in computing technology, particularly in the fields of autonomous robots and self-driving vehicles. This involves processing various signal data from sensors, including a large amount of data processing and computation, especially vector and tensor data computation.

[0003] The underlying processing circuitry typically includes general-purpose integrated circuits such as central processing units (CPUs) and graphics processing units (GPUs), or dedicated NPUs, each with its own advantages and disadvantages.

[0004] CPU: Highly versatile and flexible, adept at complex control flow and serial tasks, but low energy efficiency and inefficient at handling large-scale parallel matrix operations (AI core).

[0005] GPUs: Powerful parallel computing capabilities (SIMT architecture) perform well in training and some inference tasks, but they suffer from high power consumption, memory access efficiency bottlenecks, and insufficient flexibility of fixed functional units. Especially for inference tasks deployed on edge devices, energy efficiency and latency remain challenges. Although they are highly versatile, they are not optimized from scratch for AI workloads.

[0006] Dedicated NPUs / ASICs: These are highly customized designs for specific AI models or operators, offering extremely high energy efficiency and computational density. However, they are extremely inflexible, struggling to adapt to rapidly evolving model architectures and new operators, and are characterized by high development costs and long development cycles.

[0007] Existing NPUs are divided into two categories. One category is mainly developed for traditional CNN and RNN network models, which is insufficient for LLM (Large Language Model) with high bandwidth requirements and cannot meet the high bandwidth requirements. The other category is developed based on GPU solutions, similar to NVIDIA's architecture, which completes AI computing through CUDA Cores and Tensor Cores.

[0008] These two existing NPU implementations can no longer keep up with the rapidly evolving AI network models. Nowadays, network models are iterating rapidly at a rate of months or weeks, with new models emerging one after another, and new algorithms and operators being constantly updated. Meanwhile, hardware updates are measured in years. Therefore, to ensure that the hardware design can efficiently support the ever-changing algorithms and operators, a more general-purpose NPU design is needed. Summary of the Invention

[0009] In response to the above situation and to overcome the shortcomings of the existing technology, this invention provides a high-efficiency GPNPU chip architecture that integrates memory and computing, effectively solving the problems currently on the market.

[0010] The technical solution adopted in this invention is as follows: This invention proposes a high-efficiency GPNPU chip architecture that integrates memory and computing, including multiple array PE units. Each PE unit includes local memory and computing units, which can independently execute vector and tensor computing tasks. Moreover, multiple PE units are connected through data channels to realize direct data exchange and sharing, thereby reducing the dependence on external memory and improving data processing efficiency.

[0011] Furthermore, the PE unit includes a two-dimensional MAC array, where the MAC array performs two-dimensional tensor calculations through an independent state machine, achieving efficient data parallel processing.

[0012] Furthermore, multiple PE units are connected through a mesh network structure, supporting local data sharing and scalability of computing power, effectively reducing the demand for storage bandwidth.

[0013] Furthermore, the PE unit executes operations through instruction control, and all PE units share external instructions and data. The programmability is achieved by software-controlled instructions, providing flexible task scheduling.

[0014] Furthermore, the PE unit can receive source data from neighboring PE units and transmit the calculation results to other PE units along the horizontal or vertical direction, realizing multi-directional data flow, reducing the relocation of data from PE units to external memory, and improving the locality of data processing.

[0015] Furthermore, the chip architecture includes a DMA architecture for efficient data transfer, supporting parallel computing and fusion control of multi-directional computing units, and optimizing the data transmission process.

[0016] Furthermore, the architecture design supports future updates of new AI network models and operators, exhibiting excellent scalability and adaptability.

[0017] Furthermore, the architecture reduces reliance on external memory and improves convolution operation efficiency by reusing local data and performing local computation during convolution operations.

[0018] Furthermore, the chip's PE unit supports mixed workloads, including the processing of AI core operators and general tasks, enabling efficient utilization of computing resources.

[0019] Furthermore, the architecture optimizes the memory subsystem by combining local memory with external storage, significantly alleviating the memory wall problem and improving overall energy efficiency.

[0020] The beneficial effects of this invention using the above structure are as follows: It can support a wide variety of existing AI network models, especially the hardware-based, scaled-up, and instruction-based processing of operators in the recently popular LLM (Large Language Model), meeting the AI ​​computing needs of large data volumes. Through specific instruction combinations, it achieves parallel computing, improving the efficiency and speed of AI computing, and can quickly process complex AI tasks. For new algorithms and operators that may emerge in the future, they can be decomposed into scalar and tensor processing units for hardware-based computing, exhibiting good scalability and foresight, and continuously supporting the development of AI technology. As a general-purpose NPU, it adopts a systematic architecture design, achieving parallel computing and fusion control of various operations through hierarchical control, overall instruction system control, efficient data transfer through a two-layer DMA architecture, and a multi-directional computing unit structure. This enables various independent systems to work together to achieve optimal computing performance. It has an efficient multi-layer memory management system, a general-purpose instruction set architecture, and a high-performance computing acceleration unit. These innovative designs make the chip superior to other NPUs in terms of versatility and efficiency, providing stronger support for AI computing. Attached Figure Description

[0021] Figure 1 As the basic unit of computation for fusion; Figure 2 An ARRAY composed of PEs forms a computing array; Figure 3 This is a source diagram for PE source data; Figure 4 This is a schematic diagram of the convolution operation; Figure 5 This is a schematic diagram of a 3x3 convolution completed in 2 cycles.

[0022] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0024] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0025] like Figures 1-5 As shown, this invention proposes a high-efficiency GPNPU chip architecture that integrates memory and computing, including multiple array PE units. Each PE unit includes local memory and computing units, which can independently execute vector and tensor computing tasks. The multiple PE units are connected through data channels to realize direct data exchange and sharing, thereby reducing the dependence on external memory and improving data processing efficiency.

[0026] The PE unit includes a two-dimensional MAC array, in which the MAC array performs two-dimensional tensor computation through an independent state machine, achieving efficient data parallel processing.

[0027] Multiple PE units are connected through a mesh network structure, supporting local data sharing and scalability of computing power, effectively reducing the demand for storage bandwidth.

[0028] The PE unit executes operations through instruction control, and all PE units share external instructions and data. The programmability is achieved by software-controlled instructions, providing flexible task scheduling.

[0029] PE cells can receive source data from neighboring PE cells and transmit the calculation results to other PE cells horizontally or vertically, realizing multi-directional data flow, reducing the relocation of data from PE cells to external memory, and improving the locality of data processing.

[0030] The chip architecture includes a DMA architecture for efficient data transfer, supporting parallel computing and fusion control of multi-directional computing units, and optimizing the data transfer process.

[0031] The architecture is designed to support future updates of new AI network models and operators, and has good scalability and adaptability.

[0032] The architecture reduces reliance on external memory and improves convolution operation efficiency by reusing local data and performing local computation during convolution operations.

[0033] The chip's PE unit supports mixed workloads, including AI core operators and general tasks, enabling efficient utilization of computing resources.

[0034] The architecture optimizes the memory subsystem by combining local memory with external storage, which significantly alleviates the memory wall problem and improves overall energy efficiency.

[0035] Detailed design and functional implementation of the PE unit: Each PE unit is a basic computing module in the chip architecture, containing local memory, computing units, and control logic. Local memory stores intermediate computation results, instructions, and a small amount of input / output data, employing high-speed SRAM to provide low-latency data access and ensure efficient operation of the computing units. Each computing unit integrates a MAC array, managing two-dimensional computation based on an independent state machine. Each MAC unit independently executes multiplication and addition operations. The state machine coordinates data flow and computation order, dynamically adjusting MAC operations. The control logic parses instructions and schedules computation and data transfer tasks, ensuring that the PE unit executes computations according to program instructions. For example, when performing matrix multiplication, the PE unit's local memory stores part of the input matrix's data and weights, the computing unit performs element-wise multiplication and addition operations, and the state machine controls data flow and computation order to achieve efficient matrix multiplication.

[0036] Efficient connection and data interaction mechanism for multiple PE units: Multiple PE units are connected through a mesh network topology to form a regular two-dimensional grid. There are bidirectional data channels between adjacent PE units, supporting direct data exchange and sharing. The data channels are designed with sufficient bandwidth, and the data distribution and collection mechanisms are optimized to improve data transmission efficiency. To avoid conflicts caused by multiple PE units accessing the same data at the same time, an arbitration mechanism is introduced to ensure smooth data interaction. For example, when performing large-scale matrix operations, the input matrix is ​​divided into small blocks and distributed to the local memory of each PE unit and transmitted through the mesh network. The data channels between adjacent PE units share boundary data in a timely manner, and the arbitration mechanism coordinates the access order to achieve efficient large-scale matrix operations.

[0037] Efficient implementation of convolution operations: Taking a 5x5 input matrix and a 3x3 convolution kernel as an example, the 3x3 convolution operation can be completed in two cycles by sharing local data among neighboring PE units. The input matrix is ​​divided into multiple sub-matrices and distributed to the PE units. The convolution kernel parameters are stored in the local memory of each PE unit. In the first cycle, the PE unit performs element-wise multiplication and addition operations on the sub-matrices and the convolution kernel to obtain a partial convolution result. In the second cycle, the PE unit obtains the data of the adjacent sub-matrices from neighboring PE units through the data channel, and combines it with the local data to complete the remaining multiplication and addition operations to obtain the complete convolution result. The whole process makes full use of local data and data from neighboring PE units, reduces the number of accesses to external memory, and improves the efficiency of convolution operation.

[0038] Data transfer and parallel computing support in DMA architecture: The DMA architecture is responsible for efficient data transfer tasks, quickly transferring large amounts of data from external memory to the local memory of the PE unit. It also supports multiple PE units to execute different computing tasks in parallel, improving overall computing performance. The DMA controller prefetches data according to instructions, optimizes the data transfer process, and reduces the interference of data transfer on the computing units. For example, when processing image data, the DMA architecture transfers image data from external memory to the local memory of the PE unit in batches. Each PE unit executes tasks such as image filtering and edge detection in parallel. The DMA controller prefetches the data needed later, realizing an efficient data processing flow.

[0039] The flexibility and adaptive expansion of the architecture: Through instruction control and programmability, this architecture can adapt to rapid updates of future AI models. Software updates support new algorithms and operators, ensuring the architecture remains advanced and practical throughout technological iterations. For example, when new neural network layer types or unique operator operations emerge, the instruction set and control logic of the PE units can be updated, enabling the architecture to efficiently execute new computational tasks and adapt to ever-changing AI computing needs.

[0040] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0041] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0042] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A high-efficiency GPNPU chip architecture that integrates memory and computing, characterized in that: It includes multiple array PE units, each PE unit includes local memory and computing units, and can independently perform vector and tensor computing tasks. Multiple PE units are connected through data channels to realize direct data exchange and sharing, thereby reducing dependence on external memory and improving data processing efficiency.

2. The high-efficiency GPNPU chip architecture integrating memory and computing as described in claim 1, characterized in that: The PE unit includes a two-dimensional MAC array, in which the MAC array performs two-dimensional tensor computation through an independent state machine, achieving efficient data parallel processing.

3. The high-efficiency GPNPU chip architecture integrating memory and computing as described in claim 2, characterized in that: Multiple PE units are connected through a mesh network structure, supporting local data sharing and scalability of computing power, effectively reducing the demand for storage bandwidth.

4. The high-efficiency GPNPU chip architecture integrating memory and computing as described in claim 3, characterized in that: The PE unit executes operations through instruction control, and all PE units share external instructions and data. The programmability is achieved by software-controlled instructions, providing flexible task scheduling.

5. The high-efficiency GPNPU chip architecture integrating memory and computing as described in claim 4, characterized in that: PE cells can receive source data from neighboring PE cells and transmit the calculation results to other PE cells horizontally or vertically, realizing multi-directional data flow, reducing the relocation of data from PE cells to external memory, and improving the locality of data processing.

6. The high-efficiency GPNPU chip architecture integrating memory and computing as described in claim 5, characterized in that: The chip architecture includes a DMA architecture for efficient data transfer, supporting parallel computing and fusion control of multi-directional computing units, and optimizing the data transfer process.

7. The high-efficiency GPNPU chip architecture integrating memory and computing as described in claim 6, characterized in that: The architecture is designed to support future updates of new AI network models and operators, and has good scalability and adaptability.

8. The high-efficiency GPNPU chip architecture integrating memory and computing as described in claim 7, characterized in that: The architecture reduces reliance on external memory and improves convolution operation efficiency by reusing local data and performing local computation during convolution operations.

9. The high-efficiency GPNPU chip architecture integrating memory and computing as described in claim 8, characterized in that: The chip's PE unit supports mixed workloads, including AI core operators and general tasks, enabling efficient utilization of computing resources.

10. The high-efficiency GPNPU chip architecture integrating memory and computing as described in claim 9, characterized in that: The architecture optimizes the memory subsystem by combining local memory with external storage, which significantly alleviates the memory wall problem and improves overall energy efficiency.

Images