Npu architecture, data processing method, electronic device and chip

Abstract

This application discloses an NPU architecture, data processing method, electronic device, and chip, belonging to the field of artificial intelligence. The NPU architecture may include: a PE array, a reconfigurable interconnect network, and a configuration and control subsystem; the PE array consists of multiple PEs, and the reconfigurable interconnect network connects each PE in the PE array; wherein, the configuration and control subsystem is configured to: load target hardware configuration information corresponding to the target computing task according to a reconfiguration request corresponding to the target computing task; and dynamically configure the path topology of the reconfigurable interconnect network and the working mode of the PEs in the target PE region of the PE array according to the target hardware configuration information, thereby reconfiguring the target PE region into a hardware structure for executing the target computing task.

Description

Technical Field

[0001] This application belongs to the field of terminal technology, specifically relating to an NPU architecture, a data processing method, an electronic device, and a chip. Background Technology

[0002] Artificial intelligence (AI) workloads exhibit significant heterogeneity in their computational characteristics.

[0003] Specifically, AI workloads not only include computationally intensive tasks such as General Matrix Multiplication (GEMM), but also involve numerous element-level operations, data rearrangements, and complex control flows. Current common hardware accelerators, such as application-specific integrated circuit (ASIC) systolic arrays optimized for GEMM or graphics processing units (GPUs) designed for graphics processing, form isolated "hardware architecture islands." This highly specialized design leads to significant system bottlenecks: dedicated hardware units often remain idle when handling computational modes they are not suited for.

[0004] Thus, in order to execute different computing modes, various dedicated hardware units need to be set up, which leads to an increase in the number of dedicated hardware units and low end-to-end utilization. Summary of the Invention

[0005] The purpose of this application is to provide a Neural Processing Unit (NPU) architecture, data processing method, electronic device, and chip that can improve the versatility of dedicated hardware units and end-to-end utilization.

[0006] In a first aspect, embodiments of this application provide an NPU architecture, which may include: a Processing Element (PE) array, a reconfigurable interconnect network, and a configuration and control subsystem; the PE array consists of multiple PEs, and the reconfigurable interconnect network connects each PE in the PE array; wherein, the configuration and control subsystem is configured to: load target hardware configuration information corresponding to the target computing task according to a reconfiguration request corresponding to the target computing task; and dynamically configure the path topology of the reconfigurable interconnect network and the working mode of the PEs in the target PE region of the PE array according to the target hardware configuration information, thereby reconfiguring the target PE region into a hardware structure for executing the target computing task.

[0007] Secondly, embodiments of this application provide an electronic device including the Neural Processing Unit (NPU) architecture as described in the first aspect.

[0008] Thirdly, embodiments of this application provide a data processing method, which includes: loading target hardware configuration information corresponding to the target computing task according to a reconstruction request corresponding to the target computing task through a configuration and control subsystem; and dynamically configuring the path topology of the reconfigurable interconnection network and the working mode of the PE in the target PE region of the PE array according to the target hardware configuration information through the configuration and control subsystem, thereby reconstructing the target PE region into a hardware structure for executing the target computing task.

[0009] Fourthly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the third aspect.

[0010] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the third aspect.

[0011] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method described in the third aspect.

[0012] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the third aspect.

[0013] In this embodiment, the NPU architecture may include: a PE array, a reconfigurable interconnect network, and a configuration and control subsystem. The PE array consists of multiple PEs, and the reconfigurable interconnect network connects each PE in the PE array. The configuration and control subsystem is configured to: load target hardware configuration information corresponding to the target computing task according to a reconfiguration request corresponding to the target computing task; and dynamically configure the path topology of the reconfigurable interconnect network and the working mode of the PEs in the target PE region of the PE array according to the target hardware configuration information, thereby reconfiguring the target PE region into a hardware structure for executing the target computing task. In this solution, since the configuration and control subsystem in the NPU architecture can load instantiated hardware configuration information according to the reconfiguration request corresponding to the target computing task, dynamically configure the path topology of the reconfigurable interconnect network and the working mode of the PEs in the PE array, thereby reconfiguring the target PE region in the PE array into a hardware structure for executing the target computing task, the NPU architecture can dynamically switch between computing tasks corresponding to different computing paradigms according to the computing task requirements (represented or reflected by the reconfiguration request) corresponding to the AI ​​workload. This fundamentally solves the problem of architectural silos caused by different computing paradigms relying on different dedicated hardware architectures, enabling the computing paradigms supported by the NPU architecture to adapt to AI workloads and significantly improving the end-to-end utilization of the NPU architecture. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the NPU architecture provided in some embodiments of this application;

[0015] Figure 2 This is a schematic diagram of the PE structure in the NPU architecture provided by some embodiments of this application;

[0016] Figure 3 These are schematic diagrams of the structure of a multifunctional ALU in a PE provided by some embodiments of this application;

[0017] Figure 4 This is a schematic diagram of the structure of the local memory of the PE provided in some embodiments of this application;

[0018] Figure 5 These are some embodiments of this application providing functional diagrams of DPR in the NPU architecture;

[0019] Figure 6 These are schematic diagrams illustrating the memory architecture of the NPU architecture provided in some embodiments of this application;

[0020] Figure 7 These are some embodiments of the present application providing example diagrams of reconstructing a PE array into a pulsating array in an NPU architecture;

[0021] Figure 8These are some embodiments of the present application providing example diagrams of reconstructing a PE array into a SIMD array in an NPU architecture;

[0022] Figure 9 This is an example schematic diagram of the PE array of the NPU architecture provided in some embodiments of this application being reconstructed into a hybrid array;

[0023] Figure 10 This is a flowchart illustrating a data processing method provided in some embodiments of this application;

[0024] Figure 11 These are schematic diagrams of the structure of an electronic device provided in some embodiments of this application;

[0025] Figure 12 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application;

[0026] in, Figures 1 to 8 The accompanying figure labels are as follows:

[0027] 100-NPU architecture;

[0028] 10-PE array, 11-PE, 111-ALU, 12-local memory, 121-local register file, 122-local configuration memory;

[0029] 20 - Reconfigurable interconnection network; 21 - Transmission unit; 22 - Switch;

[0030] 30-Configuration and Control Subsystem, 31-Global Configuration Manager, 32-DPR Manager

[0031] 40 - On-chip shared memory; 50 - Memory access units;

[0032] 60 - Off-chip memory, 70 - DMA controller. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0034] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0035] The following explains the nouns or terms used in the embodiments of this application.

[0036] 1.0 AI Workload: This refers to the computational tasks and processes involved in the development, training, and deployment of AI models. These computational tasks include computationally intensive operations such as data processing, model training, and model inference; large-scale element-level operations; data rearrangement operations; and complex data flow control operations, all requiring efficient execution.

[0037] 1.1. Dedicated Dataflow Architecture: Systolic Array

[0038] A systolic array is a highly optimized, dedicated hardware architecture for specific computations, primarily matrix and convolution operations, where convolution involves multiple vector multiplications. Its design philosophy is to break down computational tasks into a series of small, repetitive steps and process data in a pipelined manner through a two-dimensional grid of PEs (Programmers and Activators). Data (such as weights and activations) flows through the PE array in a “systolic” manner, with each PE performing a multiply-accumulate (MAC) operation every clock cycle and passing the result to its neighboring PE.

[0039] This architecture maximizes data locality, enabling significant data reuse and minimizing the need for high-bandwidth memory access. Due to its simplicity, regularity, and scalability, systolic arrays have become the cornerstone of modern neural processing units (such as NPUs), with the most famous application being Google's Tensor Processing Unit (TPU). The TPU utilizes its massive systolic arrays to achieve extremely high MAC unit density and energy efficiency when performing GEMM and convolution operations.

[0040] 1.2. Data Parallel Architecture: Single Instruction Multiple Data (MIMD) is a parallel processing technique whose core feature is that it allows a single instruction to execute on multiple data elements simultaneously. This model is widely used in advanced vector extensions of modern central processing units (CPUs) such as AVX and graphics processing units (GPUs).

[0041] In AI workloads, neural networks not only involve computationally intensive MAC operations but also a large amount of highly parallel computation across the data dimension, such as element-wise activation functions (e.g., ReLU, GELU, Sigmoid), vector addition, data normalization (e.g., LayerNorm, L2Norm), and various data preprocessing operations. For these operations, the SIMD architecture can process a data vector (e.g., 128-bit, 256-bit, or 512-bit) with a single instruction, significantly improving computational throughput. More importantly, SIMD significantly reduces the overhead of instruction fetching and decoding and alleviates pressure on the memory subsystem through aligned, contiguous memory access patterns.

[0042] 1.3. Spatial Computing Architecture: Coarse-Grained Reconfigurable Array (CGRA)

[0043] Coarse-grained reconfigurable arrays (CGRAs) are a type of programmable accelerator architecture that bridges the gap between the high efficiency of dedicated ASICs and the high flexibility of FPGAs. Unlike FPGAs, which reconfigure at the bit level, CGRAs typically consist of an array of PEs that perform arithmetic and logic operations at the word level (e.g., 32-bit or 64-bit).

[0044] The core working principle of CGRA is spatial computation. The compiler extracts computationally intensive loops and transforms them into a Data Elow Graph (DFG). This DFG is then "mapped" onto the CGRA's physical array of PEs: nodes (operations) in the DFG are assigned to PEs, and edges (data dependencies) in the DFG are assigned to a configurable interconnect network between PEs. Once data enters the array, it flows between PEs according to the paths defined by the DFG, thus achieving pipelined parallelism with extremely low control overhead.

[0045] 1.4. Parallel Task Architecture: Multiple Instruction, Multiple Data (MIDDLE)

[0046] MIMD is the most flexible paradigm in parallel computing. A MIMD architecture contains multiple processor cores (or PEs) that can run asynchronously and independently. At any given time, different processor cores can execute different instructions to process different data.

[0047] General-purpose multi-core CPUs and many parallel computer systems are examples of the MIMD architecture. In the field of AI, the value of MIMD lies in its ability to handle complex and irregular workloads. This includes algorithms with complex data dependencies and control flows (e.g., dynamic computation in neighbor aggregation or attention mechanisms in graph neural networks, GNNs), and scenarios requiring the concurrent execution of multiple different AI tasks (e.g., running perception, localization, and planning models simultaneously in autonomous driving). MIMD offers the highest level of flexibility, enabling it to handle heterogeneous computing tasks and dynamic workloads.

[0048] The NPU architecture, data processing method, electronic device, and chip provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0049] The NPU architecture, data processing method, electronic device, and chip provided in this application can be applied to scenarios involving various AI workload-related computing tasks. For example, they can be applied to various computing tasks that require dedicated hardware acceleration, such as intensive computing tasks, large-scale element-level operations, data rearrangement tasks, or complex control flow tasks.

[0050] In the relevant technologies, the hardware architectures corresponding to the AI ​​workloads shown in 1.1 to 1.4 above clearly reveal a fundamental challenge in the current field of AI hardware acceleration: AI workloads are heterogeneous, while mainstream hardware architectures are homogeneous (or a single paradigm). In other words, a complete, advanced AI model (such as Transformer or GNN) in the relevant technologies is computationally highly heterogeneous. It not only includes dense GEMMs suitable for systolic arrays (1.1), but also a large number of element-level operations suitable for SIMD (1.2), as well as complex data rearrangement, sparse computation, and control flow-intensive operations suitable for CGRA (1.3) or MIMD (1.4).

[0051] It should be noted that the hardware architectures corresponding to 1.1 to 1.4 above can all be referred to as an "architectural paradigm". Referring to Table 1, the hardware architectures corresponding to AI workloads in related technologies exhibit a "holistic architecture" dilemma. For example, TPUs are good at systolic array computing in 1.1 above, GPUs are good at SIMD computing in 1.2 above, and CPUs are good at MIMD computing in 1.4 above.

[0052] Table 1

[0053] Architectural Paradigm Computational granularity Computational paradigm Programmability Main applications Architectural defects ASIC (Inductively Coupled Array) tensor Dedicated data stream Extremely low GEMM It is relatively rigid and inefficient for non-MAC operations. GPU (SIMD / SIMT) vector SIMD middle Vector computation High power consumption and low control flow instruction efficiency CGRA Granularity Spatial data flow graph high Computation-intensive loops Reconfiguration overhead is high CPU (MIMD) scalar Arbitrary logic Extremely high Control flow intensive operations Inefficient at performing computationally intensive operations

[0054] As shown in Table 1, while each architectural paradigm in the relevant technologies has achieved success in specific domains, its inherent limitations prevent it from independently addressing all the challenges of modern AI workloads. These shortcomings collectively constitute a "flexibility-efficiency gap." Specifically:

[0055] 2.1 The rigidity and low utilization of dedicated architectures

[0056] Specialized architectures such as systolic arrays prioritize extreme specialization for maximum efficiency, but this also leads to significant rigidity.

[0057] (1) Narrow workload coverage: Systolic arrays are deeply tailored for GEMM and convolutional operations. However, in modern neural networks (especially Transformer and State-Space Model SSM), the proportion of non-MAC operations has increased dramatically. This includes a large number of element-wise computations (such as exp(), rowmax()), activation functions (such as GELU), reduction and complex data rearrangement operations.

[0058] (2) Bottleneck of non-MAC operators: The systolic array cannot efficiently execute these non-MAC operations on its PE. Therefore, the systolic array-based NPU architecture must rely on external, independent computing units (such as vector processors (VPUs) or scalar CPUs) to handle these computations. This leads to the aforementioned "architectural silo" bottleneck: data must be expensively moved back and forth between the systolic array and external units, causing the systolic array to be idle while waiting for data, which greatly reduces the end-to-end hardware utilization.

[0059] (3) Matrix shape sensitivity: The utilization rate of PE in a systolic array is highly sensitive to whether the physical array can be filled "before computation". When processing non-square matrices (e.g., "tall" or "fat" matrices commonly found in NLP) or small-batch inference tasks, the array pipeline cannot be effectively filled, resulting in a large number of PEs being idle and a sharp drop in actual computational efficiency.

[0060] (4) Difficulty in processing sparse data: The rigid data flow of the systolic array forces the PE to process all incoming data, regardless of whether it is zero or not. This makes it difficult to skip zero values ​​in sparse data, resulting in a speedup effect that is far less than expected when dealing with increasingly important sparse models.

[0061] 2.2 Control Flow Limitations of SIMD Architecture

[0062] The core advantage of SIMD architectures (such as GPUs) lies in their "single instruction" characteristic, but this is also their core limitation:

[0063] (1) Control flow divergence, requiring hardware to execute each branch path serially, leading to decreased efficiency: When different "threads" (data channels) in a SIMD unit (such as a GPUWarp) encounter data-dependent branches (if-else), since the hardware can only execute one instruction at a time, it is forced to execute all branch paths serially (through masking / predication techniques). For algorithms with complex or irregular control flow, this "divergence" can cause a sharp decrease in the execution efficiency of the SIMD unit.

[0064] (2) Task granularity limitation: SIMD excels at fine-grained data-level parallelism (DLP), but its architecture (lockstep execution) fundamentally cannot natively support asynchronous, independent task-level parallelism (TLP), i.e. MIMD.

[0065] 2.3 The Compilation and Refactoring Challenges of Traditional CGRA

[0066] CGRA theoretically offers a perfect solution for balancing efficiency and flexibility, but in practice, it has long been plagued by two fundamental problems:

[0067] (1) Extreme complexity of compilation mapping: The flexibility of CGRA comes entirely from its complex software (compiler). Mapping an application (represented as DFG) to the physical PE array of CGRA requires simultaneous scheduling, placement, and routing. This mapping problem has been proven to be NP-complete.

[0068] (2) Excessive compilation time: Due to its NP-complete nature, CGRA's compilation toolchain is extremely complex and time-consuming. For a medium-sized computing kernel, the compilation time can be as long as several hours or even days. In today's world where AI algorithm models iterate rapidly on a weekly basis, such compilation delays are completely unacceptable.

[0069] (3) High refactoring overhead:

[0070] (3.1) Bottleneck of Single-Context CGRA: Many CGRAs can only maintain one configuration (i.e., one DFG mapping) at any given time. When the AI ​​model needs to switch from one computation kernel (e.g., Conv layer) to the next kernel (e.g., ReLU layer), a completely new configuration bitstream must be loaded from off-chip memory.

[0071] (3.2) Overhead offsets benefits: This loading process may take microseconds (µs) or even milliseconds (ms). If the computation kernel itself has a short execution time (e.g., performing ReLU on a small vector), then the time spent reconfiguring the configuration will far exceed the time saved by CGRA executing the kernel. This forces the host processor (such as the CPU) to frequently intervene to coordinate configuration and data flow, causing the speedup benefits of CGRA to be completely offset by overhead.

[0072] (3.3) Limitations of Multi-Context CGRA: To address the shortcomings of single-context CGRA, some traditional CGRAs allow the PE to store multiple configurations (contexts) locally and achieve fast switching in a single cycle. However, this significantly increases the hardware complexity, area, and power consumption of the PE. More importantly, the number of local contexts (typically only a few dozen) is limited and far from sufficient to store all the configurations of a modern large-scale AI model (which may contain thousands of cores).

[0073] In summary, as analyzed in sections 2.1 to 2.3 above, the core hardware obstacle for AI algorithm models in related technologies is the "flexibility-efficiency gap": On the one hand, ASIC systolic arrays possess extremely high peak efficiency, but their average efficiency (i.e., end-to-end efficiency across the entire AI workload) is very low. This is because they are severely hampered by non-GEMM operations, irregular matrix shapes, and sparse data, causing them to be idle or inefficient most of the time. On the other hand, traditional CGRAs offer extremely high theoretical flexibility, capable of adapting to different operators through refactoring; however, this flexibility is constrained by the overhead of "compilation" and "refactoring," significantly reducing their practical usable flexibility.

[0074] Therefore, based on the "heterogeneous" architectural paradigm in related technologies, in order to execute a complete AI task, the system is forced to adopt a heterogeneous hardware integration scheme, such as integrating "TPU (for GEMM) + VPU (for SIMD)" or integrating "NPU + CPU". This scheme of integrating multiple dedicated hardware not only significantly increases hardware cost, area and design complexity, but also results in a large number of dedicated hardware architectures and poor end-to-end utilization because different hardware architectures are not good at different calculations.

[0075] To address the issues of increased hardware costs and poor end-to-end utilization resulting from stacking dedicated hardware units, embodiments of this application provide a general-purpose NPU architecture that can adapt itself to all computing tasks in AI workloads.

[0076] For example, the NPU architecture may include: a PE array, a reconfigurable interconnect network, and a configuration and control subsystem; the PE array consists of multiple PEs, and the reconfigurable interconnect network connects each PE in the PE array; wherein, the configuration and control subsystem is configured to: load target hardware configuration information corresponding to the target computing task according to a reconfiguration request corresponding to the target computing task; and dynamically configure the path topology of the reconfigurable interconnect network and the working mode of the PEs in the target PE region of the PE array according to the target hardware configuration information, thereby reconfiguring the target PE region into a hardware structure for executing the target computing task. In this solution, since the configuration and control subsystem in the NPU architecture can load instantiated hardware configuration information according to the reconfiguration request corresponding to the target computing task, dynamically configure the path topology of the reconfigurable interconnect network and the working mode of the PEs in the PE array, thereby reconfiguring the target PE region in the PE array into a hardware structure for executing the target computing task, the NPU architecture can dynamically switch between computing tasks corresponding to different computing paradigms according to the computing task requirements (represented or reflected by the reconfiguration request) corresponding to the AI ​​workload. This fundamentally solves the problem of architectural silos caused by different computing paradigms relying on different dedicated hardware architectures, enabling the computing paradigms supported by the NPU architecture to adapt to AI workloads and significantly improving the end-to-end utilization of the NPU architecture.

[0077] Furthermore, since the NPU architecture in this application can introduce hardware configuration information corresponding to computing tasks, the NPU architecture in this embodiment can switch between AI computing tasks of different computing paradigms by loading "hardware configuration information" without having to load and compile a complete new configuration bitstream from off-chip slow memory like CGRA in related technologies. Therefore, the switching speed and efficiency between computing paradigms can be greatly shortened, thereby further improving the performance of the NPU architecture.

[0078] This application provides an NPU architecture, such as... Figure 1 As shown, the NPU architecture 100 may include: a processing element PE array, a reconfigurable interconnect network 20, and a configuration and control subsystem 30; the PE array is composed of multiple PE11, and the reconfigurable interconnect network 20 connects each PE11 in the PE array.

[0079] The configuration and control subsystem 30 is configured as follows:

[0080] Based on the reconfiguration request corresponding to the target computing task, load the target hardware configuration information corresponding to the target computing task;

[0081] Based on the target hardware configuration information, the path topology of the reconfigurable interconnect network 20 and the working mode of PE11 in the target PE region of the PE array are dynamically configured, thereby reconfiguring the target PE region into a hardware structure for performing the target computing task.

[0082] In some embodiments of this application, the PE array described above can be an N×M PE array, where M and N are both positive integers.

[0083] In some embodiments of this application, the aforementioned target hardware configuration information is related to the target computing paradigm corresponding to the target computing task.

[0084] For example, the aforementioned target hardware configuration information may be configuration information instantiated from the preset hardware configuration information corresponding to the target computing paradigm, and the target computing paradigm is determined according to the operator type of the target computing task.

[0085] Thus, since the target hardware configuration information is related to the target computing paradigm corresponding to the target computing task, the target hardware configuration information corresponding to the target computing task can be quickly locked and loaded according to the computing paradigm corresponding to the target computing task, thereby reducing the latency of loading the target hardware configuration information.

[0086] It is understood that the "hardware configuration information" in the embodiments of this application can be configured as a hardware configuration template. For example, the aforementioned "target hardware configuration information" can be configured as a "target hardware configuration template". Unless otherwise specified, the "target hardware configuration template" in the following embodiments all include "hardware configuration information". For example, a SIMD configuration template refers to SIMD configuration information.

[0087] In some embodiments of this application, the reconfigurable interconnect network serves as a "data path" connecting the PE array and is a key enabling component for achieving "dynamic reconfigurability." It should be noted that this reconfigurable interconnect network must support dynamic routing.

[0088] In some embodiments of this application, the configuration and control subsystem is the "brain" of the NPU architecture, responsible for loading configuration (i.e., target hardware configuration information) and coordinating the execution of the PE array.

[0089] The NPU architecture provided in this application aims to offer a unified architecture for various computing paradigms present in AI workloads. The PE array in this NPU architecture is no longer a fixed ASIC or a general-purpose CGRA, but rather a "polymorphic architecture." This NPU architecture can be dynamically configured at runtime to at least one of the following based on the computational task requirements of the application layer: a high-efficiency systolic array (for GEMM / convolution); a massively parallel SIMD array (for element-level operations); or a flexible MIMD array (for task parallelism and complex control flow).

[0090] In some embodiments of this application, the aforementioned PEs natively support all key types of operators or operations involved in various computational paradigms.

[0091] It is understood that this application provides a single, dynamically reconfigurable NPU architecture. This architecture is based on a PE array supporting multiple computing paradigms and a reconfigurable interconnect network. Each PE in the PE array supports multiple computing paradigms and has local memory and multiple transmission paths with the configurable interconnect network, thus enabling interconnection between any two PEs in the PE array.

[0092] It should be noted that, through the collaboration of pre-configured hardware configuration information associated with multiple computing paradigms, PE arrays supporting multiple computing paradigms, and reconfigurable interconnect networks, the NPU architecture can freely and quickly switch between hardware architectures that support different computing paradigms. Here, "hardware architecture" can also be referred to as "paradigm architecture".

[0093] In some embodiments of this application, the target computation paradigm includes at least one of the systolic array paradigm, SIMD paradigm, and MIMD paradigm.

[0094] It is understandable that the three main computing paradigms—systolic array paradigm, SIMD paradigm, and MIMD paradigm—can cover the most critical computing tasks in artificial intelligence workloads, such as AI tasks involving computationally intensive, data-parallel intensive, and control flow intensive operators. This ensures that the NPU architecture provides efficient support for modern and future mainstream AI models (such as Transformer and GNN), offering high flexibility without sacrificing the execution efficiency of critical operations.

[0095] Thus, since the target computing paradigm can include at least one of the systolic array paradigm, SIMD paradigm, and MIMD paradigm, it can be ensured that the NPU architecture can dynamically switch between these three main computing paradigms and can be reconfigured simultaneously as a hardware structure to execute at least one computing paradigm, thereby improving the versatility of the NPU architecture.

[0096] Furthermore, since the NPU architecture can be reconfigured into a hardware structure that executes at least one computing paradigm, the intermediate data involved in a complete AI computing task can remain within the NPU architecture throughout the entire process, thereby reducing data transfer and improving the computing flexibility of the NPU architecture.

[0097] In some embodiments of this application, when faced with a target computation task in an AI workload, the compiler can analyze the operator type or operation type in the target computation task to determine the target computation paradigm corresponding to the target computation task. Then, the compiler can load predefined hardware configuration information corresponding to the target computation paradigm from the hardware configuration information library; and instantiate the hardware configuration information according to the computation parameters of the target computation task to obtain the target hardware configuration information.

[0098] For example, a compiler can parse an AI model, identify the types of operators in the target computation task to be performed by the AI ​​model, and determine the target computation paradigm corresponding to the target computation task based on the types of operators in the target computation task. The operator types can include: matrix multiplication or convolution computation, element-wise parallel computation, irregular data dependencies, or complex control flow computation, etc.

[0099] After determining the target computing paradigm, the compiler can select hardware configuration information corresponding to the target computing paradigm from a predefined hardware configuration information library based on the computing parameters of the target computing task (also known as task parameters, such as matrix size). According to the specific task parameters, the compiler instantiates the selected hardware configuration information to generate the final executable target hardware configuration information and delivers the target hardware configuration information to the NPU architecture driver or runtime. That is, the NPU architecture configures the working mode of the PE array and the path topology of the reconfigurable interconnect network according to the instantiated configuration information.

[0100] For example, if the target computation task involves intensive matrix multiplication or convolution (such as vector multiplication) computations, the above target computation paradigm may include the systolic array paradigm.

[0101] If the target computation task contains a large number of rule-based element-level parallel operations (such as activation and normalization), then the target computation paradigm can include the SIMD paradigm.

[0102] If the target computation task contains irregular data dependencies or complex control flows, the target computation paradigm may include the MIMD paradigm.

[0103] It should be noted that the instantiation of hardware configuration information can be performed online or offline, i.e., offline execution, and this application embodiment does not limit it.

[0104] For example, consider offline instantiation. The compiler can pre-instantiate the hardware configuration information corresponding to each computing task in the computing task queue, which can further improve the efficiency of the NPU architecture in executing computing tasks.

[0105] In some embodiments of this application, after the compiler obtains the target hardware configuration information, the main processor or load scheduler can send a reconstruction request corresponding to the target computing paradigm to the configuration and control subsystem based on the target hardware configuration information.

[0106] In some embodiments of this application, the configuration and control subsystem may receive a reconfiguration request sent by the main processor or the load scheduler.

[0107] In some embodiments of this application, the configuration and control subsystem dynamically configures the path topology of the reconfigurable interconnect network 20 and the operating mode of PE11 in the target PE region of the PE array based on the target hardware configuration information. This reconfigures the target PE region into a hardware structure capable of performing the target computing task, similar to mapping or loading the target hardware configuration information into the PE array and the reconfigurable interconnect network. After completing the hardware mapping, the configuration and control subsystem can perform data mapping, importing the corresponding computing data from the target computing task into the target PE region. This allows the target PE region to process data according to the configured operating mode and path topology, and outputs the processing results.

[0108] In some embodiments of this application, the hardware configuration information corresponding to the computing paradigm is predefined.

[0109] For example, the computation paradigms corresponding to various computation tasks can be determined according to the operator types of various computation tasks involved in the target computation task, and predefined hardware configuration information corresponding to each computation paradigm can be set to obtain a hardware configuration information database.

[0110] It should be noted that the hardware configuration information concept in this application treats the "computational paradigm" itself as a loadable software configuration, thereby achieving "computational paradigm adaptability to workload" on a unified NPU architecture. Unlike traditional CGRA, the NPU architecture in this application provides optimized hardware configuration information for key computational paradigms. Thus, when faced with a specific computational task, the template can be converted into a template supporting that computational task by loading the hardware configuration information and instantiating it. This greatly simplifies compilation issues, achieving ASIC-level efficiency while retaining the hardware flexibility required to handle future novel AI models (such as Transformer and GNN).

[0111] In some embodiments of this application, the hardware configuration information in the hardware configuration information database is parameterized. Instantiating the hardware configuration information is similar to filling in or scaling the size of a table with a specific format based on the actual computation parameters of the computation task. For example, the computation parameters of the target computation task are filled into the parameter field of the hardware configuration information corresponding to the target computation paradigm.

[0112] In some embodiments of this application, the computational parameters of the target computational task may include at least one of the following: operator type, input / output data precision, weight data, data flow direction of each operator, etc.

[0113] Because each hardware configuration template corresponds to a computational paradigm, the configuration templates in the hardware configuration information database can also be called computational paradigm templates. For the same hardware configuration information, by inputting different parameters (such as matrix size M, N, K, and data flow direction), it can be instantiated into an applicable template with a large number of specific operators, such as the target hardware configuration information mentioned above.

[0114] In some embodiments of this application, the hardware configuration information corresponding to the computing paradigm is a general configuration template corresponding to a type of computing task.

[0115] For example, the MIMD paradigm corresponds to the MIMD configuration template, which can also be simply referred to as the MIMD template.

[0116] For example, the SIMD paradigm corresponds to the SIMD configuration template, which can also be simply referred to as the SIMD template.

[0117] For example, the pulsation array paradigm corresponds to the pulsation array configuration template, which can also be simply referred to as the pulsation array template.

[0118] In some embodiments of this application, each computing paradigm may correspond to at least one hardware configuration information.

[0119] Method 1: Each computational paradigm corresponds to a hardware configuration information. The compiler can adaptively adjust the specification parameters of the corresponding general hardware configuration information based on the operator size information of the target computational task, such as the size of the computational parameters, to obtain the instantiated target hardware configuration information.

[0120] For example, taking an 8x8 systolic array template as an example, if the target computation task requires a 4x4 systolic array for its operator size, then the general systolic array template can be transformed into a 4x4 systolic array. For instance, the 4x4 template area in the upper left corner of the general systolic array template can be used as the target hardware configuration information. This is illustrated using a systolic array template as an example. In actual implementation, for other templates, the same or corresponding methods can be used to instantiate the hardware configuration information corresponding to the computation paradigm to obtain hardware configuration information adapted to the specific computation task.

[0121] In some embodiments of this application, the compiler can adaptively adjust the specification parameters of the corresponding hardware configuration information based on the operator size information of the target computing task and the running status of the PE array, and instantiate it to obtain the target hardware configuration information.

[0122] In some embodiments of this application, the operating state of the PE array may include the operating state of each PE in the PE array. The operating state of a PE may include: operating state and idle state. It can be understood that the operating state of the PE array can indicate the currently available PEs in the PE array.

[0123] For example, if the template size of a general systolic array is 8*8, and the target computation task requires a 16*16 systolic array template, but only 64 PEs in the PE array are idle, then the compiler can instantiate the 8*8 systolic array template into the target hardware configuration information. It should be noted that, generally, hardware configuration information matching the operator size of the target computation task can maximize the execution performance of the target computation task, such as maximizing speed and minimizing latency. However, the target computation task can also be executed using a template corresponding to an idle PE in the PE array without waiting for the other tasks in the PE array to finish. Therefore, in actual implementation, the waiting time for freeing up a PE region in the PE array that matches the operator size of the target computation task, and the latency increment of using a hardware template matching the current number of idle PEs, can be comprehensively considered to obtain the most suitable target hardware configuration information for instantiation.

[0124] In some embodiments of this application, the running status of the PE can be monitored by a configuration and control subsystem. Further optionally, the configuration and control subsystem can feed back the monitored running status to the compiler.

[0125] Method 2: Each computational paradigm corresponds to at least two commonly used hardware configuration templates, referred to as commonly used hardware configuration templates. The compiler can select a hardware configuration template that matches the operator size from the commonly used hardware configuration templates corresponding to the target computational paradigm, and instantiate it to obtain the target hardware configuration template, based on the operator size of the target computational task.

[0126] For example, for the systolic array paradigm, an 8*8 systolic array template and a 10*10 systolic array template can be set as commonly used templates.

[0127] In some embodiments of this application, in method 2, hardware configuration information matching both the operator size information and the operating status of the PE array can be selected from commonly used configuration information corresponding to the target computing paradigm, based on the operator size of the target computing task and the operating status of the PE array. The selected template is then instantiated to obtain the target hardware configuration information. For specific considerations, please refer to the relevant description in method 1.

[0128] In some embodiments of this application, since the reconfigurable interconnect network and each PE have multiple connection ports, the path topology of the reconfigurable interconnect network can be reconfigured to make one PE in the PE array logically or data flowly "adjacent" to any other PE in the PE array.

[0129] It is understandable that, based on the reconfiguration flexibility of reconfigurable interconnects, the aforementioned target hardware configuration information can also be called "logical hardware configuration information" or "virtual hardware configuration information." In other words, when instantiating a piece of hardware configuration information, it is only necessary to follow the constraint of the number of PEs in the PE array, without being limited by the hardware shape and size of the PE array.

[0130] For example, the target hardware configuration information can configure an N×M PE array into a logically deeper and narrower systolic array, such as a (2N)×(M / 2) systolic matrix, thereby solving the matrix shape sensitivity problem of ASIC and always maintaining high PE utilization, where M and N are both positive integers.

[0131] For example, assuming the PE array is a 40×30 array, the target hardware configuration information can configure it as a logically 80×15 pulsed array. It can be understood that, in this application, under the constraint of the number of PEs in the PE array, the PE array can be reconstructed into an array of arbitrary size and shape.

[0132] Thus, since the hardware configuration information corresponding to the computational paradigm is parameterized, and the instantiation process only involves filling the computational parameters of the target computational task into the parameter field of the template, the arbitrary data flow graph mapping of the NP-complete problem in traditional CGRA is successfully reduced to a simple parameter selection and filling problem. This greatly simplifies the compiler design and compilation process, enabling extremely fast dynamic refactoring configuration generation, which can meet the needs of rapid iteration of AI algorithms.

[0133] It should be noted that after reconstructing the target PE region into a hardware structure that executes the target computing paradigm, the target computing task can be executed through the target PE region.

[0134] For example, "performing a target computation task through a target PE region" may include:

[0135] When the target computation paradigm includes the systolic array paradigm, in the reconstructed systolic array architecture, data can flow through each PE in a systolic manner, and each PE automatically performs multiplication and accumulation operations based on the data arriving at its input register.

[0136] When the target computing paradigm includes the SIMD paradigm, a single operation instruction can be broadcast to a cluster of PEs via a broadcast bus configured in a reconfigurable interconnect network. The cluster of PEs then synchronously executes the instruction and manipulates their respective data. It can be understood that at the same time, different PEs within the same cluster perform the same operation, but the data they manipulate may differ.

[0137] When the target computing paradigm includes the MIMD paradigm, each PE asynchronously retrieves and executes task instructions from its own local configuration memory (Context / instruction memory), and communicates and synchronizes through the reconstructed path topology of the reconfigurable interconnect network.

[0138] It is understood that, within the same time period, the aforementioned PE array can be reconfigured into a hardware structure that executes at least one computational paradigm. Different PE regions execute computational tasks corresponding to different computational paradigms.

[0139] In this embodiment, when the target computing task involves multiple computing paradigms, proximity computing can eliminate data movement between "architectural silos," ensuring that expensive computing resources maintain high utilization throughout all stages of the AI ​​workflow. Specifically, proximity computing can reduce data movement overhead by ensuring that data does not leave the PE array or the on-chip shared memory of the NPU architecture.

[0140] In some embodiments of this application, when the target computing paradigm includes at least two computing paradigms, the configuration and control subsystem can determine whether to load the instantiated hardware configuration information corresponding to these computing tasks in stages or load the hardware configuration information simultaneously, based on the data flow relationship between the target computing task and the computing tasks corresponding to each computing paradigm, and the operating status of the PE array. It can be understood that "loading" here refers to dynamically configuring the operating mode of the PE array and the path topology of the reconfigurable interconnect network based on the configuration in the instantiated hardware configuration information.

[0141] In some embodiments of this application, the target hardware configuration information may include PE configuration information and network topology configuration information;

[0142] Among them, PE configuration information can be used to configure the working mode of PE in the target PE area, and network topology configuration information is used to configure the path topology of the reconfigurable interconnection network.

[0143] In some embodiments of this application, mapping the target hardware configuration information to the target PE region can be: configuring the PE configuration information in the target hardware configuration information to each PE in the target PE region, so as to configure the working mode of these PEs to support the working mode of the target computing task.

[0144] Mapping the target hardware configuration information to the reconfigurable interconnect network can be done by configuring the path topology of the network area corresponding to the target PE area in the reconfigurable interconnect network to the topology form corresponding to the topology configuration information in the target hardware configuration information.

[0145] It is understandable that PE configuration information includes all configurations related to the PE.

[0146] Specifically, the PE configuration information in this application includes the PE's "identity setting configuration" and "operation setting configuration." The "identity setting configuration" can configure the computing paradigms supported by the PE, the function of the PE's local memory, the conduction status of each port in the PE, and some parameters shared with the PE in the target PE region under the supported computing paradigms. The "operation setting configuration" configures the specific operations performed by the PE under a specific identity. The PE's local memory may include local register files and local configuration memory.

[0147] For example, the "Identity Setting Configuration" can configure whether the PE is a PE in the MIMD or a PE in the pulse array; and can configure which port of the PE is used as the data input port, which port is used as the data output port, which PE the input data of the PE comes from, and which PE the operation result of the PE is used as the input of.

[0148] For example, such as Figure 4 As shown, the PE configuration information can configure which transport block 21 in the reconfigurable interconnection network the first multiplexer 15 in the PE is connected to, whether the first multiplexer 15 is connected to the input RF or the output RF in the PE's local register file, and what operation the PE performs.

[0149] In some embodiments of this application, some parameters shared by PE may include, but are not limited to: input / output data precision, weight data, etc.

[0150] For example, if the target PE array is configured as a pulsating array, then the PEs in the target PE region may share input / output data precision or weight data.

[0151] It is understandable that the "identity settings configuration" is configured statically, such as being written once and remaining unchanged until the target computing task is completed. The "operation settings configuration," on the other hand, varies depending on the identity of the PE.

[0152] For example, in the pulsating array paradigm: both the PE's identity and operation settings are configured statically, and the PE is configured with a fixed identity and unique operation. That is, the PE configuration information is the configuration information that fixes the PE's identity and unique operation. Specifically, the PE configuration information fixes the PE as a MAC unit node in the pulsating array. This includes:

[0153] Identity / Mode Configuration: Lock the PE to MAC operation mode;

[0154] Data path configuration: If the input data of a PE comes from a neighboring PE in the north / west, and the output data of the same PE is sent to a neighboring PE in the south / east, it can be understood that the interface configuration here can be the interface configuration inside the PE, such as the conduction configuration of the multiplexer inside the PE as described below.

[0155] Operation configuration: "Multiplication and accumulation" is executed unconditionally in each cycle. The operation configuration here is only for ease of understanding. In actual implementation, since each node (PE) in the pulsating array only performs MAC operations, the identity setting configuration can be set in the PE configuration information, without setting the "operation configuration" information, so as to reduce the configuration size.

[0156] It is understandable that for PEs in a pulsating array, operation settings are completed simultaneously with identity settings, so there is no need to distribute operation setting configurations again. In other words, for PEs in a pulsating array, there is no need to distribute dynamic operation commands.

[0157] For example, in the SIMD paradigm: the identity settings of PEs are configured statically, while the operation settings of PEs belonging to the same SIMD cluster are dynamic and uniformly configured. That is, the operation instructions of PEs within the same SIMD cluster are uniformly distributed. In other words, in the SIMD paradigm, the PE configuration information in the target hardware configuration information can configure at least one cluster of PEs as "SIMD lockstep execution units". The operation instructions of the PEs in each SIMD lockstep execution unit are uniformly distributed by the configuration and control subsystem, such as the global configuration manager described below, so that PEs within the same SIMD cluster execute the operation corresponding to the same operation instruction.

[0158] Understandably, in the SIMD paradigm, operation instructions for PEs within the same SIMD cluster are uniformly distributed, such as through broadcast instructions. This "broadcast instruction" essentially broadcasts a new "configuration word" to update the configuration of the "currently executed operation" in all PEs within a SIMD cluster. It is still a configuration behavior, characterized by synchronization, uniformity, and low frequency, such as broadcasting once per operator.

[0159] For example, in the MIMD paradigm: the identity configuration of a PE is configured statically, while the operational configuration of each PE is an independent instruction stream or instruction sequence. Specifically, the first step is to perform static PE identity configuration. This sets the PE as an "independent microprocessor core." This includes identity / mode configuration:

[0160] (1) Switch the PE’s local memory interface to instruction memory and enable its internal program counter, fetch and decode unit, etc.

[0161] (2) Data path configuration: Configure the network interface of the PE to support the general NoC for packet routing. It can be understood that the interface configuration here can be the interface configuration inside the PE, such as the multiplexer conduction configuration inside the PE as described below.

[0162] The second step is to configure each PE in the SIMD array with its own instruction sequence, so that each PE can independently execute the operation corresponding to the instructions in its own instruction sequence. It can be understood that the instruction sequence is configured after the PE is determined to be a SIMD PE; this instruction sequence can be deleted after the PE is configured as a non-SIMD PE.

[0163] In some embodiments of this application, the instruction sequences configured in different PEs may differ in the SIMD paradigm.

[0164] In some embodiments of this application, the above instruction sequence includes multiple task instructions, each task instruction indicating an operation (or operator). For example, for a PE that performs the MIMD paradigm, the PE independently and sequentially executes each operation indicated by the instruction sequence corresponding to the PE.

[0165] In some embodiments of this application, under the MIMD paradigm, PEs are configured to: fetch task instructions from their respective instruction memories and execute them, and communicate and synchronize data via a reconfigurable interconnect network. Thus, because PEs can be configured to asynchronously execute task instructions from their respective configuration memories and communicate and synchronize via a configurable interconnect network under the MIMD paradigm, the NPU architecture can handle algorithms with irregular data dependencies and complex control flows in a multi-instruction, multi-data (MID) manner. Therefore, the NPU architecture can overcome the limitations of traditional SIMD and systolic arrays, and efficiently handle emerging workloads such as graph neural networks.

[0166] Thus, since the target hardware configuration information includes PE configuration information for configuring the working mode of the PE in the target PE area and network topology configuration information for configuring the path topology of the reconfigurable interconnection network, the configuration and control subsystem can quickly switch the PE array and the reconfigurable interconnection network to an architecture that supports the target computing task based on the PE configuration information and the network topology configuration information, thereby improving the architecture switching speed.

[0167] In some embodiments of this application, configuring the working mode of the PE may include: configuring the operations performed by the PE, the data flow between PEs, etc.

[0168] In some embodiments of this application, the reconfigurable interconnect network can be used or configured to transmit data or PE configuration information between the PEs in the PE array.

[0169] In some embodiments of this application, the reconfigurable interconnect network can also be used to transmit configuration information for reconfiguring the path topology of the reconfigurable interconnect network. For example, the configuration and control subsystem can load the network topology configuration information in the target hardware configuration information into the corresponding network unit in the reconfigurable interconnect network, such as the switch and / or transmission unit described below, so that the path topology of the reconfigurable interconnect network is the same as the path topology indicated by the target hardware configuration information.

[0170] In some embodiments of this application, each of the above-mentioned PEs may include: a multifunctional arithmetic logic unit (ALU), which includes at least one of the following: a MAC subunit that performs MAC operations, an arithmetic logic subunit that performs arithmetic logic operations, and an SFU subunit that performs special function unit (SFU) operations for nonlinear activation.

[0171] In some embodiments of this application, "arithmetic logic operations" may also be referred to as "arithmetic and logic operations". Arithmetic and logic operations may include: addition, multiplication, subtraction, logic operations (AND, OR, Boolean operation XOR), comparison, and other operations required to support SIMD and MIMD paradigms.

[0172] In some embodiments of this application, the SFU operation may include: activation functions, such as exp(), sigmoid(), division functions, etc.

[0173] It is understandable that MAC operations correspond to the systolic array paradigm, arithmetic logic operations and SFU operations correspond to the MIMD paradigm, and arithmetic logic operations and SFU operations can also correspond to the SIMD paradigm.

[0174] Thus, because each PE in the PE array contains a multi-functional ALU that supports multiplication and accumulation, arithmetic logic, and special function operations, each PE has the ability to perform the core operations required for all computational paradigms. This allows the same PE hardware to seamlessly switch between performing MAC operations, performing element-level arithmetic logic operations, and performing special functions for complex activations. This is also the microscopic basis for PE arrays to be uniformly reconfigured into hardware for different computational paradigms.

[0175] In some embodiments of this application, such as Figure 3 As shown, the ALU 111 described above may include a MAC subunit, an arithmetic logic subunit, and an SFU subunit. Thus, since dedicated hardware subunits for performing MAC operations, arithmetic logic operations, or special function operations can be set in the ALU of each PE, the ALU in this embodiment can further improve the speed of operation switching and operational performance compared to a traditional general-purpose ALU.

[0176] The integration of multiple dedicated hardware architectures in related technologies not only significantly increases hardware cost, area, and design complexity, but more seriously, it introduces expensive data transfer and synchronization overhead between these independent "island" units. For example, after data is computed in a GEMM unit, it must be written to on-chip cache (such as Static Random-Access Memory, SRAM) or even main memory (such as DRAM), and then read by the corresponding SIMD hardware to execute the activation function. This on-chip and even off-chip data transfer is rapidly becoming a major performance and energy consumption bottleneck in AI computing, namely the "memory wall" problem.

[0177] In some embodiments of this application, such as Figure 2 As shown, each of the above PEs may also include a local memory 12. The local memory 12 is used to temporarily store the operands and results of the PE11 operation, as well as the PE configuration information.

[0178] In some embodiments of this application, such as Figure 2 As shown, each of the above PE11 may further include: a local register file (RF) 121 and a local configuration memory 122;

[0179] The aforementioned local register file 121 is used to temporarily store the operands and results of the PE11 operation;

[0180] The aforementioned local configuration memory 122 is used to store PE configuration information.

[0181] It is understood that the aforementioned local memory 12 includes local RF and local configuration memory 122.

[0182] In some embodiments of this application, the local register file is a cluster of local general-purpose registers, including input registers and output registers. The output registers, in particular, serve a multimodal function.

[0183] In some embodiments of this application, the local register file of each PE may include input registers and output registers; the input registers may be used to store the input data of the PE, i.e., operands.

[0184] In the pulsating array paradigm, the above output register is used to forward the operation results of PE11 to its neighbor PE11;

[0185] In the SIMD paradigm, the above output register is used to temporarily store the local vector data of PE11;

[0186] In the MIMD paradigm, the above output register is used to temporarily store local variables of PE11.

[0187] It should be noted that "PE's neighbor PE" refers to logical adjacency, not physical adjacency. In other words, a PE's neighbor PE can be physically adjacent, or it can be logically adjacent but not physically adjacent, but achieved through a reconfigurable interconnection network.

[0188] Thus, since the local register file of a PE includes input registers and output registers, and the specific uses of the output registers differ depending on the computation paradigm in which the PE operates, the output registers of the PE are used for different purposes. Specifically, the output registers of the PE forward results in the systolic array paradigm, temporarily store vector data in SIMD, and temporarily store variables in MIMD. Therefore, the output registers of the PE can flexibly adapt to the data flow pattern requirements corresponding to different computation paradigms.

[0189] In some embodiments of this application, the local configuration memory may also be referred to as local instruction / configuration memory (Context Memory): it is a small, fast local SRAM. This memory is the core of enabling the NPU architecture to quickly switch between multiple computing paradigms. The local configuration memory can be used to store PE configuration information or instruction sequences for each PE.

[0190] In some embodiments of this application, when the PE is reconfigured to execute a systolic array or SIMD paradigm, the local configuration memory stores the configuration information of the PE generated after instantiation; when the PE is reconfigured to execute a MIMD paradigm, the local configuration memory can store the instruction sequence corresponding to the PE.

[0191] For example, the PE configuration information above is used to configure: what operations the PE's ALU should perform, and which neighboring PE the data originates from.

[0192] For example, in a multi-context configuration mode, the local configuration memory can also store multiple simplified context configurations, such as hardware configuration information. Therefore, compared to the method of storing specific and complete operator streams in related technologies, it can save storage space and simplify the process of switching computational paradigms.

[0193] For example, in the MIMD paradigm, local configuration memory can be redefined as local instruction cache / memory, enabling PEs to independently acquire and execute task instruction streams. Here, "task instruction stream" and "instruction sequence" have the same meaning; both refer to a series of operations that a specific PE needs to perform under the MIMD paradigm. For instance, a PE might be configured with a task instruction sequence instructing the following operations in sequence: addition operation 1, addition operation 2, comparison operation 1. This task instruction sequence can be described as follows: first, perform two addition operations with operands: data group 1 (data 1, data 2) and data group 2 (data 3, data 4); then, perform a comparison operation with the operand being the result of the two addition operations.

[0194] In some embodiments of this application, such as Figure 4 As shown, local memory 12 may include two configuration memories and a local register file. These two configuration memories are: Local Configuration Memory (Local Context / instruction memory) and Shadow Backup Configuration Memory (Shadow Context / instruction memory). The local backup configuration memory can be used to back up the relevant configuration information of the PE to prevent configuration loss. Alternatively, the local backup configuration memory can also be used to store the configuration information of the PE in the next target computing task. For example, if PE1 is currently a PE in the MIMD array, but in the next target computing task, PE1 may be configured as a PE in a systolic array, then the local backup configuration memory can pre-load the configuration information of the PE in the systolic array.

[0195] In some embodiments of this application, the local memory of the PE is a core component for solving the "memory wall" problem under heterogeneous architecture in related technologies. In other words, through the local memory of the PE, the calculation results or intermediate data of the PE can be stored in the PE array so that the next PE with data association with the PE can use it, thereby reducing the overhead of data transfer between PEs.

[0196] In some embodiments of this application, such as Figure 2As shown, the PE may further include a local controllogic 13, a first multiplexer 14, and a second multiplexer 15. The local controllogic 13 is connected to the first multiplexer 14, the second multiplexer 15, a multi-function ALU unit 111, and a local configuration memory 122. The first multiplexer 14 and the second multiplexer 15 are connected to four connection blocks (CBs) 21 located in four directions of the PE. The ALU 111 is also connected to a local register file 121. The first multiplexer 14 is connected to both the local register file 121 and the local controllogic 13, and the second multiplexer 15 is connected to both the local register file 121 and the local controllogic 13. In this way, the local controllogic 13 can control the data flow and configuration flow of the PE by controlling the on / off state of the first and second multiplexers. The first multiplexer 14 is an input selector, and the second multiplexer is an output selector. That is, by controlling the conduction of the first multiplexer 14, the configuration of the PE and the input data are determined from which CB to be input, and by controlling the conduction of the first multiplexer 14, the output data or forwarding configuration of the PE is determined from which CB to be output.

[0197] For example, the first multiplexer 14 can be referred to as a 1 to 4 Mux, and the second multiplexer 15 can be referred to as a 4 to 1 Mux.

[0198] For example, it is possible to control the 1 to 4 Mux in PE11 to conduct with CB21 located north of PE11, and to control the 4 to 1 Mux in PE11 to conduct with CB21 located south of PE11.

[0199] Thus, since each PE also includes a local register file and a local configuration memory, the local register file provides a temporary storage space for data with extremely low latency, while the local configuration memory can store PE configuration information. This provides support for local computation of data at the hardware level, reduces data movement, and supports fast switching between multiple paradigms, thus solving the "memory wall" problem in related technologies.

[0200] In some embodiments of this application, such as Figure 2 As shown, under the MIMD paradigm, the local configuration memory 122 of PE11 is configured as an instruction memory;

[0201] In the systolic array paradigm or SIMD paradigm, the local configuration memory 122 of PE11 is configured to store the configuration information of the PE generated after instantiation.

[0202] In some embodiments of this application, under the MIMD paradigm, the above configuration and control subsystem can also be configured to write the task instruction sequence corresponding to the PE to the instruction memory.

[0203] In other words, the configuration and control subsystem can reconfigure the functionality of the PE's local configuration memory according to the computational paradigm executed by the PE, thereby adapting the PE's local configuration memory to the computational paradigm executed by the PE.

[0204] Thus, by limiting the PE's local configuration memory—serving as instruction memory in the MIMD paradigm and storing PE configuration information in the systolic / SIMD paradigm—efficient dual-mode utilization of the same physical storage resources of the PE is achieved. This avoids the area overhead of designing separate storage units for two purposes and ensures the PE's computing performance in different computing scenarios through hardware-level mode switching.

[0205] The NPU architecture provided in this application can execute known, regular data flows (such as GEMM and SIMD) in a static, high-density, and zero-overhead manner, just like an ASIC, and can also execute unknown, irregular data flows in a dynamic, asynchronous, and low-overhead manner, just like a MIMD processor. It has strong versatility and scalability.

[0206] It is understandable that reconfigurable interconnect networks can be reconfigured into networks that support the computational paradigms corresponding to the computational requirements of the target computational task, in order to adapt to the computational needs.

[0207] In some embodiments of this application, the path topology of the reconfigurable interconnect network can vary according to changes in the target computing paradigm or instantiated hardware configuration information.

[0208] In some embodiments of this application, the reconfigurable interconnect network described above can be configured to support any of the following:

[0209] 1) In the systolic array paradigm, it serves as a static direct path for systolic data forwarding between adjacent PEs. This allows data to be forwarded from one PE to an adjacent PE with extremely low latency, typically <1 clock cycle, perfectly simulating the rigid data flow of an ASIC systolic array.

[0210] 2) In the SIMD paradigm, this acts as a broadcast bus for broadcasting the same operation instruction to a cluster of PEs; this broadcast bus can be called the SIMD instruction / data broadcast bus. This allows the same operation instruction to be distributed to a cluster of PEs through a single operation instruction source, which is an efficient way to achieve the "single instruction" characteristic of SIMD. For example, an operation instruction can be called an operation configuration or an instruction word.

[0211] 3) In the MIMD paradigm, this serves as a routing network supporting communication between any two PEs. For example, this reconfigurable interconnect network can be a general and flexible routing network supporting point-to-point routing between any two PEs. For instance, this routing network can be a packet-switched Network on Chip (NoC). This allows any PE to asynchronously send data packets carrying routing information to any other PE, enabling flexible data flow triggering and inter-task communication under the MIMD paradigm.

[0212] In some embodiments of this application, "adjacent PE" refers to logical adjacency, that is, two PEs are adjacent in data flow. In other words, "adjacent PE" in this application can be physically adjacent PEs or physically non-adjacent PEs.

[0213] In some embodiments of this application, within the same time window, the operations performed by PEs in the same cluster under the SIMD paradigm are the same, but the PEs in the cluster can perform their operations asynchronously. That is, the operations are the same, but the operation data and operation progress can be different.

[0214] For example, the above-mentioned cluster of PEs can also be referred to as a group of PEs, and the two have the same meaning.

[0215] For example, in the MIMD paradigm, the configuration and control subsystem can configure at least one row of PEs in the PE array as a PE cluster that performs the same operation.

[0216] For example, in the MIMD paradigm, the configuration and control subsystem can configure at least one column of PEs in the PE array as a PE cluster that performs the same operation.

[0217] It is understandable that static direct-connect circuits can achieve low-latency pulsed flow, broadcast buses can achieve efficient instruction distribution, and general-purpose NoCs can support flexible point-to-point communication.

[0218] In some embodiments of this application, the broadcast bus can be a ring bus, through which the PE configuration information can be sequentially transmitted to each PE in the cluster of PEs. Because the ring bus allows configuration information to be transmitted sequentially, compared with the traditional star broadcast structure, it can significantly reduce the fan-out of global wiring and the difficulty of signal driving, thereby saving power consumption, improving signal integrity, and facilitating the expansion of large-scale PE arrays.

[0219] Thus, since the reconfigurable interconnect network can be configured as a static direct path, broadcast bus, or routing network, it can provide optimal data communication support for three distinct computing paradigms: systolic arrays, SIMD, and MIMD, thereby ensuring that each computing paradigm can operate under its optimal path topology.

[0220] In some embodiments of this application, such as Figure 1 As shown, the reconfigurable interconnection network 20 may include a plurality of transmission units 21, and the plurality of transmission units 21 are connected to at least two other transmission units 21, and each PE11 is connected to at least two transmission units 21.

[0221] In some embodiments of this application, the transmission unit can act as a transmission relay between PEs. For example, the transmission unit can forward data or configuration transmitted by one PE to any transmission unit or PE connected to it. Of course, how the transmission unit forwards data and configuration can be determined by the network topology configuration information issued to the transmission unit by the configuration and control subunit.

[0222] In some embodiments of this application, such as Figure 1 As shown, the reconfigurable interconnection network 20 may include: multiple Conection Blocks (CBs) 21 and multiple Switches 22, with each Conection Block 21 connected to two Switches 22;

[0223] Each of the aforementioned PEs is connected to four transmission units 21, and adjacent PEs 11 share a common transmission unit 21.

[0224] In some embodiments of this application, the configuration and control subsystem can enable or disable the transmission path between CBs connected by the switch by configuring the on / off state of the switch.

[0225] In some embodiments of this application, such as Figure 1 As shown, the NPU architecture 100 described above may also include: on-chip shared memory 40 and memory access unit 50;

[0226] The memory access unit 50 is connected to the on-chip shared memory 40 and the PE array, respectively.

[0227] The memory access unit 50 can be used for data transmission and address management between the on-chip shared memory 40 and the PE array.

[0228] In some embodiments of this application, the on-chip shared memory can be used to cache data or control information to be loaded into the PE array; or it can be used to cache the calculation results of some PEs in the PE array so that the calculation results can be loaded into other PEs in the PE array for further calculation.

[0229] In some embodiments of this application, the PEs in the PE array described above can be referred to as computing PEs, i.e., processing elements with computing capabilities. The memory access units described above can be referred to as memory processing elements (Memory PEs, MPEs); an MPE is an intermediate data transfer module between the on-chip shared memory and the PE array. An MPE does not contain any computing resources, does not change data values, and has functions such as data prefetching and physical address management.

[0230] In some embodiments of this application, the on-chip shared memory may include a data cache area and a configuration cache area.

[0231] For example, the configuration and control subsystem can preload multiple instantiated hardware configuration information into the on-chip shared memory configuration cache for fast loading. This further improves the speed of switching between computational paradigms because it eliminates the need to load the instantiated hardware configuration information from external memory.

[0232] In some embodiments of this application, combined with Figure 1 As shown, there are multiple memory access units 50; these multiple memory access units are distributed at the edge of the PE array, and each row or column of PE in the PE array corresponds to at least one memory access unit.

[0233] It is understandable that the number of memory access units 50 is related to the number of rows and columns of the PE array, as well as the amount and distribution of on-chip memory.

[0234] For example, such as Figure 3 As shown, the NPU architecture includes three on-chip shared memory modules 40, located on the left, right, and bottom sides of the PE array, respectively. Each row of PEs is connected to two on-chip shared memory modules 40 located on the left and right sides of the PE array via two memory access units 50, and each column of PEs is connected to the on-chip shared memory module on the bottom side of the PE array via one memory access unit 50. Of course, in actual implementation, a larger on-chip shared memory module can also be used, with each row and column of PEs in the PE array connected to this on-chip shared memory module via its respective memory access unit. The number and location of the on-chip shared memory modules in this application are for illustrative purposes only and do not constitute a structural limitation on the NPU architecture provided in this embodiment.

[0235] Thus, since the NPU architecture can also include on-chip shared memory and a dedicated memory access unit, and the memory access unit can be used to manage data transfer and address management between the PE array and the on-chip shared memory, the on-chip shared memory provides a high-speed, shared data buffer for the PE array. This effectively alleviates external memory bandwidth pressure, reduces data access latency, and provides the necessary high-bandwidth memory support for large-scale data parallel computing.

[0236] In some embodiments of this application, the NPU architecture has a multi-layered memory structure, consisting of the PE's local memory and on-chip shared memory.

[0237] In some embodiments of this application, the on-chip shared content described above can be connected to off-chip memory.

[0238] The NPU architecture provided in this application, through the dynamic reconfigurable design of the PE array and reconfigurable interconnect network, achieves the following key technical improvements:

[0239] (A) Improve end-to-end hardware utilization.

[0240] In related technologies, dedicated NPU architectures (such as systolic arrays) have their expensive main computing units (MAC arrays) completely idle when executing operators they are not good at (such as activation functions).

[0241] The PE array in this application is "all-around". When performing GEMM, 100% of the PEs in the PE array can be configured as a systolic array; when performing ReLU, a portion of the PEs in the PE array can be quickly reconstructed into a SIMD array. This "transformation" capability ensures that expensive computing resources maintain high utilization at all stages of the AI ​​workflow, thereby greatly improving end-to-end performance and energy efficiency. It should be noted that the reason why a portion of the PE array can be quickly reconstructed is that this application instantiates the hardware configuration information corresponding to the computing paradigm plus the computing parameters of the specific computing task, replacing the operator full compilation and mapping schemes in related technologies, such as DFG mapping.

[0242] (B) Eliminate data transfer between "architectural silos".

[0243] Related technology: Data is transferred between different hardware accelerators (such as NPU architecture - Core and VPU / CPU) via on-chip bus or on-chip shared memory, resulting in high data transfer overhead. The NPU architecture in this application, however, can execute all operations of a computing task on the same PE array.

[0244] For example, consider a target computation task that includes systolic array computation and SIMD computation. The computation results of GEMM (systolic array mode) are directly stored in the local RF of the PE, without requiring the PE array. Subsequently, the PE array switches to SIMD mode in place and directly performs activation functions on the data in the local RF of the PE. This completely eliminates the communication overhead between "islands" and greatly reduces data transfer energy consumption and latency.

[0245] (C) Reduce the complexity of compilation and refactoring.

[0246] 1. For compilation

[0247] In related technologies, as discussed in section 2.3 above, the general CGRA faces the challenge of NP-complete compilation.

[0248] This application addresses this problem by targeting a specific domain application. It is not a general CGRA that requires mapping arbitrary DFGs. Instead, it provides optimized hardware support and predefined configuration templates for three specific paradigms in the AI ​​domain (Systolic, SIMD, MIMD). This successfully reduces the NP-complete "arbitrary DFG mapping" problem to a simple "selection and parameterization" problem. The compiler no longer needs to design a dataflow graph from scratch, but instead selects a pre-optimized hardware template (e.g., map_systolic(M, N, K)) and instantiates it to a specific region of the PE array. Since the configuration is template-based, this architecture can reuse these configurations through fast, partial, and dynamic refactoring, achieving switching speeds close to those of multi-context CGRAs while avoiding their high hardware costs.

[0249] Specifically, in this application, the same type of computational task corresponds to the same hardware configuration information. By inputting different parameters at runtime (such as matrix size M, N, K, and data flow direction), it can be instantiated into countless specific operators. That is, by using hardware configuration information, the traditional CGRA DFG mapping problem is simplified to hardware configuration information loading + parameterization, thereby greatly reducing compilation complexity, compilation volume, and compilation time.

[0250] For example, in traditional CGRA: a 3×3 convolution requires a complete operator configuration, a 5×5 convolution requires a completely different operator configuration, and a fully connected layer requires yet another operator configuration. The number of operator configurations increases almost linearly with the number of operator types. An "operator" can be understood as an "operation".

[0251] In this application, all GEMM / convolution operations, regardless of shape or size, share the same "systolic array" hardware template; all element-level operations (such as ReLU, GELU, Add, Mul, etc.) share the same "SIMD" template (the only difference being the specific operands executed by the ALU). This simplifies the compiler's task from "generating thousands of unique configuration bitstreams" to "calling a few templates from the hardware configuration information library and passing parameters according to the computation task."

[0252] 2. Regarding the restructuring.

[0253] Related technologies: See the relevant descriptions in section 2.3 above. CGRA in related technologies suffers from high runtime reconfiguration overhead, such as large reconfiguration latency for single contexts and large on-chip or intra-chip memory requirements for multiple contexts. It should be noted that for multiple contexts, since traditional CGRA stores specific operators, such as 3x3 convolution operators, 5x5 convolution operators, and multiple activation operators, even if all operator configurations are pre-stored locally in the PE, addressing and reading the entire bitstream from a large local repository is still required during switching, resulting in long retrieval times. Furthermore, the switching of traditional CGRA is global; for example, the PE array is either a systolic array, a SIMD array, or a MIMD array.

[0254] In this application, there is no need to cache the complete configuration bit stream. Only a small amount of hardware configuration information (such as systolic array template, SIMD template, MIMD template, etc.) needs to be loaded and instantiated according to the needs of the computing task to realize the reconstruction of the PE array. Therefore, there is no need to set a large SRAM locally on the PE, and only a few hardware configuration information and parameters of the specific computing task need to be loaded, so the reconstruction overhead is very small.

[0255] Furthermore, the PE array in this application can be reconstructed in blocks, such as by independently reconstructing a portion of the path topology of a reconstructable interactive network through DPR, thereby enabling different PEs in the PE array to be reconstructed into hardware structures that execute different computing paradigms.

[0256] (D) Native support for emerging AI workloads.

[0257] Pulsating arrays and SIMD in related technologies cannot efficiently handle emerging models with control flow intensive and data dependency irregularities.

[0258] The NPU architecture in this application is a general-purpose, reconfigurable computing substrate. This architecture design enables the NPU architecture to efficiently execute emerging AI workloads, as well as a variety of computationally intensive tasks beyond AI workloads.

[0259] For example, the NPU architecture can be extended to the following key scenarios:

[0260] 1) Perform advanced signal processing, such as 5G, 6G and radar.

[0261] The PE array in this application is well-suited for executing various kernels in traditional digital signal processing (DSP).

[0262] For example, in 5G / 6G baseband processing, radar signal processing, or advanced image / video encoding and decoding, there are a large number of data stream-intensive computations (such as FFT, filters, etc.). The NPU architecture in this application can be configured as a dedicated SIMD pipeline or spatial data stream mode to handle these computational tasks, providing higher energy efficiency and throughput than general-purpose DSPs or FPGAs.

[0263] 2) Perform multi-tasking concurrency and hardware virtualization, such as cloud computing and automotive computing.

[0264] The NPU architecture in this application supports DPR capability, allowing the PE array to be spatially divided into multiple independent computing regions. This means that the same NPU architecture can run multiple different computing paradigms simultaneously.

[0265] For example, in the NPU architecture of a cloud server: the NPU architecture can be virtualized to serve multiple tenants simultaneously. One PE area in the NPU architecture runs GEMM (Pulse Array Paradigm), and another PE area runs data analytics (MIMD Paradigm), which can greatly improve the utilization of hardware resources in the data center.

[0266] For example, in the field of autonomous driving: one PE region can run perception models (such as Lidar point cloud processing), while another PE region concurrently executes path planning or sensor fusion algorithms, realizing the efficient integration of heterogeneous tasks on the same NPU architecture.

[0267] 3) Perform high-performance computing (HPC) and scientific simulations.

[0268] HPC workloads are also highly heterogeneous, including both dense linear algebra (such as BLAS) suitable for systolic array patterns and a large number of analog kernels with irregular data access and complex control flow suitable for the MIMD paradigm. Under the MIMD paradigm, the NPU architecture can act as a high-efficiency dataflow processor to handle dynamic graph analysis or sparse computation tasks that are difficult for traditional accelerators to handle.

[0269] 4) As a general-purpose computing acceleration to replace traditional general-purpose accelerators. In complex SoCs, the NPU architecture provided in this application has the potential to "fold" multiple independent accelerators (such as NPU architecture, GPU, DSP, security engine).

[0270] For example, when the system needs graphics rendering, the PE array can be configured as a hardware structure using the SIMD computing paradigm; when AI computing is required, it can be configured as a hardware structure using a systolic array; and when complex control flow encryption algorithms need to be run, it can be configured as a hardware structure using the MIMD paradigm. This concept of a "computing resource pool" can significantly reduce chip area, cost, and power consumption.

[0271] In this application, each PE in the PE array supports MAC operations, arithmetic logic operations, and SFU operations for nonlinear activation. Furthermore, the path topology of the interconnect network connected to the PE array can be reconstructed, thus providing native hardware support for the MIMD paradigm. For example, in the MIMD paradigm, each PE can execute instructions asynchronously and independently, and synchronization between PEs is achieved through a data flow mechanism. This enables the NPU architecture to efficiently handle complex scenarios that fixed ASICs cannot handle, such as graph neural networks (GNNs), Transformer attention mechanisms, reinforcement learning, or multi-task concurrency.

[0272] Furthermore, since the hardware configuration information in this application is parameterized, it provides support and expansion capabilities at the software level to ensure that new computing paradigms involved in future new AI models are included.

[0273] In some embodiments of this application, such as Figure 1 As shown, the NPU architecture 100 described above may further include: a data access controller 70; wherein the data access controller 70 is connected to the off-chip memory 60 and the on-chip shared memory 40, respectively.

[0274] In some embodiments of this application, the data access controller may be a Direct Memory Access (DMA) controller.

[0275] Thus, since the NPU architecture also includes a data access controller, which can handle data transfer between off-chip memory and on-chip shared memory within the NPU architecture, large data transfers can be efficiently performed by dedicated hardware, relieving the burden on computing units and the processor. This allows operations such as data prefetching and result write-back to fully overlap with the computation process, further hiding memory access latency and improving overall system efficiency.

[0276] In some embodiments of this application, the NPU architecture has a layered configuration and control subsystem, which includes: a global configuration manager and a dynamic partial reconfiguration (DPR) controller. The DPR controller can be used to dynamically configure the path topology of the reconfigurable interconnection network, and the global configuration manager can be used to dynamically configure the operating mode of the PE array.

[0277] In some embodiments of this application, the global configuration controller and the DPR manager can work asynchronously.

[0278] In some embodiments of this application, such as Figure 1 As shown, the configuration and control subsystem 30 may include: a global configuration manager 31 and a DPR manager 32. The global manager 31 is connected to each PE11 in the PE array, and the DPR manager 32 is connected to the reconfigurable interconnection network.

[0279] DPR Manager 32 is configured as follows:

[0280] Load the target hardware configuration information based on the reconfiguration request corresponding to the target computing task;

[0281] Based on the network topology configuration information in the target hardware configuration information, dynamically configure the path topology in the reconfigurable interconnection network 20 corresponding to the target PE area;

[0282] Global Configuration Manager 31 is configured as follows:

[0283] Based on the PE configuration information in the target hardware configuration information, dynamically configure the working mode of the PE in the target PE area.

[0284] It should be noted that after completing the path topology configuration of the reconfigurable Internet path and the working mode configuration of the PE, the target PE region can execute the computing tasks corresponding to the target computing paradigm in the target computing tasks, such as MIMD tasks or pulsating array tasks.

[0285] In some embodiments of this application, a communication channel exists between the DPR manager and the global configuration manager.

[0286] In some embodiments of this application, the global configuration manager described above can be an on-chip controller. It is responsible for retrieving the hardware configuration information and computational parameters required for the computing task from main memory, loading and instantiating them into the local configuration memory of the PE array to configure the operating mode of the PE array. In other words, the global configuration manager can configure the programs within each PE, such as what operations the PE should perform and the flow of operation data.

[0287] For example, the global configuration manager can be a small RISC-V core.

[0288] For example, the global configuration manager can include multiple cluster controllers, each of which controls a portion of the PEs in the PE array.

[0289] In some embodiments of this application, the DPR manager allows the NPU architecture to independently and dynamically reconstruct sub-regions (Regions) within the PE array, such as the pathway topology of the sub-regions, without halting the operation of the entire PE array. This is crucial for enabling AI model pipelines (e.g., one region running GEMM, adjacent regions running activation) and multi-task concurrency.

[0290] In some embodiments of this application, the DPR manager may include: a first controller, an on-chip configuration cache, and a first configuration network.

[0291] The first controller, on-chip configuration cache, and first configuration network are described below.

[0292] First, the primary controller is the "brain" of the DPR manager, implemented by a hardware state machine or a RISC-V CPU. Combined with... Figure 5 The first controller in DPR mainly performs the following functions:

[0293] 1. Receive reconstruction requests from the main control unit (such as the CPU) or the load scheduler, such as the reconstruction request corresponding to the target AI load mentioned above;

[0294] 2. Perform task management. Manage the partitions of the PE array. For example, the first controller acts as a task manager and arbiter: it maintains a resource table through which it can query which PEs in the PE array are idle or running, as well as the task identifier of the currently running task, etc. In some embodiments of this application, the DPR can provide the resource table to the compiler.

[0295] 3. Physical Address Generation: Calculate the physical address of the reconstructed PE region, such as the physical address of the target PE region;

[0296] 4. State Control. Specifically, the first controller can control the state of the configuration stream, such as controlling whether the configuration bit stream is in a "start" state or a "stop" state. When the configuration bit stream is in the start state, it begins transmitting through the first configuration network to the corresponding PE region; when the configuration bit stream is in the stop state, it indicates that the transmission has stopped. Furthermore, the first controller can control the operating state of the PE region, such as controlling whether the PE region is in an "activate" state or an "idle" state.

[0297] For example, the first controller can control the distribution of configuration streams corresponding to PEs and / or reconfigurable interconnects, as well as the activation of the corresponding PEs. For instance, the configuration stream might begin distribution at one time and end at another. The configuration stream might only be activated and take effect at a subsequent time.

[0298] II. On-chip Configuration Cache. This is a specialized on-chip SRAM cache used to store configuration bitstreams, such as instantiated hardware configuration information. This SRAM cache stores the most frequently used hardware configuration information, such as an 8×8 systolic array configuration or a 4-lane SIMD configuration template. When the first controller receives a reconfiguration request, it first searches the on-chip configuration cache for hardware configuration information that matches the AI ​​workload. If no match is found, the hardware configuration information is retrieved from external storage via DMA. If the requested hardware configuration information is found in a commonly used hardware configuration module, it can be retrieved directly from the on-chip configuration cache to improve configuration speed and reduce configuration latency.

[0299] III. First Configuration Network. A dedicated network independent of the data interconnect network, used to efficiently send configuration bit streams from the on-chip configuration cache to the corresponding target PE region, such as the first PE region and / or the second PE region mentioned above. The first configuration network is typically implemented as a chained network.

[0300] It should be noted that the aforementioned "DPR manager includes a first configuration network" actually refers to the first configuration network corresponding to the DPR controller. This first configuration network includes: the DPR manager, S and CB in the reconfigurable Internet path, and a configuration transmission path set between the DPR manager, S, and CB. The transmission path is as follows: Figure 1 As shown by the "dashed arrow" in the image.

[0301] In some embodiments of this application, the global configuration manager also corresponds to a configuration network independent of the data interconnection network, hereinafter referred to as the second configuration network. For example... Figure 1As shown, the second configuration network may include a global configuration manager, a PE array, and a dotted-line arrow positioned between the global configuration manager and the PEs. In this way, the global configuration manager can configure the corresponding PE in the PE array through the configuration network corresponding to the dotted-line arrow.

[0302] In some embodiments of this application, the data transmission path corresponding to the aforementioned data interconnection network includes: Figure 1 The solid arrow in the diagram indicates the path.

[0303] It should be noted that, Figure 1 The connection path between the DPR manager, global configuration manager, and on-chip shared cache is omitted. In actual implementation, DPR can connect to the on-chip shared cache through the first configuration network and MPE.

[0304] It is understandable that, since the global configuration manager and the DPR manager can perform operations asynchronously, the DPR manager can reconstruct the path topology corresponding to another PE region in advance without stopping operation in some regions of the PE array, thereby further reducing the reconstruction latency of inter-calculation paradigm reconstruction.

[0305] Thus, since the configuration and control subsystem includes a global configuration manager and a DPR manager, and the DPR manager is responsible for loading hardware configuration information and dynamically configuring the path topology in the reconfigurable interconnect network corresponding to the target PE area according to the network topology configuration information in the target hardware configuration information; and the global configuration manager is responsible for configuring the working mode of the PE in the target area, hardware-level division of labor and collaboration of configuration tasks are realized, which improves the speed and flexibility of the PE array to switch computing paradigms.

[0306] Furthermore, due to the DPR manager's ability to independently configure the path topology, dynamic partial reconstruction of the network can be achieved. This allows the NPU architecture to update the path topology of another PE region in advance without interrupting the computation of one PE region, greatly improving the operational flexibility and resource utilization of the NPU architecture.

[0307] In some embodiments of this application, the target computing task described above may include at least one computing task;

[0308] The above configuration and control subsystem can be specifically configured as follows:

[0309] Based on the reconstruction request corresponding to the target computing task, load at least one hardware configuration information that corresponds one-to-one with the at least one computing task;

[0310] Based on at least one hardware configuration information, at least one PE region of the PE array is dynamically reconstructed into a hardware structure for executing the at least one computing task.

[0311] Based on the data flow relationship between the at least one hardware configuration information, the path topology between the at least one PE region is dynamically reconstructed.

[0312] It should be noted that each of the above-mentioned computing tasks corresponds to at least one computing paradigm, and these at least one computing paradigm may be entirely or partially different. The above-mentioned hardware configuration information constitutes a template chain.

[0313] For example, the target computation task includes a third computation task and a fourth computation task, whereby the third computation task corresponds to a third computation paradigm, and the fourth computation task corresponds to a fourth computation paradigm, and the third and fourth computation paradigms are different. In other words, the aforementioned target computation paradigm includes both a third and a fourth computation paradigm.

[0314] In some embodiments of this application, in the operator chain process, if there is data association between at least one of the above-mentioned computing tasks, the path topology between at least one of the above-mentioned PE regions in the reconfigurable configuration network can be dynamically reconstructed so that the reconstructed path topology is adapted to the data association between at least heterogeneous computing tasks.

[0315] For example, consider a target computing task that includes a third computing task and a fourth computing task. If the output data of the third computing task is the input data or part of the input data of the fourth computing task, then the data output terminal of the first PE region can be connected to the data input terminal of the second PE region by configuring the path topology of the reconfigurable interconnection network.

[0316] For example, consider a target computing task that includes a third computing task and a fourth computing task. If the input data or part of the input data of the third computing task is the output data of the fourth computing task, then the data input end of the first PE region can be connected to the data output end of the second PE region by configuring the path topology of the reconfigurable interconnection network.

[0317] It is understood that when the target hardware configuration information includes at least one piece of hardware configuration information, the target hardware configuration information is similar to a template chain, and the target hardware configuration may also include configurations that indicate the data flow between at least one piece of hardware configuration information.

[0318] In some embodiments of this application, "dynamically reconstructing the path topology between at least one PE region based on the data flow relationship between at least one hardware configuration information" can be: the configuration and control subsystem can configure the path topology of the network portion located between at least one PE region in the reconstructable interconnection network to the optimal topology form based on nearest neighbor calculation and the data flow relationship between at least one hardware configuration information, so as to improve data transmission performance.

[0319] For example, taking a target computing task that includes a third computing task and a fourth computing task, with the hardware configuration information corresponding to the third computing task being configured in the first PE region and the hardware configuration information corresponding to the second computing task being configured in the second PE region, the reconfigurable interconnection network can be configured as a path topology with the shortest transmission path between the first PE region and the second PE region.

[0320] For example, such as Figure 9 As shown, the first PE region is the first three rows of the PE array, and the second PE region is the fourth row of the PE array. Therefore, the configuration and control subsystem can, based on nearest neighbor calculation, configure the south-facing port of the third row of PEs in the first PE region as an output port, and connect it to the north-facing port of the PEs in the same column in the second PE region. The north-facing port of the fourth row of PEs is configured as an input port. In this way, data in the output RF of the third row of PEs can flow into the input RF of the fourth row of PEs through a north-to-south transmission path.

[0321] It should be noted that the above example illustrates the nearest neighbor calculation performed by the configuration and control subsystem. In actual implementation, the compiler can also perform the nearest neighbor calculation and add the configuration of the planned transmission path between PE regions to the network topology configuration information of the target hardware configuration information. In this case, the configuration and control subsystem only needs to perform configuration and does not need to perform calculation.

[0322] In some embodiments of this application, taking the target computing task as including a third computing task and a fourth computing task as an example, where the third computing task corresponds to a third computing paradigm and the fourth computing task corresponds to a fourth computing paradigm, then: the third computing paradigm and the fourth computing paradigm can be any of the following:

[0323] The third computational paradigm is the SIMD paradigm, and the fourth computational paradigm is the pulsating array paradigm.

[0324] The third computational paradigm is the MIMD paradigm, and the fourth computational paradigm is the SIMD paradigm.

[0325] The third computational paradigm is the MIMD paradigm, and the fourth computational paradigm is the pulsating array paradigm.

[0326] In some embodiments of this application, the DRP manager can be used to dynamically configure the path topology between PEs in at least one PE region, as well as the path topology between different PE regions.

[0327] In some embodiments of this application, the working mode of PE in at least one PE region can be configured through a global configuration manager.

[0328] In some embodiments of this application, the different PE regions within the at least one PE region may not overlap. However, in actual implementation, different PE regions may partially overlap. For example, if the third and fourth computational paradigms have a sequential operation order, the PE regions corresponding to the third and fourth computational paradigms may overlap.

[0329] It should be noted that the data flow relationship between the above-mentioned at least one hardware configuration information in the above embodiments needs to be determined with the data flow relationship between the above-mentioned at least one computing task.

[0330] Thus, when the target computing task includes at least one computing task corresponding to different computing paradigms, the configuration and control subsystem can reconfigure the partitions in the PE array into hardware structures that execute different computing paradigms. Based on the data flow relationship between at least one computing task, it configures a reconfigurable interconnect network to establish transmission paths between at least one PE region. This achieves tight coupling and pipelined operation between the working areas of different computing tasks at the hardware level, eliminating the data transfer overhead between different dedicated hardware units in traditional heterogeneous architectures. This allows the computation results of the previous computing task to directly flow to the working area of ​​the next computing task for processing, achieving ultimate operator fusion and energy efficiency improvement. Operator fusion refers to the fusion of operator chains of at least two computing tasks.

[0331] The complete structure of the NPU architecture provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0332] like Figure 1 As shown, the NPU architecture in this application can be an improved CGRA-based NPU architecture. The core of this NPU architecture is a PE array-based CGRA.

[0333] In some embodiments of this application, the NPU architecture mainly includes a PE array, a flexible reconfigurable interconnect network, and a hierarchical configuration and control subsystem.

[0334] 1. The PE array described above. The PE array can be a two-dimensional array consisting of N×M PEs.

[0335] The Processing Entity (PE) is the basic computing unit in the NPU architecture. Each PE is designed as a lightweight yet fully functional processing core, containing the following key components:

[0336] 1a. Multifunctional ALU, also known as: Multifunctional FU: Each PE contains a multifunctional arithmetic logic unit. This multifunctional ALU must natively support all the key operations involved in multiple computing paradigms.

[0337] For example, the multi-functional ALU of each PE supports the following key operations:

[0338] 1) MAC operation: used for efficient execution of GEMM and convolution in systolic array mode.

[0339] 2) Arithmetic and logical operations, such as addition, subtraction, logical operations (AND, OR, XOR), comparison, etc. required for SIMD and MIMD paradigms.

[0340] 3) Special function operations: A small SFU can be integrated to accelerate complex operations such as nonlinear activation functions (e.g., exp(), sigmoid()) or division at the hardware level, further improving the computational efficiency of the SIMD / MIMD paradigm.

[0341] 2. Local Storage: The PE's local storage is crucial for achieving "on-premises computing" and low-latency data streaming. In other words, local storage is a key design feature for reducing "data transfer overhead." Local storage can consist of two parts:

[0342] 2a. Local Register File: A set of local general-purpose registers, including input registers and output registers. The output registers, in particular, function in a multimodal manner:

[0343] In the pulsating array paradigm, it acts as a pipeline buffer for forwarding data to neighboring PEs;

[0344] In SIMD mode, it stores local vector data;

[0345] In MIMD mode, it acts as the CPU's register file, used to temporarily store local variables.

[0346] 2b. Local configuration memory, also known as local instruction / configuration memory (Context Memory): This is a small, fast local SRAM. This memory is core to enabling the NPU architecture's ability to quickly switch between multiple computing paradigms. Local configuration memory can be used to store PE configuration information.

[0347] 3. Reconfigurable interconnect network: It is the "data path" connecting the PE array and a key enabling technology for achieving "dynamic reconfigurability".

[0348] 3.1 Reconfigurable interconnected networks must support dynamic routing and be configurable by the configuration manager into three distinct topology modes (i.e., topology forms) to support different computing paradigms.

[0349] For example, a reconfigurable interconnect network can support the following topology patterns:

[0350] 3a. For Systolic Neighbor Data Forwarding: The network is configured as a static, direct path between PEs. This allows data to be forwarded from the output RF of one PE to the input RF of an adjacent PE with extremely low latency (typically <1 clock cycle), perfectly simulating the rigid data flow of an ASIC systolic array.

[0351] 3b. For SIMD instruction / data broadcast bus: The network is configured as a broadcast network of a cluster of PEs, such as a broadcast network of PEs within a row or column. This allows a single source of operational instructions to simultaneously distribute the same operational instructions to the same cluster of PEs, which is an efficient way to implement the "single instruction" characteristic of SIMD.

[0352] 3c. General-purpose point-to-point routing network for MIMD: The network is configured as a general-purpose, flexible routing network, such as a packet-switched network-on-chip (NoC) network. This allows any PE to asynchronously send data (data packets carrying routing information) to any other PE to enable data flow triggering and inter-task communication in MIMD mode.

[0353] 4. Configuration and Control Subsystem. This configuration and control subsystem is the "brain" of the NPU architecture, responsible for loading and instantiating hardware configuration information, and coordinating the execution of the PE array. The configuration and control subsystem may include:

[0354] 4.1 Global Configuration Manager: This can be an on-chip controller (e.g., a small RISC-V core) responsible for acquiring the configuration bitstream required by the application (such as the target computing task), such as the PE configuration information mentioned above, and configuring it into the local configuration memory (ContextMemory) of the PE array through a second configuration network. Simply put, the global configuration manager is used to configure the programs within each PE in the PE array.

[0355] 4.2 DPR Manager: The DPR manager allows the NPU architecture to independently and dynamically reconstruct sub-regions within the PE array without halting the entire PE array, such as reconstructing the path topology corresponding to a sub-region. This is crucial for enabling AI model pipelines (e.g., one region running GEMM, adjacent regions running activation) and multi-task concurrency. A DPR can include a controller (as described in the first controller above), an on-chip configuration cache, and a configuration network (as described in the first configuration network above).

[0356] 4.3 Multi-paradigm instruction distribution mechanism: The control instruction distribution mechanism in this application varies depending on the target computation paradigm.

[0357] For example, in the Systolic paradigm: no instruction dispatch. The PE enters the data trigger state immediately after the systolic array template is loaded. Once data arrives at the PE's input RF, the MAC operation is automatically executed.

[0358] In the SIMD paradigm: a centralized instruction distribution mechanism. The global configuration manager or a dedicated cluster controller can broadcast the same instruction to a group of target PEs via a broadcast bus. The cluster controller can be a sub-controller within the global configuration manager.

[0359] In the MIMD paradigm: a decentralized instruction dispatch mechanism is adopted. Each PE independently retrieves instructions (such as the instruction sequence mentioned above) from its local configuration / instruction memory, achieving asynchronous execution.

[0360] 5. Layered memory architecture

[0361] For efficient operation, the memory architecture in an NPU architecture can rely on a multi-layered memory structure.

[0362] Combination Figure 1 ,like Figure 6 As shown, a layered memory structure can include:

[0363] 5.1 L0-PE Local Register (LocalRF) file: Fastest access speed, extremely small capacity, used for direct operands of PE.

[0364] 5.2 L1 - On-chip shared memory, also known as global shared memory (SHM): This consists of multiple large on-chip SRAM buffers. It is the main "working area" of the PE array. It is much larger than L0, but much faster than off-chip shared memory.

[0365] 5.3 L2 - External Double Data Rate Memory (DDR): Off-chip main memory. It has the largest capacity but the slowest speed.

[0366] It is understandable that the above-described layered memory architecture can enable multi-level memory data interaction:

[0367] First data layer interaction: Data exchange between SHM and PE local register files. The two exchange data via MPE.

[0368] The second layer of data interaction: data interaction between SHM and external DDR. The DMA controller loads data (such as the weights of the entire model) from DDR into the on-chip SHM of the NPU architecture.

[0369] The third layer of data interaction: data interaction between multiple SHMs. DMA is not only responsible for data interaction between DDR and SHM, but also has independent data channels for data interaction between adjacent SHMs.

[0370] The NPU architecture provided in this application embodiment will be further explained below with specific reconstruction examples.

[0371] Example 1: such as Figure 7 As shown, the target computation paradigm is the pulsating array paradigm.

[0372] When AI workloads encounter computationally intensive tasks such as GEMM or convolution, the NPU architecture will be configured as a systolic array paradigm.

[0373] 1. Mapping Mechanism and Principle

[0374] This application employs a mapping strategy of "Virtual Systolic Array (VSA)". The compiler first analyzes the cyclic dependencies of GEMM / convolution and generates a logically optimal systolic array topology (VSA), i.e., a systolic array template. Subsequently, the compiler maps this VSA onto a physical PE array to obtain the target hardware configuration information. Specifically, PEs in the target PE region of the PE array can be assigned as a simulated node (Systolic PE) in the VSA, while the reconfigurable interconnect network is configured to simulate the logical data flow path between VSA nodes.

[0375] 2. Hardware Configuration Details

[0376] 2.1 PE Configuration: All PEs in the PE array (or a DPR area, i.e., a PE area) are configured by the global configuration manager to perform a single, fixed MAC operation.

[0377] 2.2 Configurable Interconnect Network Configuration: Configurable interconnect networks are configured by the DPR manager as strict, unidirectional nearest neighbor connections. It should be noted that all common routing logic in configurable interconnect networks is bypassed to achieve minimal forwarding latency. For example, data from the output RF of the North / West PE enters the network and is then passed by the network to the input RF of the South / East PE.

[0378] 2.3 Local Register File Usage of PE: In the systolic array paradigm, the local RF of the PE acts as a pipelined register. The PE reads operands from a neighbor's RF, performs MAC, and writes the calculation result (or forwarded input) to its output RF for the next neighbor PE to read in the next cycle. This perfectly replicates the "heartbeat" data flow of the systolic array.

[0379] 2.4 Data flow between PEs: By configuring different interconnect templates and input / output (I / O) interfaces, multiple data flow modes can be flexibly supported to adapt to the optimal data reuse requirements of different layers (such as convolution vs. fully connected).

[0380] In some embodiments of this application, I / O interface is a generic term for various components in the NPU architecture. For example, the I / O interface of the PE can be configured, such as dynamically configuring the conduction paths of the first multiplexer and the second multiplexer, for example, configuring the first multiplexer to conduct with the CB to the north of the PE, and configuring the second multiplexer to conduct with the CB to the right of the PE. Similarly, an input terminal in a CB can be configured to conduct with the north-facing output terminal of a PE connected to that CB, and an output terminal in that CB can be configured to conduct with the south-facing input / output terminal of another PE connected to that CB.

[0381] It is understood that in the NPU architecture of this application, the data flow strategy between PEs is not fixed, but can be abstracted into loadable "interconnect templates" and "I / O interface configurations" by using different data flow strategies such as fixed weights, fixed outputs, and fixed inputs. The interconnect template determines the direction and path of data (input, weights, partial sums) flowing in the PE array, such as horizontal, vertical, or diagonal. The I / O interface configuration determines the direction from which data enters the PE array or PE sub-region, and the direction from which computation results are written out. By combining different interconnect templates and I / O interfaces, systolic arrays with different logical behaviors can be instantiated on the same set of physical hardware.

[0382] In some embodiments of this application, the data flow mode may include any of the following: Weight Stationary (WS), Output Stationary (OS), or Input Stationary.

[0383] For example, when faced with systolic array computation tasks in convolutional layers: the compiler instantiates the systolic array template with fixed weights, and the configuration result is similar to that of the TPU mentioned above, maximizing weight reuse.

[0384] For example, when dealing with a fully connected layer: the compiler might use a fixed output approach, instantiating a systolic array template to obtain the target hardware configuration information. In this way, each node in the target hardware configuration information is assigned to a PE for accumulation, weighting, and input data flow. This helps reduce the writing of intermediate results and optimizes specific scenarios.

[0385] For example, when dealing with other operations, the pulse array template can be flexibly instantiated in a fixed input or other ways based on the data reuse characteristics.

[0386] from Figure 7 As can be seen, the PE array is configured as a pulsating array with fixed weights. Figure 7 In the pulsating array shown, the thick horizontal arrows from left to right indicate the direction of weight data flow in the pulsating array, i.e., flowing from left to right through each column of PEs. The thick vertical arrows from bottom to top indicate that the input activation data enters from the bottom (south) of the pulsating array and "pulsates" vertically upwards (north) through each row of PEs. Each PE uses its locally fixed weights to perform multiply-add calculations with the input activations flowing through it. Furthermore, the partial sums or final output results generated by the pulsating array calculations flow out or are collected from the right side of the array. It can be understood that in the fixed-weight mode, partial sums are typically passed horizontally (or diagonally) between PEs and accumulated, ultimately outputting from one side of the array.

[0387] Thus, this can be achieved through Example 1:

[0388] (A) Computational density: Achieves high computational density and high parallelism comparable to ASIC systolic arrays. Because data flow is controlled by hardware (configured interconnect network) rather than software (instructions), its operating overhead is extremely low.

[0389] (B) Shape Flexibility: This is a significant improvement over fixed ASICs. The shape of ASIC systolic arrays in related technologies is fixed, which is inefficient when handling mismatched matrix shapes. This application allows for dynamic configuration of the dimensions and shape of the logical VSA.

[0390] For example, for a “tall” matrix, the N×M PE array in this application can be configured as a logically deeper and narrower (e.g., (2N) × (M / 2)) pulsating array, thereby solving the matrix shape sensitivity problem of ASIC and always maintaining high PE utilization, where M and N are both positive integers.

[0391] (C) Simplified compilation: The compiler does not need to solve the general DFG mapping, but instead performs mapping based on the optimized VSA template (the mapping here refers to software mapping), which greatly reduces the difficulty of compilation.

[0392] It is understandable that the target hardware configuration information in Example 1 can be a VSA template.

[0393] Example 2: such as Figure 8 As shown, the target computation paradigm is the SIMD paradigm.

[0394] When an AI workload encounters element-level operations (such as activation or normalization), the NPU architecture array (or a portion thereof) will be reconfigured into SIMD mode.

[0395] 1. Mapping Mechanism and Principle

[0396] The compiler identifies data-level parallelism (DLP) loops for the target computation task in the AI ​​model, such as VectorAdd, ReLU, and LayerNorm. Subsequently, based on the number of operators in the target computation task and the SIMD hardware configuration information, the PE array (or a sub-region thereof) is logically divided into multiple SIMD clusters to obtain the target hardware configuration information. For example, each row (or column) of PEs can constitute an independent SIMD execution unit.

[0397] 2. Hardware Configuration Details

[0398] 2.1 PE Configuration: All PEs within a SIMD cluster are configured to perform the same operation (e.g., ALU is configured as "Add" or "Max(0, x)").

[0399] 2.2 Command Distribution: Commands are distributed using a broadcast bus. A cluster controller (or global configuration manager) broadcasts the same command / configuration word to all PEs within the cluster via this bus. All PEs execute the same command in the same cycle, but operate on different data; thus, PEs in the same SIMD cluster execute the operation corresponding to the same command in the same cycle.

[0400] 2.3 Local RF and Data Path of PE: Each PE independently reads different data (i.e. "MultipleData" in SIMD) from its local RF or shared local data storage (SPM) and executes it. After execution, the results are written back to the local RF.

[0401] 2.4 Ring Configuration Network: To further reduce the power consumption and fan-out pressure of the broadcast bus, a ring network can be used for instruction distribution. The configuration word (instruction) is passed cyclically between PEs within the SIMD cluster, and only needs to be injected at the first PE, which greatly saves power consumption.

[0402] like Figure 8 As shown, the PE array is reconstructed into a SIMD array, where each column of PE is a SIMD cluster, and the input activation of the SIMD array is from the SHM input located on the right side of the PE array and output to the SHM located on the left side of the PE array.

[0403] Thus, using Example 2, the following can be achieved:

[0404] (A) High-efficiency execution of non-MAC operations: Executes element-level operations that account for a large proportion of AI workloads with extremely high energy efficiency, perfectly making up for the shortcomings of the pulsating array mode.

[0405] (B) Alleviating memory bandwidth constraints: SIMD instructions have extremely high "instruction density" (one instruction completes N operations), significantly reducing the energy consumption of instruction fetching. At the same time, its regular vector access pattern makes data prefetching extremely efficient, reducing memory bandwidth pressure.

[0406] (C) Co-processing and Pipelining: With the support of DPR, this NPU architecture array can be dynamically divided into a large region running Systolic mode (processing GEMM) and a small region running SIMD mode (processing activation). The two work together in a pipelined manner, achieving "back-to-back" execution. GEMM results do not need to be written to DRAM; they flow directly to the SIMD region on-chip, thus achieving on-chip fusion and eliminating DRAM read / write operations for intermediate results.

[0407] Example 3: The target computation paradigm is the MIMD paradigm.

[0408] When AI workloads encounter irregular data dependencies, complex control flows, or require task-level parallelism, the NPU architecture will be configured in MIMD mode.

[0409] 1. Mapping Mechanism and Principle

[0410] Within the MIMD paradigm, NPU architectures can be used to handle complex algorithms that traditional NPU architectures (Systolic and SIMD) cannot accelerate at all. For example:

[0411] a) Irregular data dependency: As shown in the neighbor aggregation of a graph neural network (GNN), the number of neighbors and data access patterns of each node are dynamic and irregular.

[0412] b) Complex control flow: such as sparse attention in Transformer or if-else-intensive algorithms.

[0413] c) Task-level parallelism: Execute multiple different AI kernels simultaneously on a single PE array, or process data streams from multiple sensors concurrently.

[0414] In the MIMD paradigm, the compiler can treat each PE (or a small cluster of PEs) as an independent, asynchronous processor.

[0415] 2. Hardware Configuration Details

[0416] 2.1 PE Configuration: Each PE is configured to perform different operations according to its local task requirements.

[0417] 2.2 Instruction Distribution: This is the core of the MIMD model. Instruction distribution is decentralized. Each PE utilizes a local instruction / configuration memory as its local instruction memory. Like a miniature CPU, the PE independently fetches and executes instructions from this local memory. For example, a sequence of instructions is configured for each PE in the MIMD by the global configuration manager; this sequence is determined by the instantiated SIMD hardware configuration information, i.e., the target hardware configuration information.

[0418] 2.3 Reconfigurable Interconnect Network Configuration: The reconfigurable interconnect network is configured as a general point-to-point routing network (e.g., MeshNoC).

[0419] 2.4 Execution and Synchronization Model: A data flow or data-driven execution model is adopted. The PE fetches instructions from local memory (e.g., ADDR1, R2, R3). The local controller in the PE can automatically check whether its operands (e.g., R1, R2) are ready, that is, check whether the operands have arrived at the PE's local RF from other PEs via the interconnect network. If the operands have not arrived, the PE will automatically pause. Once all operands are available (i.e., data flow dependencies are satisfied), the PE is "triggered" to execute the instruction.

[0420] Thus, this can be achieved through Example 3:

[0421] (A) Optimal flexibility and workload coverage: The MIMD mode enables this NPU architecture to handle any type of computing task, including complex control flow and irregular algorithms that traditional NPU architectures (Systolic and SIMD) cannot accelerate at all, achieving 100% coverage of AI workloads.

[0422] (B) High-efficiency task-level parallelism (TLP): It can execute multiple different AI kernels simultaneously on a PE array (e.g., concurrently process data streams from multiple sensors), or map different components of a complex AI model (e.g., Transformer) (e.g., multi-head attention) to different MIMD regions, greatly improving system throughput and resource utilization.

[0423] (C) Adaptability and Flexibility: The AI ​​field has extremely rapid algorithm iteration. The flexibility of the MIMD model ensures that this NPU architecture will not be immediately obsolete when the next generation of algorithms (e.g., new models that combine GNN, Transformer and sparse computing) emerges, unlike fixed ASICs.

[0424] Example 4: Target computation paradigms include the systolic array paradigm and the SIMD paradigm.

[0425] Example 4 is the final manifestation of the NPU architecture in this application's "elimination of architectural silos" and "improvement of end-to-end utilization." For example... Figure 9 As shown, a typical AI computation task flow of a convolutional layer followed by an activation layer (Conv->ReLU) is used as an example, and a 4×4 PE array is used for illustration.

[0426] 1. Mapping Mechanism and Principles (PE Partitioning Based on DPR)

[0427] In Example 4, instead of mapping a single hardware configuration piece of information to the PE array, the compiler maps two hardware configuration pieces of information, also known as an operator chain, to the PE array to create a hybrid hardware pipeline. Specifically:

[0428] When analyzing the entire AI model, the compiler discovered an operator chain of "convolutional layer + activation layer". This led to the determination that the computational tasks in the convolutional layer corresponded to the systolic array paradigm, and the computational tasks in the activation layer corresponded to the SIMD paradigm.

[0429] Next, the compiler can decide how much PE resource to allocate to the systolic array and how much to allocate to SIMD based on the computational load and data dependencies of the two computational tasks, as well as the PE resource information provided by the DPR manager, such as a 4×4 PE array.

[0430] For example, a 4×4 PE array can be spatially divided into two independent, functionally heterogeneous regions, A and B. Region A is a pulsating array partition. For instance, the 3×4 region at the top of the PE array (containing 12 PEs) is configured as a "pulsating array" (e.g., ...). Figure 9 Region A in the diagram is specifically designed for performing high-density GEMM / convolution computations. Region B is the SIMD region. For example, the 1×4 region at the bottom of the PE array (containing 4 PEs in total) is configured as a "SIMD array" (as shown in the diagram). Figure 9 Region B in the diagram is specifically used to execute subsequent element-level activation functions.

[0431] The compiler can then instantiate a `map_systolic` template (filled with specific convolution parameters) for the "systolic array partition" and a `SIMD_vector_op` template (filled with ReLU opcodes) for the "SIMD partition". These two instantiation results constitute two hardware configuration information sets, such as the first and second hardware configuration information mentioned above. Optionally, the compiler can also generate instructions specifying the physical locations of these two partitions on the hardware (e.g., top region, bottom region) and how the data between them should be connected. All this information together constitutes a complete "hybrid task descriptor," namely the target hardware configuration information mentioned above.

[0432] 2. Hardware configuration details, this section is executed by the configuration and control subsystem, such as... Figure 9 As shown.

[0433] 2.1 Hardware Configuration

[0434] Region A ([1,1] to [3,4]) PEs are loaded with a “systolic array” configuration, i.e., an instantiated systolic array template, and the ALU of each PE in region A is locked for MAC operation. Reconfigurable interconnect network (S / CB): configured as a strict nearest neighbor data forwarding path, forming a 3×4 systolic array with fixed weights or fixed outputs.

[0435] Region B ([4,1] to [4,4]) PEs are loaded with a “SIMD” configuration, i.e., an instantiated SIMD template. And the reconfigurable interconnect network (S / CB): the reconfigurable interconnect network in this row is configured as an instruction broadcast bus.

[0436] The reconfigurable interconnection network between region A and region B is configured to directly connect the output ports of region A (such as the southbound outputs of the third row of PEs) to the input ports of region B (such as the northbound inputs of the fourth row of PEs).

[0437] 2.2 Instruction Distribution: Within the same cycle, a cluster controller (or global configuration controller) broadcasts the same instruction, such as ReLU (i.e., MAX(0, x)), to the four PEs in region B through the instruction broadcast bus corresponding to region B.

[0438] 2.3 Key data flow (on-chip operator fusion), including the following steps 1 to 5:

[0439] Step 1, Calculation. Convolution calculation begins in region A (3×4 systolic array).

[0440] Step 2, Forwarding. When region A computes the first batch of output results (for example, the PEs in row 3 produce an output vector of 4 elements), these results are not written back to off-chip DDR or on-chip SHM, but are stored in the local output RF of region A.

[0441] Step 3, Cross-regional communication. The convolution result of region A is transmitted to the input RF of region B through the reconstructed direct connection between region A and region B. For example, the convolution result of the PEs in the 3rd row of the PE array is output from the south end of the PE in that row, transmitted downwards through the reconstructed direct connection between region A and region B, and input from the north port of the PE in the 4th row to enter the input RF of the PE in the 4th row.

[0442] Step 4, Activation. Region B (bottom 1×4 SIMD array) receives the convolution results output by the third row of PEs, such as the four vectors corresponding one-to-one with the four PEs in the third row, and immediately and in parallel performs ReLU operations on these four vectors under the control of the broadcast command.

[0443] Step 5, Write Back. The PEs in Region B (executing the SIMD paradigm) write the final, activated results back to the SHM through the configured I / O interface path. For example, the output port of the PE is transmitted to an MPE through a configured path of the reconfigurable interconnect network, and the MPE transmits the results to the SHM.

[0444] like Figure 9 As shown, the 4×4 PE array is divided into region A and region B. Region A is configured as a systolic array and region B is configured as a SIMD array. The weights of the systolic array are input from the SHM on the left side of the PE array. The calculation results of each column of PE in the systolic array are passed down sequentially through the reconfigurable network until the 3rd row of PE in the systolic array passes the calculation results to the SIMD array.

[0445] Thus, this can be achieved through Example 4:

[0446] (A) Achieves "computation task fusion, also known as operator chain fusion": This solution achieves the fusion of the Conv and ReLU operators at the hardware level. This completely eliminates the overhead of data exchange through memory between the dedicated NPU architecture (executing Conv) and the dedicated VPU / CPU (for executing ReLU), greatly reducing memory bandwidth pressure and data transfer latency.

[0447] (B) Extremely High Hardware Utilization and Pipeline Parallelism: In traditional architectures, the systolic array is idle when the SIMD unit is working (and vice versa). In this scheme, region A (systolic array) and region B (SIMD) work simultaneously in a pipelined manner. While region A is processing the i-th batch of data, region B is processing the activation of the (i-1)-th batch of data. This ensures that the expensive PE computing resources remain highly utilized throughout the end-to-end task, where i is a positive integer.

[0448] (C) High Flexibility: This partitioning is dynamic. For a compute-intensive layer (high GEMM ratio), the DPR controller can be configured with a 4×3 systolic array and a 1×3 SIMD array. For a memory-intensive layer (high element-level operation ratio), a 2×4 systolic array and a 2×4 SIMD array can be configured. This flexibility ensures that hardware resources can always be configured to the “optimal” topology required for the current AI layer.

[0449] It is understandable that Example 4 is a specific example of a hybrid paradigm pipeline, but in actual implementations, it can be any other possible hybrid paradigm pipeline.

[0450] In some embodiments of this application, the above-described NPU architecture can be implemented in electronic devices, servers, or any other possible device entities.

[0451] In the NPU architecture provided in this application embodiment, the configuration and control subsystem can load instantiated hardware configuration information according to the reconfiguration request corresponding to the target computing task, dynamically configure the path topology of the reconfigurable interconnect network and the working mode of the PE in the PE array, thereby reconfiguring the target PE region in the PE array into a hardware structure for executing the target computing task. Therefore, the NPU architecture can dynamically switch between computing tasks corresponding to different computing paradigms based on the computing task requirements (represented or reflected by the reconfiguration request) corresponding to the AI ​​workload. This fundamentally solves the architecture silo problem caused by tasks of different computing paradigms relying on different dedicated hardware architectures, enabling the computing paradigms supported by the NPU architecture to adapt to AI workloads, and significantly improving the end-to-end utilization of the NPU architecture.

[0452] This application also provides an electronic device, which may include... Figures 1 to 9 The NPU architecture shown is illustrated. This electronic device can implement all the functions of the aforementioned NPU architecture and achieve the same technical effect; therefore, it will not be described in detail here to avoid repetition.

[0453] The data processing method provided in this application is executed by a data processing device, which may be an electronic device, or a functional module or entity within an electronic device, or... Figures 1 to 9 The NPU architecture shown in any of the above embodiments is not limited to this application. The following will use a data processing device as an example to illustrate the data processing method provided in the embodiments of this application.

[0454] This application also provides a data processing method, such as... Figure 10 As shown, the method may include the following steps 201 and 202.

[0455] Step 201: The data processing device loads the target hardware configuration information corresponding to the target computing task through the configuration and control subsystem, based on the reconstruction request corresponding to the target computing task.

[0456] Step 202: The data processing device, through the above-mentioned configuration and control subsystem, dynamically configures the path topology of the reconfigurable interconnection network and the working mode of the PE in the target PE area of ​​the PE array according to the target hardware configuration information, thereby reconfiguring the target PE area into a hardware structure for performing the target computing task.

[0457] It should be noted that after the target PE region is reconstructed into a hardware structure for executing the target computing task, the operator streams related to the target computing paradigm in the target computing task can be executed through the target PE region.

[0458] In some embodiments of this application, when the target hardware configuration information includes multiple items, they can be loaded simultaneously or in stages. This application does not limit the scope of the embodiments.

[0459] In some embodiments of this application, the target computation paradigm may include at least one of the systolic array paradigm, SIMD paradigm, and MIMD paradigm.

[0460] Thus, since the target computing paradigm can include at least one of the systolic array paradigm, SIMD paradigm, and MIMD paradigm, it can be ensured that the NPU architecture can dynamically switch between these three main computing paradigms and can be reconfigured simultaneously as a hardware structure to execute at least one computing paradigm, thereby improving the versatility of the NPU architecture.

[0461] Furthermore, since the NPU architecture can be reconfigured into a hardware structure that executes at least one computing paradigm, the intermediate data involved in a complete AI computing task can remain within the NPU architecture throughout the entire process, thereby reducing data transfer and improving the computing flexibility of the NPU architecture.

[0462] In some embodiments of this application, the target hardware configuration information may include PE configuration information and network topology configuration information;

[0463] The PE configuration information can be used to configure the working mode of the PE in the target PE area, and the network topology configuration information can be used to configure the path topology of the reconfigurable interconnection network.

[0464] Thus, since the target hardware configuration information includes PE configuration information for configuring the working mode of the PE in the target PE area and network topology configuration information for configuring the path topology of the reconfigurable interconnection network, the configuration and control subsystem can quickly switch the PE array and the reconfigurable interconnection network to an architecture that supports the target computing task based on the PE configuration information and the network topology configuration information, thereby improving the architecture switching speed.

[0465] In some embodiments of this application, step 202 may include steps 202A to 202C.

[0466] Step 202A: The data processing device loads the target hardware configuration information corresponding to the target computing task through the DPR manager in the configuration and control subsystem, based on the reconstruction request corresponding to the target computing task.

[0467] Step 202B: The data processing device dynamically configures the path topology in the reconfigurable interconnection network corresponding to the target PE area according to the network topology configuration information in the target hardware configuration information through the DPR manager.

[0468] Step 202C: The data processing device dynamically configures the working mode of the PE in the target PE area according to the PE configuration information in the target hardware configuration information through the global configuration manager in the configuration and control subsystem, so as to reconstruct the target PE area into a hardware structure for executing the target computing task.

[0469] In some embodiments of this application, the Global Configuration Manager and the DPR Manager support asynchronous execution.

[0470] In some embodiments of this application, after the working mode configuration and path topology configuration of the PE are completed, the target PE region is reconstructed into a hardware structure that executes the target computing paradigm.

[0471] Thus, since the configuration and control subsystem includes a global configuration manager and a DPR manager, and the DPR manager is responsible for loading hardware configuration information and dynamically configuring the path topology in the reconfigurable interconnect network corresponding to the target PE area according to the network topology configuration information in the target hardware configuration information; and the global configuration manager is responsible for configuring the working mode of the PE in the target area, hardware-level division of labor and collaboration of configuration tasks are realized, which improves the speed and flexibility of the PE array to switch computing paradigms.

[0472] Furthermore, due to the DPR manager's ability to independently configure the path topology, dynamic partial reconstruction of the network can be achieved. This allows the NPU architecture to update the path topology of another PE region in advance without interrupting the computation of one PE region, greatly improving the operational flexibility and resource utilization of the NPU architecture.

[0473] In some embodiments of this application, step 202 may include steps 202D and 202E.

[0474] Step 202D: The data processing device, through the configuration and control subsystem, loads the pulsating array configuration information according to the reconstruction request when the target computing task includes GEMM or convolution operators.

[0475] Step 202E: The data processing device configures the PEs in the target PE region to perform MAC operations according to the above-mentioned pulsating array configuration information, and configures the path topology of the reconfigurable interconnection network as a static direct connection between adjacent PEs, thereby reconfiguring the target PE region into a pulsating array that performs the target computing task.

[0476] In some embodiments of this application, if the target computation task includes more than a first predetermined number of general GEMMs or convolution operators, systolic array configuration information is loaded according to the reconstruction request. The first predetermined number can be 1, 2, 3, etc., and can be determined specifically according to actual usage requirements.

[0477] For further descriptions of steps 202D and 202E, please refer to the relevant description of Example 1 in the above NPU architecture embodiments.

[0478] Thus, when the target computation task includes GEMM or convolution operators, since systolic array configuration information can be loaded and the PEs in the target PE region can be configured to perform MAC operations according to the above systolic array configuration information, and the path topology of the reconfigurable interconnect network can be configured as static direct connections between adjacent PEs, the NPU architecture can be reconfigured into a systolic array, thereby improving the versatility of the NPU architecture.

[0479] In some embodiments of this application, step 202 may include steps 202F and 202G.

[0480] Step 202F: The data processing device, through the configuration and control subsystem, loads SIMD configuration information according to the aforementioned reconstruction request, provided that the target computation task includes element-level parallel operators.

[0481] Step 202G: The data processing device logically divides the target PE region into at least one SIMD cluster according to the aforementioned SIMD configuration information, configures the PEs in each SIMD cluster to execute the same element-level parallel operator, and configures the path topology of the reconfigurable interconnection network as a broadcast bus corresponding one-to-one with the aforementioned at least one SIMD cluster, thereby reconfiguring the target PE region into a SIMD array that performs the aforementioned target computing task.

[0482] In some embodiments of this application, if the target computation task includes more than a second predetermined number of element-level parallel operators, SIMD configuration information is loaded according to the aforementioned reconstruction request. The second predetermined number can be 1, 2, 3, etc., and can be determined specifically according to actual usage requirements.

[0483] For further descriptions of steps 202F and 202G, see the relevant descriptions of Example 2 in the above NPU architecture embodiments.

[0484] Thus, when the target computing task includes more than a second predetermined number of element-level parallel operators, since the target PE region can be logically divided into at least one SIMD cluster according to the SIMD configuration information, and the PEs in each SIMD cluster can be configured to execute the same element-level parallel operators, and the path topology of the reconfigurable interconnection network can be configured as a broadcast bus corresponding one-to-one with the at least one SIMD cluster, the NPU architecture can be reconfigured into a SIMD array that executes SIMD tasks, thereby improving the versatility of NPU architecture reconfiguration.

[0485] In some embodiments of this application, step 202 may include steps 202H and 202I.

[0486] Step 202H: The data processing device, through the configuration and control subsystem, loads MIMD configuration information according to the above-mentioned reconstruction request when the target computing task has irregular data dependency characteristics, complex control flow characteristics, or task-level parallel characteristics.

[0487] Step 202I: The data processing device configures each PE in the target PE area as an asynchronous processor according to the MIMD configuration information, and configures the path topology of the reconfigurable interconnection network as a routing network that supports communication between any two PEs, thereby reconfiguring the target PE area into a MIMD array that performs the target computing task.

[0488] For further descriptions of steps 202H and 202I, see the relevant description of Example 3 in the above NPU architecture embodiments.

[0489] Thus, when the target computing task has irregular data dependency characteristics, complex control flow characteristics, or task-level parallel characteristics, the configuration and control subsystem can load MIMD configuration information according to the above-mentioned reconfiguration request, configure each PE in the target PE region as an asynchronous processor according to the above-mentioned MIMD configuration information, and configure the path topology of the above-mentioned reconfigurable interconnection network as a routing network that supports communication between any two PEs, thereby reconfiguring the target PE region into a MIMD array that executes the above-mentioned target computing task, thus improving the versatility of the NPU architecture.

[0490] In some embodiments of this application, the target computing task includes determining at least one computing task; step 202 may include steps 202J to 202K below.

[0491] Step 202J: The data processing device, through the above-mentioned configuration and control subsystem, dynamically reconstructs at least one PE region of the PE array into a hardware structure for executing the at least one computing task based on at least one hardware configuration information corresponding to the at least one computing task.

[0492] Step 202K: The data processing device dynamically reconstructs the path topology between the at least one PE region based on the data flow relationship between the at least one hardware configuration information through the above-mentioned configuration and control subsystem.

[0493] Thus, when the target computing task includes at least one computing task corresponding to different computing paradigms, the configuration and control subsystem can reconfigure the partitions in the PE array into hardware structures that execute different computing paradigms, and configure a reconfigurable interconnection network to establish transmission paths between at least one PE region according to the data flow relationship between at least one computing task. This achieves tight coupling and pipelined operation between the working areas of different computing tasks at the hardware level, thereby eliminating the data transfer overhead between different dedicated hardware units in the traditional "heterogeneous" architecture. This allows the computing results of the previous computing task to flow directly to the working area of ​​the next computing task for processing, achieving ultimate operator fusion (referring to the fusion of operator chains of at least two computing tasks) and energy efficiency improvement.

[0494] In some embodiments of this application, the target computation task includes determining a first computation task and a second computation task, wherein the first computation task includes more than a first predetermined number of general matrix multiplication (GEMM) or convolution operators, and the second computation task includes more than a second predetermined number of element-level parallel operators; the above steps 202L to 202O.

[0495] Step 202L: The data processing device loads the pulsating array configuration information and SIMD configuration information according to the above-mentioned reconstruction request through the configuration and control subsystem.

[0496] Step 202M: The data processing device configures the PEs in the first PE region of the PE array to perform MAC operations according to the above-mentioned pulsating array configuration information, and configures the path topology between the above-mentioned reconfigurable interconnection network and the first PE region as a static direct connection between adjacent PEs, thereby reconfiguring the first PE region into a pulsating array that performs the above-mentioned first computing task.

[0497] Step 202N: The data processing device logically divides the second PE region of the PE array into at least one SIMD cluster according to the SIMD configuration information, configures the PE in each SIMD cluster to execute the same element-level parallel operator, and configures the path topology in the reconfigurable interconnection network (20) corresponding to the second PE region as a broadcast bus corresponding to the at least one SIMD cluster, thereby reconfiguring the target PE region into a SIMD array that performs the second computing task.

[0498] Step 2020: The data processing device dynamically reconstructs the path topology between the first PE region and the second PE region based on the data flow relationship between the pulsating array configuration information and the SIMD configuration information, thereby reconstructing the target PE region into a hardware structure for executing the target computing task.

[0499] For further descriptions of steps 202L to 202O, see the above embodiments for the target computation paradigm including a first computation paradigm and a second computation paradigm, and the relevant descriptions in Example 4.

[0500] Thus, when the target computing paradigm includes a first computing paradigm and a second computing paradigm, the configuration and control subsystem can reconfigure the two physical partitions of the PE array into hardware structures that execute different computing paradigms, and configure a reconfigurable interconnect network to establish a data path between them. This achieves tight coupling and pipelined operation between the working areas of different computing paradigms at the hardware level, thereby eliminating the data transfer overhead between different dedicated hardware units in the traditional "heterogeneous" architecture. This allows the computing results of the previous computing paradigm to flow directly to the working area of ​​the next computing paradigm for processing, achieving ultimate operator fusion and energy efficiency improvement.

[0501] In the data processing method provided in this application embodiment, since the configuration and control subsystem in the NPU architecture can load instantiated hardware configuration information according to the reconfiguration request corresponding to the target computing task, dynamically configure the path topology of the reconfigurable interconnection network and the working mode of the PE in the PE array, thereby reconfiguring the target PE region in the PE array into a hardware structure for executing the target computing task, the NPU architecture can dynamically switch between computing tasks corresponding to different computing paradigms according to the needs of the computing tasks corresponding to the AI ​​workload (represented or reflected by the reconfiguration request). This fundamentally solves the problem of architectural silos caused by tasks of different computing paradigms relying on different dedicated hardware architectures, realizes that the computing paradigms supported by the NPU architecture can adapt to AI workloads, and significantly improves the end-to-end utilization of the NPU architecture.

[0502] In some embodiments of this application, before step 201 above, the data processing method provided in the embodiments of this application may further include the following steps 203 and 204.

[0503] Step 203: The data processing device determines the target computation paradigm corresponding to the target computation task based on the operator type of the operator in the target computation task.

[0504] Step 204: The data processing device instantiates the hardware configuration information corresponding to the target computing paradigm based on the computing parameters of the target computing task, and obtains the target hardware configuration information.

[0505] In some embodiments of this application, the electronic device can determine the target computation paradigm corresponding to the target computation task based on the operator type of the operator in the target computation task using a compiler. Then, using the compiler, the hardware configuration information corresponding to the target computation paradigm is instantiated based on the computation parameters of the target computation task to obtain the target hardware configuration information.

[0506] In some embodiments of this application, the electronic device instantiates the hardware configuration information corresponding to the target computing paradigm based on the computing parameters of the target computing task, thereby obtaining the target hardware configuration information. This can also be referred to as: the process by which the hardware configuration information corresponding to the computing paradigm is mapped to the NPU architecture at the algorithm level through a compiler, i.e., software mapping. The process by which the configuration and control subsystem loads the target hardware configuration information and configures the path topology of the reconfigurable interconnection network and the working mode of the PE array can be referred to as hardware mapping of the target hardware configuration information.

[0507] In some embodiments of this application, the hardware configuration information corresponding to the aforementioned computational paradigm is parameterized. The electronic device instantiates the hardware configuration information corresponding to the target computational paradigm based on the computational parameters of the target computational task to obtain the target hardware configuration information. This can be achieved by the data processing device, through a compiler, filling in the parameter fields of the hardware configuration information corresponding to the target computational paradigm based on the computational parameters of the target computational task. For example, the computational parameters are filled into the parameter fields of the hardware configuration information corresponding to the target computational paradigm. It should be noted that the "parameter field" is only a specific example of the hardware configuration information corresponding to the computational paradigm in this application. In actual implementation, the hardware configuration information can also be any other template that can quickly map to the same type of operator.

[0508] In this way, the target computation paradigm can be intelligently determined based on the operator type, thus accurately matching the optimal hardware execution mode for heterogeneous computing tasks in AI workloads. Then, predefined hardware configuration information is instantiated based on specific computation parameters, simplifying the complex hardware mapping and configuration generation problem into an efficient parameter filling process. Therefore, it achieves automated, optimized compilation and mapping from high-level computing tasks to low-level hardware configurations, fundamentally overcoming the "architectural silos" problem caused by fixed hardware paradigms in traditional solutions, as well as the extremely complex compilation and mapping challenges faced by general reconfigurable architectures. This provides a precise and ready-to-use "action blueprint" for subsequent efficient and dynamic hardware reconfiguration.

[0509] For the remaining descriptions of the data processing methods provided in the embodiments of this application, please refer to the relevant descriptions in the above NPU architecture embodiments. To avoid repetition, they will not be repeated here.

[0510] The data processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0511] The data processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0512] The data processing device provided in this application embodiment can achieve... Figure 10 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0513] Optionally, such as Figure 11As shown, this application embodiment also provides an electronic device 600, including a processor 601 and a memory 602. The memory 602 stores a program or instructions that can run on the processor 601. When the program or instructions are executed by the processor 601, they implement the various steps of the above-described data processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0514] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0515] Figure 12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0516] The electronic device 1500 includes, but is not limited to, components such as: radio frequency unit 1501, network module 1502, audio output unit 1503, input unit 1504, sensor 1505, display unit 1506, user input unit 1507, interface unit 1508, memory 1509, and processor 1510.

[0517] Those skilled in the art will understand that the electronic device 1500 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 12 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0518] It should be understood that, in this embodiment, the input unit 1504 may include a graphics processing unit (GPU) 15041 and a microphone 15042. The GPU 15041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1506 may include a display panel 15061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1507 includes a touch panel 15071 and at least one of other input devices 15072. The touch panel 15071 is also called a touch screen. The touch panel 15071 may include a touch detection device and a touch controller. Other input devices 15072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0519] The memory 1509 can be used to store software programs and various data. The memory 1509 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1509 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1509 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0520] Processor 1510 may include one or more processing units; optionally, processor 1510 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1510.

[0521] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0522] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0523] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above data processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0524] It should be understood that the chip mentioned in the embodiments of this application can be an NPU or a system-on-a-chip (SoC). The SoC can also be referred to as a system chip, chip system, or system-on-a-chip, etc.

[0525] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the data processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0526] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0527] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0528] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A neural processing unit (NPU) architecture, characterized in that, The NPU architecture includes: Processing element PE arrays, reconfigurable interconnect networks, and configuration and control subsystems; The PE array is composed of multiple PEs, and the reconfigurable interconnect network connects each PE in the PE array; The configuration and control subsystem is configured as follows: Based on the reconfiguration request corresponding to the target computing task, load the target hardware configuration information corresponding to the target computing task; Based on the target hardware configuration information, the path topology of the reconfigurable interconnect network and the working mode of the PE in the target PE region of the PE array are dynamically configured, thereby reconfiguring the target PE region into a hardware structure for executing the target computing task.

2. The NPU architecture according to claim 1, characterized in that, The target hardware configuration information is related to the target computing paradigm corresponding to the target computing task; The target computation paradigm includes at least one of the following: systolic array paradigm, single instruction multiple data (SIMD) paradigm, and multiple instruction multiple data (MIMD) paradigm.

3. The NPU architecture according to claim 1, characterized in that, The target hardware configuration information includes PE configuration information and network topology configuration information; The PE configuration information is used to configure the working mode of the PE in the target PE region, and the network topology configuration information is used to configure the path topology of the reconfigurable interconnection network.

4. The NPU architecture according to any one of claims 1 to 3, characterized in that, Each PE includes a multifunctional arithmetic logic unit (ALU), which includes at least one of the following: a MAC subunit that performs a multiplication-accumulation (MAC) operation, an arithmetic logic subunit that performs arithmetic logic operations, and an SFU subunit that performs a special function (SFU) operation for nonlinear activation.

5. The NPU architecture according to claim 1, characterized in that, Each PE also includes: a local register file and a local configuration memory; The local register file is used to store PE operation data and PE operation results; The local configuration storage is used to store PE configuration information.

6. The NPU architecture according to claim 2, characterized in that, The reconfigurable interconnect network supports any of the following: Under the aforementioned pulse array paradigm, it serves as a static direct connection path for pulse data forwarding between adjacent PEs; In the SIMD paradigm, it serves as a broadcast bus that broadcasts the same operating instructions to a cluster of PEs; In the MIMD paradigm, it serves as a routing network that supports communication between any two PEs.

7. The NPU architecture according to claim 1 or 3, characterized in that, The configuration and control subsystem includes a global configuration manager and a dynamic partial reconfiguration (DPR) manager. The global manager is connected to each PE, and the DPR manager is connected to the reconfigurable interconnect network. The DPR manager is configured as follows: Load the target hardware configuration information according to the reconstruction request corresponding to the target computing task; According to the network topology configuration information in the target hardware configuration information, dynamically configure the path topology in the reconfigurable interconnection network corresponding to the target PE area; The global configuration manager is configured as follows: Based on the PE configuration information in the target hardware configuration information, dynamically configure the working mode of the PE in the target PE area.

8. The NPU architecture according to claim 1, characterized in that, The target computation task includes at least one computation task; The configuration and control subsystem is specifically configured as follows: According to the reconstruction request, load at least one hardware configuration information corresponding to each of the at least one computing task; Based on the at least one hardware configuration information, at least one PE region of the PE array is dynamically reconstructed into a hardware structure for executing the at least one computing task; Based on the data flow relationship between the at least one hardware configuration information, the path topology between the at least one PE region is dynamically reconstructed.

9. The NPU architecture according to claim 1 or 2, characterized in that, The NPU architecture also includes: on-chip shared memory, memory access unit, and data access controller; The memory access unit is connected to the on-chip shared memory and the PE array, respectively; the data access controller is connected to the off-chip memory and the on-chip shared memory, respectively. The memory access unit is used for data transmission and address management between the on-chip shared memory and the PE array. The data access controller is used for data transfer between the off-chip memory and the on-chip shared memory.

10. An electronic device, characterized in that, Includes the neural processing unit (NPU) architecture as described in any one of claims 1 to 9.

11. A data processing method, characterized in that, The method includes: The configuration and control subsystem loads the target hardware configuration information corresponding to the target computing task based on the reconfiguration request corresponding to the target computing task. The configuration and control subsystem dynamically configures the path topology of the reconfigurable interconnection network and the working mode of the PE in the target PE region of the PE array according to the target hardware configuration information, thereby reconfiguring the target PE region into a hardware structure for executing the target computing task.

12. The method according to claim 11, characterized in that, The target hardware configuration information includes PE configuration information and network topology configuration information; The PE configuration information is used to configure the working mode of the PE in the target PE region, and the network topology configuration information is used to configure the path topology of the reconfigurable interconnection network.

13. The method according to claim 12, characterized in that, The configuration and control subsystem dynamically configures the path topology of the reconfigurable interconnection network and the operating mode of the PE in the target PE region of the PE array based on the target hardware configuration information, including: The DPR manager in the configuration and control subsystem loads the target hardware configuration information corresponding to the target computing task based on the refactoring request corresponding to the target computing task. The DPR manager dynamically configures the path topology in the reconfigurable interconnect network corresponding to the target PE area according to the network topology configuration information in the target hardware configuration information. The global configuration manager in the configuration and control subsystem dynamically configures the working mode of the PE in the target PE area according to the PE configuration information in the target hardware configuration information.

14. The method according to claim 11, characterized in that, The step of dynamically configuring the path topology of the reconfigurable interconnection network and the operating mode of the PE in the target PE region of the PE array according to the target hardware configuration information through the configuration and control subsystem, thereby reconstructing the target PE region into a hardware structure for executing the target computing task, includes: In the case that the target computation task includes General Matrix Multiplication (GEMM) or a convolution operator, the configuration and control subsystem loads the systolic array configuration information according to the reconstruction request. Based on the systolic array configuration information, the PEs in the target PE region are configured to perform MAC operations, and the path topology of the reconfigurable interconnect network is configured as static direct connections between adjacent PEs, thereby reconfiguring the target PE region into a systolic array that performs the target computing task.

15. The method according to claim 11, characterized in that, The step of dynamically configuring the path topology of the reconfigurable interconnection network and the operating mode of the PE in the target PE region of the PE array according to the target hardware configuration information through the configuration and control subsystem, thereby reconstructing the target PE region into a hardware structure for executing the target computing task, includes: In the case that the target computing task includes element-level parallel operators, the configuration and control subsystem loads SIMD configuration information according to the reconstruction request. Based on the SIMD configuration information, the target PE region is logically divided into at least one SIMD cluster, and the PEs in each SIMD cluster are configured to execute the same element-level parallel operators. The path topology of the reconfigurable interconnection network is configured as a broadcast bus corresponding one-to-one with the at least one SIMD cluster, thereby reconfiguring the target PE region into a SIMD array that executes the target computing task.

16. The method according to claim 11, characterized in that, The step of dynamically configuring the path topology of the reconfigurable interconnection network and the operating mode of the PE in the target PE region of the PE array according to the target hardware configuration information through the configuration and control subsystem, thereby reconstructing the target PE region into a hardware structure for executing the target computing task, includes: In the case that the target computing task has irregular data dependency characteristics, complex control flow characteristics, or task-level parallel characteristics, the configuration and control subsystem loads MIMD configuration information according to the reconstruction request. Based on the MIMD configuration information, each PE in the target PE region is configured as an asynchronous processor, and the path topology of the reconfigurable interconnection network is configured as a routing network that supports communication between any two PEs, thereby reconfiguring the target PE region into a MIMD array that executes the target computing task.

17. The method according to claim 11, characterized in that, The target computation task includes determining at least one computation task; The step of dynamically configuring the path topology of the reconfigurable interconnection network and the operating mode of the PE in the target PE region of the PE array according to the target hardware configuration information through the configuration and control subsystem, thereby reconstructing the target PE region into a hardware structure for executing the target computing task, includes: Through the configuration and control subsystem, at least one PE region of the PE array is dynamically reconstructed into a hardware structure for executing the at least one computing task based on at least one hardware configuration information corresponding to the at least one computing task. The configuration and control subsystem dynamically reconstructs the path topology between the at least one PE region based on the data flow relationship between the at least one hardware configuration information.

18. The method according to claim 11 or 17, characterized in that, The target computation task includes determining a first computation task and a second computation task, wherein the first computation task includes more than a first predetermined number of general matrix multiplication (GEMM) or convolution operators, and the second computation task includes more than a second predetermined number of element-level parallel operators. The step of dynamically configuring the path topology of the reconfigurable interconnection network and the operating mode of the PE in the target PE region of the PE array according to the target hardware configuration information through the configuration and control subsystem, thereby reconstructing the target PE region into a hardware structure for executing the target computing task, includes: The configuration and control subsystem loads pulsating array configuration information and SIMD configuration information according to the reconstruction request. According to the systolic array configuration information, the PEs in the first PE region of the PE array are configured to perform MAC operations, and the path topology between the reconfigurable interconnect network and the first PE region is configured as a static direct connection between adjacent PEs, thereby reconfiguring the first PE region into a systolic array that performs the first computing task. According to the SIMD configuration information, the second PE region of the PE array is logically divided into at least one SIMD cluster, and the PEs in each SIMD cluster are configured to execute the same element-level parallel operator. The path topology in the reconfigurable interconnection network corresponding to the second PE region is configured as a broadcast bus corresponding one-to-one with the at least one SIMD cluster, thereby reconfiguring the target PE region into a SIMD array that executes the second computing task. Based on the data flow relationship between the pulsating array configuration information and the SIMD configuration information, the path topology between the first PE region and the second PE region is dynamically reconstructed, thereby reconstructing the target PE region into a hardware structure for executing the target computing task.

19. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the data processing method as described in any one of claims 11 to 18.

20. A chip comprising a processor and a communication interface coupled to the processor, the processor being configured to run a program or instructions to implement the steps of the data processing method as claimed in any one of claims 11 to 18.

Images