Computing unit, instruction fetching request processing method, electronic equipment and medium
By constructing an independent instruction fetch path and a differentiated scheduling mechanism, the inefficiency of traditional GPGPUs in handling large-scale matrix multiplication and tensor transformation tasks is solved, achieving efficient parallel instruction fetching and distribution at the workgroup level and thread bundle level, thus improving the overall performance of the computing unit.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU YIZHU INTELLIGENT TECH CO LTD
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-12
AI Technical Summary
Traditional GPGPUs suffer from low instruction density, high control overhead, and insufficient execution efficiency when handling large-scale matrix multiplication and addition, tensor transformation, and global data transport. Furthermore, the differences between workgroup-level instructions and thread bundle-level instructions in terms of resource consumption and latency characteristics lead to instruction supply bottlenecks.
An independent instruction fetching path and a differentiated scheduling mechanism are constructed. Thread-level and workgroup-level instructions are processed by the first and second instruction processing units, respectively, and instruction fetch request processing units are used for arbitration, merging and distributing instruction requests to optimize instruction supply efficiency and hardware utilization.
It improves the overall instruction supply efficiency and hardware utilization, reduces resource contention and supply delays caused by differences in instruction granularity, achieves efficient parallel instruction fetching and distribution, and optimizes the performance of computing units.
Smart Images
Figure CN122018993A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a computing unit, an instruction fetch request processing method, an electronic device, and a medium. Background Technology
[0002] In recent years, with the continuous growth in the scale of deep learning models, especially the widespread application of large language models and generative AI, general-purpose graphics processing units (GPGPUs), as the core computing power carrier, are facing the evolution from "general-purpose parallelism" to "heterogeneous integration" and "hybrid granular computing". Traditional GPGPUs are based on a single instruction multithreaded (SIMT) architecture, using warp as the basic scheduling unit. They perform well in handling fine-grained tasks, but when dealing with large-scale matrix multiplication and addition (MMA), tensor transformation, and global data transport, they expose problems such as low instruction density, high control overhead, and insufficient execution efficiency. Although modern AI accelerators have integrated dedicated execution units such as Tensor Cores and Matrix Cores, which execute at the workgroup (WG) granularity, and hardware modules such as Tensor Memory Accelerators (TMAs) that support batch asynchronous data transfer, the existing instruction architecture is still limited to warp-level instructions. This requires the compiler to break down high-level computational intent into a large number of low-level SIMT instructions, resulting in high instruction bandwidth pressure, redundant control logic, and decreased pipeline utilization, making it difficult to fully realize the performance potential of dedicated hardware. Against this backdrop, there is an urgent need to introduce workgroup-level instructions while retaining the original SIMT execution capabilities. These instructions would directly describe matrix operations, data copying, and synchronization configuration with high-order semantics, significantly reducing the number of instruction issues and improving hardware utilization and program locality. However, workgroup-level instructions differ fundamentally from traditional warp-level instructions in execution cycle, resource consumption, access frequency, and latency characteristics. Using a unified instruction fetch path and shared buffer structure can easily create instruction supply bottlenecks, impacting overall performance. Therefore, there is an urgent need for a new computing unit architecture that can accommodate both instruction granularities and achieve efficient parallel instruction fetching and distribution. Summary of the Invention
[0003] This disclosure provides a computing unit, an instruction fetch request processing method, an electronic device, and a medium. It aims to improve the overall instruction supply efficiency and hardware utilization by constructing an independent instruction fetch path and a differentiated scheduling mechanism to process instruction fetch requests of workgroup-level instructions and thread bundle-level instructions in parallel and efficiently, avoiding resource contention and supply delays caused by differences in instruction granularity.
[0004] According to one aspect of this disclosure, a computing unit is provided, characterized in that it comprises:
[0005] Instruction cache unit, used to cache instructions;
[0006] The first instruction processing unit is configured to generate a first instruction fetch request when the first instruction fetch condition is met, wherein the first instruction fetch request points to the thread-beam level instruction to be fetched.
[0007] The second instruction processing unit is used to execute workgroup-level instructions, wherein a workgroup-level instruction includes multiple thread bundle-level instructions;
[0008] The instruction fetch request processing unit is used to generate a second instruction fetch request when the second instruction fetch condition is met. The second instruction fetch request points to the workgroup-level instruction to be fetched. The unit arbitrates the first instruction fetch request and the second instruction fetch request according to a preset strategy, and selects one instruction fetch request to send to the instruction cache unit so that the corresponding instruction data can be sent to the first instruction processing unit or the second instruction processing unit according to the source of the instruction fetch request.
[0009] Optionally, the first instruction processing unit includes multiple execution units, each execution unit including a first instruction cache, the first instruction cache being used to cache thread-beam level instructions retrieved from the instruction cache unit.
[0010] The first instruction fetch condition includes: the instruction data in the first instruction buffer is lower than a preset capacity threshold; or the first instruction buffer is empty and subsequent instructions need to be prefetched to maintain continuous pipeline execution.
[0011] Optionally, the first instruction processing unit further includes a first scheduler.
[0012] The first scheduler is used to monitor the fill status of the first instruction cache of the plurality of execution units, and generate a first instruction fetch request when the first instruction fetch condition is met, so as to trigger the fetching of new thread-beam level instructions from the instruction cache unit.
[0013] Optionally, the instruction fetch request processing unit includes a second instruction cache, which is used to cache workgroup-level instructions fetched from the instruction cache unit.
[0014] The second instruction fetch condition includes: the instruction data in the second instruction buffer is lower than a preset capacity threshold; or the second instruction buffer is empty and subsequent instructions need to be prefetched to maintain continuous pipeline execution.
[0015] Optionally, the instruction fetch request processing unit further includes a second scheduler.
[0016] The second scheduler is configured to monitor the fill status of the second instruction cache and generate a second fetch request when the second fetch condition is met, thereby triggering the fetching of new workgroup-level instructions from the instruction cache unit.
[0017] Optionally, the instruction fetching request processing unit includes a request merging module and an arbitration module.
[0018] The request merging module is used to receive one or more first fetch requests from the first scheduler, merge the first fetch requests with the same or consecutive instruction data addresses into a first aggregated fetch request, and provide the first aggregated fetch request to the arbitration module.
[0019] The arbitration module is used to arbitrate the first aggregated instruction fetch request and the second instruction fetch request according to a preset strategy, and select one instruction fetch request to send to the instruction cache unit.
[0020] Optionally, the preset strategy includes at least one of a fixed priority strategy, a polling strategy, a dynamic priority strategy based on request waiting time, or a dynamic priority strategy based on the tension between the first instruction cache and the second instruction cache.
[0021] Optionally, the instruction fetch request processing unit further includes:
[0022] The distribution module is configured to receive instruction data packets returned from the instruction cache unit, the instruction data packets including an instruction fetch request source identifier, and distribute the instruction data packets to the first instruction processing unit or the second instruction processing unit according to the instruction fetch request source identifier.
[0023] Optionally, the instruction fetch request processing unit further includes:
[0024] The instruction fetch request sending module is used to generate an instruction fetch request data packet based on the arbitration processing result and send it to the instruction cache unit. The instruction fetch request data packet includes the instruction data address corresponding to the selected instruction request, the instruction fetch request source identifier, and the instruction fetch request sending order identifier.
[0025] The distribution module is also used to manage the order of the returned instruction data according to the instruction fetch request sending sequence identifier.
[0026] Optionally, the second instruction processing unit includes multiple heterogeneous computing units, wherein the heterogeneous computing units include at least two of a general computing unit, a tensor processing unit, and a data copying unit, and the workgroup-level instructions include general computing instructions, tensor operation instructions, and data copying instructions;
[0027] The distribution module assigns workgroup-level instructions to corresponding heterogeneous computing units for execution based on the type of the instruction.
[0028] Optionally, the workgroup-level instruction further includes a trigger instruction, which is used to trigger the first instruction processing unit to start working;
[0029] In response to the trigger command, the distribution module sends a start signal to the first command processing unit.
[0030] According to one aspect of this disclosure, a fetch request processing method is provided, applied to a computing unit as described in any of the preceding claims, the fetch request processing method comprising:
[0031] The first instruction processing unit generates a first instruction fetch request when the first instruction fetch condition is met. The first instruction fetch request points to the thread-beam level instruction to be fetched.
[0032] The second instruction processing unit executes workgroup-level instructions, wherein a workgroup-level instruction includes multiple thread bundle-level instructions.
[0033] The instruction fetch request processing unit generates a second instruction fetch request when the second instruction fetch condition is met. The second instruction fetch request points to the workgroup-level instruction to be fetched. The first instruction fetch request and the second instruction fetch request are arbitrated according to a preset strategy, and one instruction fetch request is selected and sent to the instruction cache unit so that the corresponding instruction data can be sent to the first instruction processing unit or the second instruction processing unit according to the source of the instruction fetch request.
[0034] Optionally, the instruction fetch request processing method further includes:
[0035] The instruction fetch request processing unit merges the first instruction fetch requests with the same or consecutive instruction data addresses into a first aggregated instruction fetch request.
[0036] Arbitration is performed on the first aggregated instruction fetch request and the second instruction fetch request according to a preset strategy, and one instruction fetch request is selected and sent to the instruction cache unit.
[0037] According to one aspect of this disclosure, an electronic device is provided, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program is executed by the processor to implement the instruction fetch request processing method as described above.
[0038] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores one or more programs, which can be executed by one or more processors to implement the instruction fetch request processing method as described above.
[0039] In the computing unit proposed in this embodiment, the first instruction processing unit generates a first fetch request when the first instruction fetch condition is met. The first fetch request points to the thread bundle-level instruction to be fetched. The second instruction processing unit executes a workgroup-level instruction, wherein a workgroup-level instruction includes multiple thread bundle-level instructions. The fetch request processing unit generates a second fetch request when the second instruction fetch condition is met. The second fetch request points to the workgroup-level instruction to be fetched. The first fetch request and the second fetch request are arbitrated according to a preset strategy, and one fetch request is selected and sent to the instruction cache unit so that the corresponding instruction data can be sent to the first instruction processing unit or the second instruction processing unit according to the source of the fetch request. In this way, by constructing an independent instruction fetch path and a differentiated scheduling mechanism, the fetch requests of workgroup-level instructions and thread bundle-level instructions can be processed in parallel and efficiently, avoiding resource contention and supply delays caused by differences in instruction granularity, thereby improving the overall instruction supply efficiency and hardware utilization.
[0040] Furthermore, the request merging module receives one or more first instruction fetch requests from the first scheduler, merges first instruction fetch requests with the same or consecutive instruction data addresses into a first aggregated instruction fetch request, and provides the first aggregated instruction fetch request to the arbitration module. The arbitration module arbitrates the first aggregated instruction fetch request and the second instruction fetch request according to a preset strategy, and selects one instruction fetch request to send to the instruction cache unit. In this way, since multiple threads often access the same or consecutive instruction address space when executing similar control flows, especially in the case of synchronous execution of branch convergence or loop body code, multiple first instruction fetch requests have a high degree of spatial locality. The request merging module identifies the overlap or contiguousness on these addresses and aggregates multiple independent requests into a single larger-granularity instruction fetch request. This not only reduces the number of requests sent to the instruction cache unit, but also improves the data utilization of each cache access. It can effectively reduce the frequency of multiple requests to the same instruction address range in multi-threaded parallel scenarios, and improve the access efficiency and overall throughput of the instruction cache unit.
[0041] Furthermore, the second instruction processing unit includes multiple heterogeneous computing units, which include at least two of the following: a general-purpose computing unit, a tensor processing unit, and a data copying unit. The workgroup-level instructions include general-purpose computing instructions, tensor operation instructions, and data copying instructions. The distribution module allocates the workgroup-level instructions to the corresponding heterogeneous computing units for execution based on their type. In this way, by directly mapping high-level abstract macro instructions to dedicated hardware execution units, the instruction overhead and execution latency caused by decomposing complex operations into a large number of low-level SIMT instruction sequences in traditional architectures are avoided.
[0042] Furthermore, the workgroup-level instructions also include trigger instructions to initiate the operation of the first instruction processing unit. In response to the trigger instructions, the distribution module sends a start signal to the first instruction processing unit. This enables explicit control of the thread bundle-level execution flow by the workgroup-level instructions, constructing a complete collaborative mechanism from macro-level task scheduling to micro-level thread execution. By introducing trigger instructions into the workgroup-level instructions, higher-level control logic can precisely initiate the work of the underlying SIMT execution units at appropriate times, thereby achieving phased and orderly execution of computational tasks.
[0043] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description
[0044] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.
[0045] Figure 1 This is a system architecture diagram of the computing chip used in the instruction fetch request processing method of this embodiment;
[0046] Figure 2 This is a schematic diagram of the structure of a computing unit according to an embodiment of the present disclosure;
[0047] Figure 3 This is a main flowchart of an instruction fetch request processing method according to an embodiment of the present disclosure;
[0048] Figure 4 This is one embodiment of the present disclosure. Figure 3 A flowchart of a sub-step of one embodiment of step S330;
[0049] Figure 5 This is a schematic diagram of the structure of an instruction fetch request processing apparatus according to an embodiment of the present disclosure;
[0050] Figure 6 This is an architecture diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.
[0052] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:
[0053] A thread is the smallest unit of execution in a graphics processing unit (GPU) when performing data processing tasks. Each thread can independently perform the same pattern of processing on different data.
[0054] A warp is the basic unit of scheduling and execution in a graphics processing unit (GPU). It contains a fixed number of threads, typically 32 or 64. All threads execute the same instructions within the same clock cycle, but can operate on different data, enabling Single Instruction Multiple Data (SIMD) parallel computation. When a grid is activated (equivalent to a kernel being activated, with each kernel corresponding to its own grid), the grid contains workgroups. After a workgroup is assigned to a computational unit, it is divided into multiple warps. Within a warp, all threads execute in a SIMT manner, performing the same instructions at each step, but processing private data—data that corresponds to a specific thread.
[0055] Work Group (WG): A group of threads consisting of multiple threads, also known as a thread block. Threads in a single work group are generally used to perform the same pattern of processing on different data.
[0056] A grid consists of multiple workgroups. Each workgroup is a grid point in the grid, and each workgroup has corresponding grid point coordinates. A grid can be one-dimensional, two-dimensional, or three-dimensional. The shape of the grid is defined by the user during the programming process.
[0057] A Compute Unit (CU) is a processing module in electronic devices such as a graphics processing unit (GPU). In some architectures, a CU may also be called a Stream Multiprocessor (SM). A GPU can include multiple CUs. A CU can be considered a processing core within a GPU. Each CU can execute instructions independently, and the GPU can process data in parallel through multiple CUs.
[0058] Execution Unit (EU, also known as a multi-stream processor (SM)): It is the execution unit in a microprocessor. It is responsible for executing instructions and performing computational tasks such as arithmetic operations, logical operations, and shift operations. In fact, it has both the functions of a controller and an arithmetic unit.
[0059] Shared memory (SMEM) refers to a certain amount of memory in a multiprocessor computer system that can be accessed by different processors. Because multiple processors need to access memory quickly, memory caching is necessary. When any cached data is updated, the shared memory needs to be updated immediately, otherwise different processors might use different data.
[0060] General Purpose Registers (GPRs) are used to store thread-private data, such as address indices, loop variables, or temporary calculation results. Each thread has its own independent GPR space, which is accessed extremely quickly and is a key resource for achieving thread-level parallel computing. In the Single Instruction Multithreaded (SIMT) execution model, GPRs and SMEMs work together: GPRs store thread-private states, while SMEMs carry shared data between thread bundles. Together, they support the efficient execution of high-concurrency computing tasks.
[0061] Instruction Cache Unit (ICache): This is a high-speed storage structure located at the front end of the computing unit. It is used to cache instruction data prefetched and decoded from global memory to reduce instruction access latency, improve instruction fetch bandwidth, and support the computing core to quickly and continuously fetch instructions.
[0062] Global memory refers to the large-capacity main memory inside or outside the computing chip, typically composed of high-bandwidth memory (such as GDDR6, HBM2 / HBM3), used to store program input data, output results, and persistent intermediate data for cross-kernel calls. Global memory has a large storage capacity, but suffers from high access latency, high power consumption, and limited bandwidth resources, making it one of the key performance bottlenecks in parallel computing systems.
[0063] Scalar General Purpose Register (SGPR): Used to store scalar data, such as control flow variables, loop counters, and address offsets.
[0064] Vector General Purpose Register (VGPR): Used to store vector data, such as operands for floating-point or integer vector operations.
[0065] Tensor Memory Accelerator (TMA) is a dedicated hardware data copying unit used to efficiently perform loading and storing operations on tensor data across different storage levels, particularly optimized for large-scale, high-dimensional tensor data access patterns in AI computing and GPU architectures. TMA can encapsulate complex multidimensional memory movement tasks (such as tensor block transfers from global memory to local shared memory) into single macro instructions, either declaratively or in batches. The hardware automatically handles address calculation, data partitioning, dependency management, and Direct Memory Access (DMA) transfer scheduling. In computing units supporting hybrid granularity instruction architectures, TMA is typically driven by workgroup-level macro instructions, working in conjunction with the Matrix Core to achieve decoupling and pipelining of computation and memory access. It is a key component in building high-performance AI-GPU front-end data supply pipelines.
[0066] Warp-level instructions are fine-grained instructions executed in a Single Instruction Multithreaded (SIMT) architecture, with the thread bundle as the basic unit of execution and scheduling. Each thread bundle contains a group of threads that start simultaneously and execute in the same order, sharing the same instruction stream. Warp-level instructions typically correspond to scalar or vector-level operations and are characterized by high frequency, low latency, and strong real-time access requirements.
[0067] Workgroup-level (WG-level) instructions (also known as macro instructions) are high-order semantic instructions that execute at the workgroup level and are designed for large-scale parallel operations. They are used to directly describe macro-level computational tasks such as matrix multiplication and addition (MMA), tensor data transfer (e.g., via TMA units), and synchronization or shared resource configuration between workgroups. Logically, they typically represent a composite operation completed by a group of cooperating threads, characterized by high instruction density, long execution cycles, low access frequency, but wide impact. WG-level instructions significantly reduce instruction issuance and control overhead by encapsulating the semantic functions of multiple traditional SIMT instructions, improving program locality and hardware utilization. They are a key abstraction for achieving efficient macro-operation control in hybrid-granularity computing architectures.
[0068] In traditional GPGPU architectures, computational tasks are typically scheduled and executed using thread bundles as the basic unit, employing a Single Instruction Multithreaded (SIMT) execution model. In this model, multiple threads execute the same instructions in parallel, with the thread bundle-level instruction execution unit responsible for instruction fetching, decoding, and scheduling. While performing well with fine-grained tasks, it suffers from low instruction density, high control overhead, and insufficient execution efficiency when dealing with large-scale tasks such as large-scale matrix multiplication and addition (MMA), tensor transformations, and global data transport. Although modern AI accelerators have integrated dedicated execution units such as Tensor Cores and Matrix Cores, which execute at the workgroup level, and hardware modules like TMAs that support batch asynchronous data transport, existing instruction architectures remain limited to warp-level instructions. This necessitates the compiler to break down high-level computational intent into numerous low-level SIMT instructions, leading to high instruction bandwidth pressure, redundant control logic, and decreased pipeline utilization, making it difficult to fully leverage the performance potential of dedicated hardware. Against this backdrop, there is an urgent need to introduce workgroup-level instructions while retaining the original SIMT execution capabilities. These instructions would directly describe operations such as matrix operations, data copying, and synchronization configuration with higher-order semantics, significantly reducing the number of instruction issues and improving hardware utilization and program locality. However, workgroup-level instructions differ fundamentally from traditional thread bundle-level instructions in terms of execution cycle, resource consumption, access frequency, and latency. If a unified instruction fetch path and shared cache structure are used, it is easy to create an instruction supply bottleneck, affecting overall performance.
[0069] Based on this, embodiments of this disclosure provide a computing unit, an instruction fetch request processing method, an electronic device, and a medium to realize a novel computing unit architecture that can accommodate both workgroup-level instructions and traditional thread bundle-level instructions at both instruction granularities, and achieve efficient parallel instruction fetching and distribution.
[0070] System architecture description applied in the embodiments of this disclosure
[0071] Figure 1This is a system architecture diagram of the computing chip used in the instruction fetch request processing method of this disclosure embodiment. The system mainly includes a computing chip 100, which includes multiple computing units 110 and global memory 170. Computing units 0 to 3 are shown in the figure as examples, but it should be understood that the architecture is not limited thereto, and the computing chip 100 may contain more or fewer computing units 110 according to actual design requirements. Multiple computing units 110 may be included in a cluster according to design requirements. Each computing unit 110 further includes a first instruction processing unit 120, a second instruction processing unit 130, an instruction fetch request processing unit 140, an instruction cache unit 150, and local shared memory 160. The first instruction processing unit 120 adopts a single instruction multithreaded (SIMT) execution model, including multiple execution units 121 for executing thread bundle-level instructions and a first scheduler 123 for thread bundle scheduling. Each execution unit 121 includes a first instruction cache 122. The figure shows execution units 0 to 3, which are for illustrative purposes only; in actual implementation, any number of execution units 121 may be included. The second instruction processing unit 130 includes multiple heterogeneous computing units for executing workgroup-level instructions. The heterogeneous computing units include at least two of the following: a general-purpose computing unit 131, a tensor processing unit 132, and a data copying unit 133.
[0072] The computing chip 100 is a computer processing device with parallel processing capabilities, suitable for high-concurrency, high-throughput computing tasks. Typical applications include graphics processing, scientific computing, and artificial intelligence training and inference. This computing component can be implemented using various processor architectures, such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a General-Purpose Computing on Graphics Processing Units (GPGPU), a Neural Processing Unit (NPU), or a Tensor Processing Unit (TPU). When processing graphics-related tasks, such as image recognition, 3D rendering, or animation generation, the computing chip 100 typically employs a GPU architecture to fully leverage its massively parallel computing advantages; while when performing general computing tasks, such as operating system scheduling or serial logic processing, a CPU architecture can be used to balance flexibility and control efficiency.
[0073] During task execution, the computation task is divided into several workgroups, which are then assigned to appropriate computing units 110 for processing. Each workgroup consists of multiple warps, which are further distributed to execution units 121 for parallel computation upon entering the computing unit. During execution, the register file provides low-latency data storage and access support for each thread. Local shared memory 160 serves as the global storage medium within the computing unit 110, used to temporarily store data to be shared between workgroups and intermediate computation results. Global memory 170, as the unified address space in the chip system, provides a unified data access interface for multiple computing units 110, supporting data sharing and collaborative computation across computing units 110 (i.e., across workgroups), further improving task processing efficiency. Global memory 170 is connected to each computing unit 110 via a high-bandwidth interconnect network, supporting cache coherency protocols to ensure read / write consistency of data across multiple computing units. Each computing unit 110 exclusively uses the physical resource areas of its allocated register file and local shared memory 160, ensuring resource isolation between different computing units. Within the same computing unit 110, multiple execution units 121, general-purpose computing units 131, tensor processing units 132, and data copying units 133 can share access to the local shared memory 160 resources allocated to that computing unit 110 to support inter-thread collaboration. The register file resources are divided according to threads or execution units, with each execution unit 121 exclusively using its allocated register space, thereby ensuring the independence of thread execution, avoiding data conflicts, and improving overall execution efficiency. In the computing chip 100, multiple computing units 110 can share access to global memory 170 resources, supporting cross-workgroup task collaboration and data exchange.
[0074] Multiple computing units 110 are responsible for executing computing tasks involving thread bundles and workgroups. Within each computing unit 110, an instruction cache unit 150 caches thread bundle-level and workgroup-level instructions prefetched from global memory 170 to reduce instruction access latency and improve fetch efficiency. The instruction cache unit 150 supports responding to different types of fetch requests from the fetch request processing unit 140, including thread bundle-level and workgroup-level instructions. The fetch request processing unit 140 coordinates fetch requests from the first instruction processing unit 120 and the second instruction processing unit 130, and schedules and manages fetch requests of different granularities according to a preset arbitration strategy. Within the same computing unit 110, the first instruction processing unit 120 and the second instruction processing unit 130 work together. The first instruction processing unit 120 undertakes the execution of thread bundle-level instructions, and the second instruction processing unit 130 undertakes the execution of workgroup-level instructions, forming a multi-layered, mixed-granularity instruction execution architecture. Specifically, the first instruction processing unit 120 generates a first fetch request when a first fetch condition is met. The first instruction fetch request points to the thread bundle-level instruction to be fetched. The second instruction processing unit 130 is used to execute workgroup-level instructions. A workgroup-level instruction includes multiple thread bundle-level instructions. The instruction fetch request processing unit 140 generates a second instruction fetch request when the second instruction fetch condition is met. The second instruction fetch request points to the workgroup-level instruction to be fetched. It also arbitrates the first and second instruction fetch requests according to a preset strategy, selecting one instruction fetch request to send to the instruction buffer unit 150, so that the corresponding instruction data can be sent to the first instruction processing unit 120 or the second instruction processing unit 130 according to the source of the instruction fetch request.
[0075] Understandably, the architecture of the Computing Chip 100 employs a dual-path decoupling design, ensuring that thread bundle-level instructions and workgroup-level instructions do not interfere with each other during instruction fetching, caching, scheduling, and issuing. This effectively improves the parallel processing capability and resource utilization of the instruction stream, while reducing execution bottlenecks caused by differences in instruction granularity. Batch scheduling of workgroup-level instructions reduces control overhead, while fine-grained execution of thread bundle-level instructions ensures computational flexibility. The synergy between the two meets the demands of high-performance computing while optimizing energy efficiency, providing more efficient hardware support for AI workloads.
[0076] Structure of the computing unit in the embodiments of this disclosure
[0077] Figure 2 This is a schematic diagram of the structure of a computing unit according to an embodiment of this disclosure. The following is in conjunction with... Figure 1 and Figure 2 This invention provides a detailed description of the structure and workflow of the computing unit 110 for the instruction fetch request processing method in this embodiment.
[0078] like Figure 1and Figure 2 As shown, in the computing unit 110, the first instruction processing unit 120 is based on the Single Instruction Multithreaded (SIMT) model. It uses a first scheduler 123 to uniformly schedule multiple execution units 121. Each execution unit 121 independently processes one or more thread bundle-level instructions, suitable for control-flow-intensive or data-dependent computational scenarios. During execution, each execution unit 121 uses its internal first instruction buffer 122 to cache thread bundle-level instructions fetched from the instruction buffer unit 150, achieving fast instruction fetching and low-latency execution. The first scheduler 123 monitors the fill status of the first instruction buffer 122 in the multiple execution units 121 and generates a first instruction fetch request when a first instruction fetch condition is met, triggering the fetching of new thread bundle-level instructions from the instruction buffer unit 150. It should be noted that the first instruction fetch condition includes: the instruction data in the first instruction buffer 122 being lower than a preset capacity threshold; or the first instruction buffer 122 being empty and requiring pre-fetching of subsequent instructions to maintain continuous pipeline execution. The second instruction processing unit 130 is geared towards workgroup-level parallel tasks. The instruction fetch request processing unit 140 performs unified arbitration on the workgroup-level instructions executed by the second instruction processing unit 130 and the thread bundle-level instructions executed by the first instruction processing unit 120 to ensure that they do not conflict when sharing the instruction cache unit 150. The instruction fetch request processing unit 140 includes a second instruction cache 141 and a second scheduler 142. The second instruction cache 141 caches workgroup-level instructions prefetched from the instruction cache unit 150. The second scheduler 142 monitors the fill status of the second instruction cache 141 and generates a second instruction fetch request when the second instruction fetch condition is met, triggering the instruction cache unit 150 to load new workgroup-level instructions into the second instruction cache 141. It should be noted that the second instruction fetch condition includes: the instruction data in the second instruction cache 141 being lower than a preset capacity threshold; or the second instruction cache 141 being empty and requiring the prefetching of subsequent instructions to maintain the continuous execution of workgroup-level tasks.
[0079] In the SIMT architecture, multiple thread bundles or workgroups accessing the same program segment or loop structure can cause the first instruction fetch requests to be concentrated in the same address range of the instruction cache unit 150. Furthermore, when the control flow converges to the branch successor instruction, function calls return the same target, or parallel tasks are scheduled to the same kernel function, address duplication or consecutive instruction fetching behavior will also be triggered. Based on this, a request merging module 143 is also provided in the instruction fetch request processing unit 140. The request merging module 143 is responsible for receiving one or more first instruction fetch requests from the first scheduler 123, merging the first instruction fetch requests with the same or consecutive instruction data addresses into a first aggregated instruction fetch request, so that the first aggregated instruction fetch request can be sent to the instruction cache unit 150 for single batch reads, thereby reducing bandwidth waste and cache conflicts caused by repeated accesses. Through this mechanism, the request merging module 143 identifies overlaps or contiguousness on these addresses, aggregating multiple independent requests into a single, larger-granularity fetch request. This not only reduces the number of requests sent to the instruction cache unit 150 but also improves the data utilization of each cache access. It effectively reduces the frequency of multiple requests to the same instruction address range in multi-threaded parallel scenarios, improving the access efficiency and overall throughput of the instruction cache unit 150. Simultaneously, the request merging module 143 possesses dynamic identification capabilities, distinguishing between independent request streams and mergeable request streams, and completing the aggregation operation without disrupting the execution order, ensuring correct synchronization and low-latency response for the fetching behavior of each thread bundle and workgroup.
[0080] In some embodiments, such as Figure 2As shown, the instruction fetch request processing unit 140 also includes an arbitration module 144, an instruction fetch request sending module 145, and a distribution module 146. The arbitration module 144 arbitrates the first aggregated instruction fetch request from the request merging module 143 and the second instruction fetch request generated by the scheduler 142 according to a preset strategy to determine their priorities and execution order, and selects one instruction fetch request to send to the instruction cache unit 150. It should be noted that the preset strategy adopted by the arbitration module 144 includes at least one of a fixed priority strategy, a polling strategy, a dynamic priority strategy based on request waiting time, or a dynamic priority strategy based on the tension between the first instruction cache 122 and the second instruction cache 141. The instruction fetch request sending module 145 is responsible for generating an instruction fetch request data packet for the selected instruction fetch request based on the arbitration result, and encapsulating it in a format conforming to the interface protocol of the instruction cache unit 150 before sending it to the instruction cache unit 150. It should be noted that the instruction fetch request data packet includes the instruction data address corresponding to the selected instruction request, the instruction fetch request source identifier, and the instruction fetch request sending order identifier. After the instruction cache unit 150 returns a response, the distribution module 146 receives the instruction data packet returned from the instruction cache unit 150. This instruction data packet includes instruction data, a corresponding fetch request source identifier, and a fetch request sequence identifier. The distribution module 146 manages the order of the returned instruction data according to the fetch request sending sequence identifier, and precisely routes the instruction data to the corresponding first instruction processing unit 120 or second instruction processing unit 140 according to the fetch request source identifier. Specifically, when the fetch request source identifier points to the first instruction processing unit 120, the distribution module 146 sends the instruction data to the request merging module 143, which further splits the instruction data according to the context information and stores it in the corresponding first instruction cache 122, so that the corresponding execution unit 121 can execute thread-beam level instructions. When the fetch request source identifier points to the second instruction processing unit 140, the distribution module 146 stores the instruction data in the second instruction cache 141 for use by workgroup-level instruction execution.
[0081] In some embodiments, workgroup-level instructions include general computation instructions, tensor operation instructions, and data copy instructions. The distribution module 146 also supports dynamically allocating instruction data to corresponding heterogeneous computing units for execution based on the type of workgroup-level instructions. Specifically, an allocation module 147 is also provided in the instruction fetch request processing unit 140. The allocation module 147 is responsible for distributing the instruction data stored in the second instruction buffer 141 to the corresponding heterogeneous computing units based on the instruction type identification result. General computation instructions are routed to the general computation unit 131 for execution, tensor operation instructions are sent to the tensor processing unit 132 for execution, and data copy instructions are efficiently transferred by the data copy unit 133. By directly mapping high-level abstract macro instructions to dedicated hardware execution units, the instruction overhead and execution latency caused by decomposing complex operations into a large number of low-level SIMT instruction sequences in traditional architectures are avoided. All distribution actions strictly follow low-latency, high-throughput scheduling sequences, ensuring that various instructions are executed on the optimal path, maximizing the overall computational efficiency of the system. In some embodiments, workgroup-level instructions also include trigger instructions. The distribution module 146 also supports sending a start signal to the first instruction processing unit 120 in response to a trigger instruction, coordinating cross-level instruction scheduling and execution. Specifically, the distribution module 147 is responsible for identifying the trigger instruction and sending a start signal to the first instruction processing unit 120. Upon receiving the start signal, the first instruction processing unit 120 immediately activates the corresponding thread bundle-level task queue, retrieves the prefetched instruction data stored in the first instruction buffer 122, and forms execution coordination with the workgroup-level instructions. The introduction of the trigger instruction enables dynamic linkage between the two levels of processing units, enabling explicit control of the thread bundle-level execution flow by the workgroup-level instructions, and constructing a complete coordination mechanism from macro-level task scheduling to micro-level thread execution. By introducing the trigger instruction into the workgroup-level instructions, the higher-level control logic can precisely start the work of the underlying SIMT execution unit at the appropriate time, thereby achieving phased and orderly execution of computational tasks.
[0082] The instruction fetch request processing method provided in this disclosure is an embodiment of the present invention.
[0083] Figure 3 This is a main flowchart of an instruction fetch request processing method according to an embodiment of this disclosure. The instruction fetch request processing method of this embodiment is applied to the computing unit 110 described above. Figure 3 As shown, the instruction fetch request processing method of this disclosure includes the following steps:
[0084] Step S310: The first instruction processing unit generates a first instruction fetch request when the first instruction fetch condition is met. The first instruction fetch request points to the thread bundle-level instruction to be fetched.
[0085] Step S320: Execute workgroup-level instructions through the second instruction processing unit, wherein a workgroup-level instruction includes multiple thread bundle-level instructions.
[0086] Step S330: The instruction fetch request processing unit generates a second instruction fetch request when the second instruction fetch condition is met. The second instruction fetch request points to the workgroup-level instruction to be fetched. The first instruction fetch request and the second instruction fetch request are arbitrated according to a preset strategy. One instruction fetch request is selected and sent to the instruction cache unit so that the corresponding instruction data can be sent to the first instruction processing unit or the second instruction processing unit according to the source of the instruction fetch request.
[0087] Figure 4 This is one embodiment of the present disclosure. Figure 3 A flowchart illustrating a sub-step of step S330 in one embodiment. (See attached flowchart.) Figure 4 As shown, step S330 of this embodiment includes the following sub-steps:
[0088] Step S410: The instruction fetch request processing unit merges the first instruction fetch requests with the same or consecutive instruction data addresses into a first aggregated instruction fetch request.
[0089] Step S420: Arbitrate the first aggregated fetch request and the second fetch request according to a preset strategy, and select one fetch request to send to the instruction cache unit.
[0090] The instruction fetch request processing method disclosed herein is applied to the computing unit 110 described above. Its specific processing procedure is completely consistent with the hardware architecture of the computing unit 110 in the above embodiment, and will not be repeated here.
[0091] Description of apparatus and devices according to embodiments of this disclosure
[0092] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of an instruction fetch request processing device 500 disclosed herein. The instruction fetch request processing device 500 is integrated into a computing chip and applied to the computing chip. The instruction fetch request processing device 500 includes:
[0093] The first instruction fetch request generation unit 510 is used to generate a first instruction fetch request by the first instruction processing unit when the first instruction fetch condition is met. The first instruction fetch request points to the thread-beam level instruction to be fetched.
[0094] The workgroup-level instruction execution unit 520 executes workgroup-level instructions through the second instruction processing unit, wherein a workgroup-level instruction includes multiple thread bundle-level instructions.
[0095] The instruction fetch request arbitration processing unit 530 is used to generate a second instruction fetch request when the second instruction fetch condition is met, the second instruction fetch request pointing to the workgroup-level instruction to be fetched, and to arbitrate the first instruction fetch request and the second instruction fetch request according to a preset strategy, and select one instruction fetch request to send to the instruction cache unit, so as to send the corresponding instruction data to the first instruction processing unit or the second instruction processing unit according to the source of the instruction fetch request.
[0096] The instruction fetch request processing apparatus disclosed herein is used to execute the instruction fetch request processing method as described in the above embodiments. Its specific processing procedure is the same as that of the instruction fetch request processing method described in the above embodiments, and will not be repeated here.
[0097] This disclosure also provides an electronic device 600, including:
[0098] At least one processor, and,
[0099] A memory that is communicatively connected to at least one processor; wherein,
[0100] The memory stores instructions that are executed by at least one processor to cause the at least one processor to perform the method as described in any of the above embodiments of this application when executing the instructions.
[0101] The following is combined Figure 6 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 610, a memory 620, an input / output interface 630, a communication interface 640, and a bus 650.
[0102] The processor 610 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.
[0103] The memory 620 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 620 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 620 and is invoked by the processor 610 to execute the instruction fetch request processing method of the embodiments of this disclosure.
[0104] The input / output interface 630 is used to realize information input and output;
[0105] The communication interface 640 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).
[0106] Bus 650 transmits information between various components of the device (e.g., processor 610, memory 620, input / output interface 630, and communication interface 640);
[0107] The processor 610, memory 620, input / output interface 630 and communication interface 640 are connected to each other within the device via bus 650.
[0108] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the instruction fetch request processing method of the above embodiments, which will not be described again here.
[0109] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.
[0110] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0111] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.
[0112] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.
[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0114] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0115] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.
[0116] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.
Claims
1. A computing unit, characterized in that, include: Instruction cache unit, used to cache instructions; The first instruction processing unit is configured to generate a first instruction fetch request when the first instruction fetch condition is met, wherein the first instruction fetch request points to the thread-beam level instruction to be fetched. The second instruction processing unit is used to execute workgroup-level instructions, wherein a workgroup-level instruction includes multiple thread bundle-level instructions; The instruction fetch request processing unit is used to generate a second instruction fetch request when the second instruction fetch condition is met. The second instruction fetch request points to the workgroup-level instruction to be fetched. The unit arbitrates the first instruction fetch request and the second instruction fetch request according to a preset strategy, and selects one instruction fetch request to send to the instruction cache unit so that the corresponding instruction data can be sent to the first instruction processing unit or the second instruction processing unit according to the source of the instruction fetch request.
2. The computing unit according to claim 1, characterized in that, The first instruction processing unit includes multiple execution units, and each execution unit includes a first instruction cache, which is used to cache thread-beam level instructions retrieved from the instruction cache unit. The first instruction fetch condition includes: the instruction data in the first instruction buffer is lower than a preset capacity threshold; or the first instruction buffer is empty and subsequent instructions need to be prefetched to maintain continuous pipeline execution.
3. The computing unit according to claim 2, characterized in that, The first instruction processing unit further includes a first scheduler. The first scheduler is used to monitor the fill status of the first instruction cache of the plurality of execution units, and generate a first instruction fetch request when the first instruction fetch condition is met, so as to trigger the fetching of new thread-beam level instructions from the instruction cache unit.
4. The computing unit according to claim 1, characterized in that, The instruction fetch request processing unit includes a second instruction cache, which is used to cache workgroup-level instructions fetched from the instruction cache unit. The second instruction fetch condition includes: the instruction data in the second instruction buffer is lower than a preset capacity threshold; or the second instruction buffer is empty and subsequent instructions need to be prefetched to maintain continuous pipeline execution.
5. The computing unit according to claim 4, characterized in that, The instruction fetch request processing unit also includes a second scheduler. The second scheduler is configured to monitor the fill status of the second instruction cache and generate a second fetch request when the second fetch condition is met, thereby triggering the fetching of new workgroup-level instructions from the instruction cache unit.
6. The computing unit according to claim 3, characterized in that, The instruction fetching request processing unit includes a request merging module and an arbitration module. The request merging module is used to receive one or more first fetch requests from the first scheduler, merge the first fetch requests with the same or consecutive instruction data addresses into a first aggregated fetch request, and provide the first aggregated fetch request to the arbitration module. The arbitration module is used to arbitrate the first aggregated instruction fetch request and the second instruction fetch request according to a preset strategy, and select one instruction fetch request to send to the instruction cache unit.
7. The computing unit according to claim 6, characterized in that, The preset strategy includes at least one of the following: a fixed priority strategy, a polling strategy, a dynamic priority strategy based on request waiting time, or a dynamic priority strategy based on the tension between the first instruction cache and the second instruction cache.
8. The computing unit according to claim 5 or 6, characterized in that, The instruction fetch request processing unit further includes: The distribution module is configured to receive instruction data packets returned from the instruction cache unit, the instruction data packets including an instruction fetch request source identifier, and distribute the instruction data packets to the first instruction processing unit or the second instruction processing unit according to the instruction fetch request source identifier.
9. The computing unit according to claim 8, characterized in that, The instruction fetch request processing unit further includes: The instruction fetch request sending module is used to generate an instruction fetch request data packet based on the arbitration processing result and send it to the instruction cache unit. The instruction fetch request data packet includes the instruction data address corresponding to the selected instruction request, the instruction fetch request source identifier, and the instruction fetch request sending order identifier. The distribution module is also used to manage the order of the returned instruction data according to the instruction fetch request sending sequence identifier.
10. The computing unit according to claim 9, characterized in that, The second instruction processing unit includes multiple heterogeneous computing units, which include at least two of a general computing unit, a tensor processing unit, and a data copying unit. The workgroup-level instructions include general computing instructions, tensor operation instructions, and data copying instructions. The distribution module assigns workgroup-level instructions to corresponding heterogeneous computing units for execution based on the type of the instruction.
11. The computing unit according to claim 10, characterized in that, The workgroup-level instruction also includes a trigger instruction, which is used to trigger the first instruction processing unit to start working; In response to the trigger command, the distribution module sends a start signal to the first command processing unit.
12. A fetch request processing method, applied to a computing unit as described in any one of claims 1 to 11, the fetch request processing method comprising: The first instruction processing unit generates a first instruction fetch request when the first instruction fetch condition is met. The first instruction fetch request points to the thread-beam level instruction to be fetched. The second instruction processing unit executes workgroup-level instructions, wherein a workgroup-level instruction includes multiple thread bundle-level instructions. The instruction fetch request processing unit generates a second instruction fetch request when the second instruction fetch condition is met. The second instruction fetch request points to the workgroup-level instruction to be fetched. The first instruction fetch request and the second instruction fetch request are arbitrated according to a preset strategy, and one instruction fetch request is selected and sent to the instruction cache unit so that the corresponding instruction data can be sent to the first instruction processing unit or the second instruction processing unit according to the source of the instruction fetch request.
13. The instruction fetch request processing method according to claim 12, characterized in that, The instruction fetch request processing method further includes: The instruction fetch request processing unit merges the first instruction fetch requests with the same or consecutive instruction data addresses into a first aggregated instruction fetch request. Arbitration is performed on the first aggregated instruction fetch request and the second instruction fetch request according to a preset strategy, and one instruction fetch request is selected and sent to the instruction cache unit.
14. An electronic device comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory, wherein the program is executed by the processor to implement the instruction fetch request processing method as described in claim 12 or 13.
15. A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the instruction fetch request processing method as described in claim 12 or 13.