Instruction execution method, related device and medium

Abstract

The invention provides an instruction execution method, a related device and a medium, and the method comprises the steps: issuing an independent loading instruction through a scheduling unit according to the required quantity of shared memory resources, and loading matrix data required by multiplication and addition operation from a storage medium to a special matrix data cache region; the tensor processing unit is controlled to obtain matrix data from the special matrix data cache region and execute multiplication and addition operation according to a matrix multiplication and addition operation instruction issued by the scheduling unit; wherein the loading instruction and the matrix multiply-add operation instruction are controlled by the scheduling unit to be executed asynchronously and are mutually independent. According to the method, the loading instruction and the matrix multiply-add operation instruction are decoupled and asynchronous execution is supported, so that data loading delay is hidden, the calculation efficiency of the tensor processing unit and the hardware resource utilization rate are improved, and the overall instruction throughput rate is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an instruction execution method, related apparatus and medium. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning technologies, GPUs (Graphics Processing Units) have become the core hardware for performing large-scale matrix operations (such as Generalized Matrix Multiplication, GEMM) due to their powerful parallel computing capabilities. In modern GPU architectures, to cope with the ever-increasing computing demands, large-scale tensor processing units with higher parallelism and computing power, namely Super Tensor Cores (MSUs), have emerged. These MSUs are designed specifically for handling extreme-scale matrix operations, and their performance directly impacts the efficiency of the entire deep learning training and inference system. However, in existing technologies, MSUs typically employ atomic instructions that bundle data loading with matrix multiplication-addition (MMA) operations. This means that when the scheduling unit issues a MMA instruction, the MSU must simultaneously complete the entire process of reading data from storage and performing computation. This tightly coupled design pattern exposes data loading latency directly to critical computation paths, preventing the high computing power of MSUs from being effectively utilized due to data supply bottlenecks, severely limiting overall instruction throughput. Summary of the Invention

[0003] This disclosure provides an instruction execution method, related apparatus, and medium, which aims to hide data loading latency, improve the computational efficiency and hardware resource utilization of the tensor processing unit, and thus improve the overall instruction throughput by decoupling the loading instruction from the matrix multiplication and addition operation instruction and supporting asynchronous execution.

[0004] According to one aspect of this disclosure, an instruction execution method is provided, applied to a graphics processor, wherein the graphics processor includes a scheduling unit, at least one tensor processing unit, and a dedicated matrix data buffer associated with the tensor processing unit, the instruction execution method comprising:

[0005] The scheduling unit issues independent loading instructions to load the matrix data required for multiplication and addition operations from the storage medium into the dedicated matrix data cache area.

[0006] The scheduling unit issues matrix multiplication and addition instructions to control the tensor processing unit to obtain the matrix data from the dedicated matrix data buffer and perform multiplication and addition operations.

[0007] The loading instruction and the matrix multiplication and addition operation instruction are executed asynchronously and independently by the scheduling unit.

[0008] Optionally, the dedicated matrix data cache is a static random access memory located inside the tensor processing unit.

[0009] Optionally, the scheduling unit issuing the loading instruction includes:

[0010] Determine whether the matrix multiplication and addition instruction cannot be executed immediately due to resource limitations;

[0011] If so, the loading instruction is issued first to preload the matrix data required for multiplication and addition operations into the dedicated matrix data cache.

[0012] Optionally, the resource limitation includes at least one of the following situations:

[0013] (a) One or more thread bundles within the workgroup of the graphics processor are undergoing resource initialization, resulting in the register resources used to store the calculation results of the matrix multiplication and addition instructions not being fully allocated;

[0014] (b) The calculation result of the matrix multiplication and addition instruction is the operand of other subsequent instructions, forming a data dependency chain, and the subsequent instructions have not yet released the register resources they occupy;

[0015] (c) Synchronization operations exist between the threads within the workgroup, causing the matrix multiplication and addition instructions to wait for the synchronization to complete before they can be executed.

[0016] Optionally, the tensor processing unit executes the matrix multiplication and addition operation instructions including:

[0017] Determine whether the matrix data required for the multiplication-addition operation has been pre-stored in the dedicated matrix data cache area;

[0018] If so, the matrix data is directly read from the dedicated matrix data buffer into its internal register for calculation;

[0019] If not, then the operation of loading matrix data from the storage medium into the dedicated matrix data cache is triggered.

[0020] Optionally, the internal register includes a first register and a second register, and the tensor processing unit further includes the following when executing the matrix multiplication and addition operation instructions:

[0021] The matrix data read from the dedicated matrix data buffer is loaded into the first register and the second register, respectively;

[0022] The matrix data loaded into the first register and the second register are distributed to multiple multiply-accumulate units for parallel multiply-accumulate calculations.

[0023] Optionally, the dedicated matrix data cache is divided into at least two logical partitions, which are used to cache operands from different sources in matrix multiplication operations.

[0024] Optionally, the calculation result of the matrix multiplication and addition operation instruction needs to be written into a vector general-purpose register used by multiple execution units in the graphics processor. The scheduling unit allocates vector general-purpose register resources to one or more thread bundles within the workgroup, including:

[0025] The scheduling unit provides key parameters for determining the specific register address to which the calculation result of the matrix multiplication and addition instruction should be written. These key parameters include: the initial offset, interval offset, and number of thread bundles on each execution unit. The vector general-purpose register space of the multiple execution units is a contiguous storage space.

[0026] The scheduling unit determines the starting address of the register corresponding to the i-th thread bundle in each execution unit as the starting address of the contiguous storage space + initial offset + (i-1). Interval offset, where i is the number of thread bundles allocated within the corresponding execution unit.

[0027] Optionally, after the loading instruction is executed, its state is recorded so that the scheduling unit or the tensor processing unit can perform dependency determination and subsequent instruction scheduling.

[0028] According to one aspect of this disclosure, a graphics processor is provided, comprising:

[0029] Tensor processing unit, used to perform multiplication and addition operations;

[0030] The scheduling unit is configured to issue loading instructions and matrix multiplication and addition instructions according to the instruction execution method described in any of the preceding items, so as to control the tensor processing unit to asynchronously execute the loading instructions and the matrix multiplication and addition instructions.

[0031] According to one aspect of this disclosure, an instruction execution apparatus is provided for a graphics processor, wherein the graphics processor includes a scheduling unit, at least one tensor processing unit, and a dedicated matrix data buffer associated with the tensor processing unit, and the instruction execution apparatus includes:

[0032] The matrix data loading unit is used to load the matrix data required for multiplication and addition operations from the storage medium into the dedicated matrix data cache area by issuing independent loading instructions through the scheduling unit.

[0033] The multiply-add operation execution unit is used to control the tensor processing unit to obtain the matrix data from the dedicated matrix data buffer and perform multiply-add operations through the matrix multiply-add operation instructions issued by the scheduling unit;

[0034] The loading instruction and the matrix multiplication and addition operation instruction are executed asynchronously and independently by the scheduling unit.

[0035] According to one aspect of this disclosure, an electronic device is provided, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program is executed by the processor to implement the instruction execution method as described in any of the preceding claims.

[0036] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores one or more programs, which can be executed by one or more processors to implement the instruction execution method as described in any of the preceding claims.

[0037] In the instruction execution method proposed in this embodiment, an independent loading instruction is issued by the scheduling unit to load the matrix data required for multiplication and addition operations from the storage medium into a dedicated matrix data buffer associated with the tensor processing unit. The matrix multiplication and addition operation instruction issued by the scheduling unit controls the tensor processing unit to retrieve the matrix data from the dedicated matrix data buffer and perform the multiplication and addition operations. The loading instruction and the matrix multiplication and addition operation instruction are executed asynchronously and independently by the scheduling unit. By decoupling the data loading process from the matrix multiplication and addition operations, forming an independent and asynchronously executable loading instruction, the scheduling unit can flexibly arrange the data prefetching timing according to the current hardware resource status. This effectively hides the data loading delay within the execution process of computation or other memory operations, significantly reducing the idling phenomenon of the tensor processing unit due to data incompleteness. This decoupling mechanism allows the tensor processing unit to continue the data loading process in parallel while waiting for critical resources (such as register allocation and thread bundle synchronization), thereby improving the computational efficiency and hardware resource utilization of the tensor processing unit, and ultimately increasing the overall instruction throughput.

[0038] Furthermore, it is determined whether the matrix multiplication-addition operation instruction cannot be executed immediately due to resource constraints. If so, a loading instruction is issued first to preload the matrix data required for the multiplication-addition operation into a dedicated matrix data cache. The resource constraints include at least one of the following situations: (a) one or more thread bundles within the graphics processor's workgroup are undergoing resource initialization, resulting in the register resources used to store the calculation results of the matrix multiplication-addition operation instruction not being fully allocated; (b) the calculation result of the matrix multiplication-addition operation instruction is the operand of other subsequent instructions, forming a data dependency chain, and the subsequent instructions have not yet released the register resources they occupy; (c) there are synchronization operations between thread bundles within the workgroup, causing the matrix multiplication-addition operation instruction to wait for the synchronization to complete before it can be executed. The above mechanism enables the scheduling unit to proactively initiate data prefetching operations during the waiting window period when computing resources are not yet ready, making full use of the originally idle time period to complete data preparation, avoiding data waiting on critical computing paths. By placing the loading behavior before the actual calculation, the memory access latency can be hidden to the maximum extent without changing the program semantics, thereby improving the continuous computing capability of the tensor processing unit.

[0039] Furthermore, the introduction of a dedicated matrix data cache provides a high-speed data temporary storage space close to the tensor processing unit for tensor operations, avoiding the high latency overhead caused by frequent access to the global storage medium, further improving the computational efficiency and hardware resource utilization of the tensor processing unit, and thus improving the overall instruction throughput.

[0040] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0041] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0042] Figure 1A This is a schematic diagram illustrating an instruction execution method for a tensor processing unit according to an embodiment of the related art;

[0043] Figure 1B This is a schematic diagram of an instruction execution method for a tensor processing unit according to yet another embodiment of the related art;

[0044] Figure 2 This is a system architecture diagram of the system to which the instruction execution method of this embodiment is applied;

[0045] Figure 3This is a schematic diagram of the structure of a computing unit according to an embodiment of the present disclosure;

[0046] Figure 4 This is a main flowchart of an instruction execution method according to an embodiment of the present disclosure;

[0047] Figure 5A This is a schematic diagram of loading instructions and matrix multiplication and addition instructions according to an embodiment of the present disclosure;

[0048] Figure 5B This is a schematic diagram illustrating the allocation of vector general-purpose register resources for one or more thread bundles within a workgroup according to an embodiment of this disclosure;

[0049] Figure 5C This is a schematic diagram of an instruction execution method for a tensor processing unit according to an embodiment of the present disclosure;

[0050] Figure 5D This is a schematic diagram of an instruction execution method for a tensor processing unit according to yet another embodiment of the present disclosure;

[0051] Figure 6 This is a schematic diagram of the structure of an instruction execution device according to an embodiment of the present disclosure;

[0052] Figure 7 This is an architecture diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0054] Before providing a further detailed description of the embodiments of this disclosure, the nouns and terms used in the embodiments of this disclosure are explained, and the nouns and terms used in the embodiments of this disclosure are subject to the following interpretations:

[0055] A thread is the smallest unit of execution in a graphics processing unit (GPU) when performing data processing tasks. Each thread can independently perform the same pattern of processing on different data.

[0056] A warp is the basic unit of scheduling and execution in a graphics processing unit (GPU). It contains a fixed number of threads, typically 32 or 64. All threads execute the same instructions within the same clock cycle, but can operate on different data, enabling Single Instruction Multiple Data (SIMD) parallel computation. When a grid is activated (equivalent to a kernel being activated, with each kernel corresponding to its own grid), the grid contains workgroups. After a workgroup is assigned to a computational unit, it is divided into multiple warps. Within a warp, all threads execute in a single-instruction multiple-threaded manner, executing the same instructions at each step, but processing private data—data that corresponds to a specific thread.

[0057] Work Group (WG): A group of threads consisting of multiple threads, also known as a thread block. Threads in a single work group are generally used to perform the same pattern of processing on different data.

[0058] A grid consists of multiple workgroups. Each workgroup is a grid point in the grid, and each workgroup has corresponding grid point coordinates. A grid can be one-dimensional, two-dimensional, or three-dimensional. The shape of the grid is defined by the user during the programming process.

[0059] A Compute Unit (CU) is a processing module in electronic devices such as a graphics processing unit (GPU). In some architectures, a CU may also be called a Stream Multiprocessor (SM). A GPU can include multiple CUs. A CU can be considered a processing core within a GPU. Each CU can execute instructions independently, and the GPU can process data in parallel through multiple CUs.

[0060] Execution Unit (EU, also known as a multi-stream processor (SM)): It is the execution unit in a microprocessor. It is responsible for executing instructions and performing computational tasks such as arithmetic operations, logical operations, and shift operations. In fact, it has both the functions of a controller and an arithmetic unit.

[0061] Tensor processing unit (also known as tensor core): A hardware unit designed for efficient matrix operations, widely used in deep learning operations such as convolution and fully connected layers. Its core advantage lies in supporting mixed-precision computation, enabling large-scale matrix multiplication and addition operations to be completed within a single clock cycle, significantly improving computational throughput.

[0062] Matrix Multiply-Accumulate (MMA): A composite operation combining matrix multiplication and accumulation, widely used in convolutional and fully connected layers in deep learning. This operation can be represented as D = A × B + C, where A and B are the input matrices, C is the accumulation matrix, and D is the output matrix.

[0063] Shared memory (SMEM) refers to a certain amount of memory in a multiprocessor computer system that can be accessed by different processors. Because multiple processors need to access memory quickly, memory caching is necessary. When any cached data is updated, the shared memory needs to be updated immediately, otherwise different processors might use different data.

[0064] General Purpose Registers (GPRs) are used to store thread-private data, such as address indices, loop variables, or temporary calculation results. Each thread has its own independent GPR space, which is accessed extremely quickly and is a key resource for achieving thread-level parallel computing. In the Single Instruction Multithreaded (SIMT) execution model, GPRs and SMEMs work together: GPRs store thread-private states, while SMEMs carry shared data between thread bundles. Together, they support the efficient execution of high-concurrency computing tasks.

[0065] Scalar General Purpose Register (SGPR): Used to store scalar data, such as control flow variables, loop counters, and address offsets.

[0066] Vector General Purpose Register (VGPR): Used to store vector data, such as operands for floating-point or integer vector operations.

[0067] In modern GPU architectures, to cope with the ever-increasing computational demands, large-scale tensor processing units with higher parallelism and computational power, namely Super Tensor Cores (SMAs), have emerged. SMAs typically employ atomic instructions that bundle data loading with matrix multiplication and addition operations. This means that when the scheduling unit issues an MMA instruction, the SMA must simultaneously complete the entire process of reading data from global storage and performing computation.

[0068] Figure 1A This is a schematic diagram illustrating an instruction execution method for a tensor processing unit according to an embodiment of related technology. For example... Figure 1AAs shown, in a typical GPU execution model, computational tasks are scheduled to be executed on Compute Units (CUs) in units of Work Groups (WGs). Each Work Group contains multiple Warp Threads, and the Warp Thread is the basic unit of instruction scheduling and execution in the GPU. In a Work Group containing a mixed workload of Generalized Matrix Multiplication (GEMM) and Vector Operations (VECTOR), MMA computations are typically performed by Tensor Cores, acting as data "producers." Their outputs serve as inputs for subsequent vector operations performed by Execution Units (EUs), meaning VECTOR computations act as "consumers." During the Work Group initialization phase, the scheduling unit must allocate the necessary hardware resources to each Warp Thread within the Work Group, including the Vector General Purpose Register (VGPR), used to store intermediate data and operands required for vector computations. Since the results of MMA computations will be written to the VGPR resources used by subsequent VECTOR instructions, it is essential to ensure that these registers have been correctly allocated and are writable before the MMA computation. However, under current hardware architectures, if the target VGPR resource has not yet been allocated (e.g., due to resource contention or incomplete initialization), even if the tensor processing unit (TPU) has computational capabilities, it cannot initiate MMA computation to avoid data being written to unready registers, leading to illegal access or data conflicts. More critically, in traditional architectures, the matrix multiply-accumulate (MMA) instruction executed by the TPU is an atomic operation, tightly coupling loading matrix data from storage media with the execution of the MMA operation within the same instruction cycle. This tight coupling mechanism means that the scheduling unit can only trigger the execution of the MMA instruction, thereby initiating the combined data loading and matrix multiply-accumulate operation, when all participating threads have completed resource allocation, register states are ready, and instruction issuance conditions are met. Figure 1A For example, a workgroup consists of thread bundles 0 to X (where X is a positive integer). Each thread bundle needs to independently request and lock its required VGPR resource blocks before executing the VECTOR calculation. The allocation progress of these resources may vary due to different scheduling policies, resource fragmentation, or dependencies, resulting in the initialization completion time of the entire workgroup being determined by the slowest thread bundle. Under this waiting mechanism, the combined operation of data loading and matrix multiplication and addition can only be started after all participating thread bundles have completed VGPR resource allocation and are ready to execute, leading to idle computing resources and accumulated latency.

[0069] Figure 1BThis is a schematic diagram illustrating an instruction execution method for a tensor processing unit according to yet another embodiment of the related art. For example... Figure 1B The diagram illustrates the execution flow of a working group involving hybrid MMA and VECTOR computations across multiple iterations. In this scenario, MMA computation acts as the "producer," executed by the tensor processing unit, responsible for generating the data required for subsequent VECTOR computations. VECTOR computation, on the other hand, acts as the "consumer," executed by the execution unit, relying on the output of the MMA computation as its input operands. After the first round of MMA computation, the resulting computation is written to the VGPR resources corresponding to each thread bundle and consumed by subsequent VECTOR instructions. However, since these VGPR resources are still occupied before the VECTOR computation is complete and cannot be overwritten, a second round of MMA computation cannot be initiated immediately to generate new data; otherwise, data races and computational errors will occur. To coordinate resource reuse between multiple rounds of MMA and VECTOR computations, the system must introduce an explicit synchronization mechanism. For example... Figure 1B As shown, after each round of VECTOR computation, all thread bundles within the workgroup must perform a global synchronization operation (Multi-warp Sync) to ensure that all thread bundles have completed reading and processing the MMA output results in the current VGPR. Only when this synchronization operation is completed can it be considered that all relevant VGPR resources have been released and can be safely reused, at which point the next round of MMA computation can be started. However, due to the limitations of the existing hardware architecture design of the ultra-large tensor core, its matrix multiply-add (MMA) instruction still binds data loading and matrix multiply-add operation (MMA Operation) as atomic instructions at the hardware level, making it impossible to decouple loading matrix data from the global storage medium from performing matrix multiply-add operations. Therefore, even if the storage system is idle during the synchronization operation, it is impossible to load the next batch of matrix data from the global storage medium in advance. The initiation of data loading must strictly lag behind the completion of the synchronization operation, causing the tensor processing unit to wait for the synchronization operation to complete before initiating a high-latency data loading operation in each iteration, further extending the vacuolation cycle of the computation pipeline. This data supply lag caused by the combined effect of atomic instructions and synchronization dependencies severely limits the continuous throughput of MMA instructions, especially in deep learning operators that require frequent iterative execution (such as continuous matrix operations in recurrent neural networks or attention mechanisms), becoming a key obstacle to overall performance improvement.

[0070] In summary, due to the atomicity limitation of MMA instructions in the existing hardware architecture of ultra-large tensor cores, when the scheduling unit issues an MMA instruction, the ultra-large tensor core must simultaneously complete the entire process of reading data from the storage medium and performing computation. This tightly coupled design pattern causes the data loading latency to be directly exposed on the critical computation path, making it impossible to effectively utilize the high computing power of the ultra-large tensor core due to data supply bottlenecks, severely restricting the overall instruction throughput.

[0071] Based on this, this disclosure provides an instruction execution method, related apparatus, and medium to hide data loading latency by decoupling loading instructions from matrix multiplication and addition instructions and supporting asynchronous execution, thereby improving the computational efficiency of the tensor processing unit and the utilization of hardware resources, and ultimately increasing the overall instruction throughput.

[0072] System architecture description applied in the embodiments of this disclosure

[0073] Figure 2 This is a schematic diagram of the system architecture used in the instruction execution method of this disclosure embodiment. The system mainly includes a graphics processor 200, which includes a scheduling unit 210 for workgroup scheduling and multiple computing units 220. Computing units 0 to 3 are shown in the figure as an example, but it should be understood that the architecture is not limited thereto, and the graphics processor 200 may contain more or fewer computing units 220 according to actual design requirements. Multiple computing units 220 may be included in a cluster according to design requirements. Each computing unit 220 further includes a scheduling unit 221 for thread bundle scheduling, multiple execution units 222, one or more tensor processing units 230, and shared memory 224. Each execution unit 222 includes a register file 223. Execution units 0 to 3 are shown in the figure, and are also only for illustrative purposes; in actual implementation, any number of execution units 222 may be included. A tensor processing unit 230 is shown in the figure, and is also only for illustrative purposes; in actual implementation, any number of tensor processing units 230 may be included.

[0074] The graphics processing unit 200 is a computer processing device with parallel processing capabilities, suitable for high-concurrency, high-throughput computing tasks. Typical applications include graphics processing, scientific computing, and artificial intelligence training and inference. When processing graphics-related tasks, such as image recognition, 3D rendering, or animation generation, the graphics processing unit 200 has advantages in large-scale parallel computing. During task execution, the scheduling unit 210 is responsible for dividing the received computing tasks into several workgroups and assigning these workgroups to appropriate computing units 220 for processing. Each workgroup consists of multiple warps. After entering a computing unit, the internal scheduling unit 221 further distributes the warps to various execution units 222 for parallel computation. During execution, the register file 223 provides low-latency data storage and access support for each thread. The tensor processing unit 230 is responsible for accelerating matrix operations and deep learning-related operations, exhibiting high performance, especially when performing large-scale tensor calculations. It integrates a dedicated computing core and a high-bandwidth cache structure, and can directly participate in key operators such as convolution, matrix multiplication, and normalization in neural network models. In the task flow, the tensor processing unit 230 and the execution unit 222 collaborate, exchanging data and synchronizing tasks via a high-speed interconnect bus to ensure seamless switching of computational flows between different types of processing units. When the task involves deep learning inference or training, in a workgroup with a mixed workload of general matrix multiplication (GEMM) and vector operations (VECTOR), MMA computation is typically performed by the tensor processing unit 230, acting as the "producer" of data. Its output serves as the input for subsequent vector operations performed by the execution unit 222, meaning the VECTOR computation acts as the "consumer." Shared memory 224 serves as a global storage medium, used to temporarily store data and intermediate computation results that need to be shared between workgroups. Each computation unit 220 exclusively uses its allocated register file 223 and shared memory 224 physical resource areas, ensuring resource isolation between different computation units. Within the same computing unit 220, multiple execution units 222 and tensor processing units 230 can share access to the shared memory 224 resources allocated to that computing unit 220 to support inter-thread cooperation. The resources of the register file 223 are divided according to threads or execution units, with each execution unit 222 exclusively using its allocated register space. This ensures the independence of thread execution, avoids data conflicts, and improves overall execution efficiency. This architecture provides the hardware foundation for the subsequent implementation of fine-grained, non-contiguous resource allocation mechanisms.

[0075] Figure 3 This is a schematic diagram of the structure of a computing unit according to an embodiment of this disclosure. Figure 3As shown, the computation unit 220 includes multiple execution units 222, a scheduling unit 221, at least one tensor processing unit 230, shared memory 224, and a dedicated matrix data buffer associated with the tensor processing unit 230. The dedicated matrix data buffer is a static random access memory (SRAM) located near the tensor processing unit 230, or it can be an SRAM located inside the tensor processing unit 230. Execution units 222 include general-purpose registers, namely a scalar general-purpose register 331 and a vector general-purpose register 332. Tensor processing unit 230 includes at least one multiply-accumulate computation unit 310 and internal registers. Execution units 0 to 3 are shown as examples in the figure, but it should be understood that the architecture is not limited to this; the computation unit 220 may contain more or fewer execution units 222 depending on actual design requirements. The multiple execution units 222 in the computation unit 220 dynamically allocate thread bundles through the scheduling unit 221 and select either the scalar general-purpose register 331 or the vector general-purpose register 332 for data processing based on the instruction type. The figure shows the tensor processing unit 230 and the multiply-accumulate calculation unit 310 as examples, but it should be understood that the architecture is not limited to this. The calculation unit 220 may contain more or fewer tensor processing units 230 according to actual design requirements, and the tensor processing unit 230 may contain more or fewer multiply-accumulate calculation units 310 according to actual design requirements.

[0076] In this embodiment, the instruction set used by the tensor processing unit 230 includes independent tensor core load instructions and matrix multiplication-addition instructions. The load instructions load matrix data from storage media (e.g., shared memory 224) into a dedicated matrix data buffer associated with the tensor processing unit 230. The matrix multiplication-addition instructions instruct the tensor processing unit 230 to read matrix data from this dedicated matrix data buffer and perform calculations. The scheduling unit 221 is responsible for parsing and scheduling the load instructions and matrix multiplication-addition instructions, enabling them to execute asynchronously and independently. Specifically, when the scheduling unit 221 detects that the tensor processing unit 230 cannot immediately execute the matrix multiplication-addition instructions due to resource dependencies or unavailable hardware resources, it can prioritize issuing a load instruction to prefetch the matrix data required for subsequent calculations from shared memory 224 or other global storage media into the dedicated matrix data buffer associated with the tensor processing unit 230. This dedicated matrix data cache, serving as a high-speed temporary storage space close to the tensor processing unit, significantly reduces data access latency and avoids bandwidth bottlenecks and access latency issues caused by frequent access to main memory. By decoupling data loading operations from matrix multiplication and addition operations, the scheduling unit 221 can proactively initiate data prefetching during the waiting window when computing resources are not yet ready, thereby transforming the originally serial critical path into a parallel pipeline structure. This effectively hides the data loading latency and improves the continuous computing power and hardware utilization of the tensor processing unit 230.

[0077] Furthermore, when the tensor processing unit 230 finally meets the execution conditions, it can directly read the pre-fetched matrix data from the dedicated matrix data cache and load it into the internal register, then distribute it to multiple multiply-accumulate units 310 for parallel multiply-accumulate operations. Since the required data already exists in the low-latency local cache, the tensor processing unit 230 can quickly start the calculation process without waiting for external memory access to complete, greatly shortening the instruction issue latency. In addition, the dedicated matrix data cache can be divided into at least two logical partitions, for example, the first cache 341 is used to cache the input matrix A, and the second cache 342 is used to cache the input matrix B. In some embodiments, the dedicated matrix data cache can also be divided into an accumulation cache 343, which is used to temporarily store the intermediate calculation results of matrix multiplication and accumulation operations, i.e., the accumulation matrix C. The dedicated matrix data cache supports independent management and parallel loading of operands from different sources, further improving data scheduling flexibility and bandwidth utilization. This mechanism is not only applicable to resource preparation during the initialization phase of the workgroup, but also to pipeline optimization in multi-round iterative computation. Even in scenarios with producer-consumer dependencies or synchronization requirements between thread bundles, it can still achieve efficient data prefetching and computation overlap, significantly improving the overall instruction throughput.

[0078] Overall Implementation of the Instruction Execution Method of the Embodiments of this Disclosure

[0079] The instruction execution method provided in this disclosure is applied in... Figure 3 The computing unit 220 shown, Figure 4 This is a main flowchart of an instruction execution method according to an embodiment of this disclosure. Figure 4 As shown, the instruction execution method includes the following steps:

[0080] Step S401: The scheduling unit issues an independent loading instruction to load the matrix data required for multiplication and addition operations from the storage medium into the dedicated matrix data cache.

[0081] Step S402: The tensor processing unit is controlled to obtain the matrix data from the dedicated matrix data buffer and perform multiplication and addition operations by the matrix multiplication and addition operation instructions issued by the scheduling unit; wherein the loading instruction and the matrix multiplication and addition operation instructions are executed asynchronously and independently by the scheduling unit.

[0082] In some embodiments, when the scheduling unit 221 determines that a matrix multiplication-addition instruction cannot be executed immediately due to resource constraints, it preferentially issues a load instruction to preload the required matrix data from shared memory 224 or other global storage media into a dedicated matrix data cache. The resource constraints include, but are not limited to, the following situations: a) One or more thread bundles within the workgroup are undergoing resource initialization, resulting in the vector general-purpose registers (VGPRs) used to store computation results not yet being fully allocated. In the initial stage when the workgroup is scheduled to execute on the computation unit 220, the scheduling unit 221 needs to allocate necessary hardware resources for each thread bundle within the workgroup, especially the VGPRs used to store computation results. Since the output of the matrix multiplication-addition operation will be used as an operand for subsequent vector operations (such as the VECTOR instruction), it must be written into the allocated and writable VGPRs. If some thread bundles have not yet completed the allocation and initialization of VGPR resources (e.g., due to resource contention, scheduling delays, or memory allocation delays), the entire workgroup cannot safely start MMA computation; otherwise, it will cause register access conflicts or data overwriting risks. b) The output of the current matrix multiplication-addition operation is used as an operand for subsequent instructions, forming a data dependency chain, while the subsequent instructions have not yet released the register resources they occupy. In a typical "producer-consumer" computation model, the MMA computation performed by the tensor processing unit 230 is the "producer," and its output will serve as the input operand for subsequent VECTOR instructions executed by the execution unit 222. After the previous round of computation is completed, these results are temporarily stored in the VGPR for the consumer to use. Before the consumer instructions have finished executing and released the relevant register resources, the producer cannot write to the same register again, otherwise it will lead to data races or computation errors. Therefore, even if the conditions for a new batch of computation are partially met, the MMA computation still needs to wait for the dependency chain to be released before it can be executed. c) There are synchronization operations between the thread bundles within the workgroup, and a new round of computation can only be started after global synchronization is completed. In scenarios where multiple thread bundles execute in parallel, in order to ensure data consistency and execution order, the workgroup usually needs to insert synchronization instructions (such as Multi-warp Sync) at key computation nodes. For example, after a round of MMA computation and subsequent VECTOR operations are completed, all thread bundles must be synchronized to ensure that each thread bundle has consumed the computation results of the previous round, and the relevant VGPR resources can be temporarily released for safe reuse.

[0083] In summary, the three resource limitations mentioned above will prevent matrix multiplication and addition instructions from being executed in the short term, but they do not affect the independent execution of loading instructions. This disclosure makes full use of these "waiting windows" and implements data prefetching through an asynchronous loading mechanism, effectively decoupling computation and memory access operations on the critical path, thereby improving the utilization rate of the tensor processing unit 230 and the overall system performance.

[0084] Figure 5A This is a schematic diagram of loading instructions and matrix multiplication / addition operation instructions according to an embodiment of this disclosure. Figure 5A As shown, the loading instruction and the matrix multiplication-addition instruction are represented as two independent and separable execution stages on the timeline. When it is determined that the matrix multiplication-addition instruction cannot be executed immediately due to resource constraints, the loading instruction is preemptively issued and executed by the scheduling unit 221. Its function is to load the input matrix A and input matrix B stored in shared memory 224 or other global storage media into the first cache 341 and the second cache 342 in the dedicated matrix data buffer, respectively. This process can be completed independently when the tensor processing unit 230 does not yet have the execution conditions, without waiting for the subsequent computing resources to be ready. The matrix multiplication-addition instruction is scheduled for execution at a later time, triggered by the allocation of relevant register resources, the satisfaction of data dependencies, and the completion of inter-thread synchronization operations. At this time, the tensor processing unit 230 reads the prefetched matrix data from the dedicated matrix data buffer, loads it into internal registers, and then distributes it to multiple multiplication-addition calculation units 310 to perform efficient matrix multiplication-addition operations, temporarily storing the calculation results in the third cache 343. The internal registers can be divided into a first register 321 and a second register 322. For example, the first register 321 is used to cache input matrix A, and the second register 322 is used to cache input matrix B. By decomposing the atomic "load-computation" operation in the existing GPU architecture into two asynchronously schedulable stages, this disclosure effectively breaks the limitation in the traditional architecture that data loading must strictly lag behind the completion of resource preparation. This allows high-latency memory access operations to be completed ahead of schedule during computational idle periods, significantly improving the parallel utilization of hardware resources.

[0085] Furthermore, before executing the matrix multiplication-addition operation instruction, the tensor processing unit 230 first determines whether the required matrix data already exists completely in the dedicated matrix data buffer. If yes, it directly reads the matrix data from the dedicated matrix data buffer into its internal register for operation. The data is then broadcast to multiple multiplication-addition calculation units 310 for parallel processing to improve data reuse and computational efficiency. If no, it triggers a supplementary loading operation on shared memory 224 or other global storage media to ensure the integrity of the data required for computation. This determination mechanism enables the tensor processing unit 230 to dynamically respond to data readiness status, supporting both efficient execution paths based on prefetching and fault tolerance mechanisms when data is not ready. In addition, after the loading instruction is executed, its completion status is recorded in the scheduling unit 221 or the dependency tracking logic of the tensor processing unit 230, serving as the basis for subsequent instruction scheduling and data dependency judgment. This mechanism ensures that matrix multiplication and addition instructions are only triggered to execute after the required data has been successfully loaded into the dedicated matrix data buffer. Through the above mechanism, this disclosure achieves spatiotemporal decoupling of loading and computation, significantly improving the actual computational efficiency and hardware utilization of ultra-large tensor kernels in complex dependency scenarios.

[0086] In this embodiment, the scheduling unit 221 is responsible not only for the initialization and instruction issuance of the workgroup, but also for the full lifecycle management of the vector general-purpose register resources belonging to the workgroup. Specifically, the scheduling unit 221 is responsible for determining whether the VGPR resource has been successfully requested, whether it has been released, and whether the register address space reserved for subsequent computation tasks is available. In the execution flow containing a mixed workload of matrix multiplication and vector operations, the tensor processing unit 230 acts as a "producer," and its calculation results need to be written into the VGPR used by the execution unit 222 for consumption by subsequent "consumer" instructions. Therefore, the scheduling unit 221 must ensure that the target VGPR resource has been correctly allocated and is in a writable state before triggering the tensor processing unit 230 to perform MMA calculations; otherwise, data writing will fail or resource conflicts will occur. Furthermore, the tensor processing unit 230 itself does not directly participate in the allocation and status management of VGPR resources. Instead, it uses the scheduling unit 221 to provide key parameters for determining the specific register address to which the calculation result of the matrix multiplication-addition operation instruction should be written. These key parameters include: the initial offset, the stride offset, and the number of thread bundles on each execution unit 222. The vector general-purpose register space of the multiple execution units 222 is a contiguous storage space. Specifically: the initial offset refers to the starting address of the result register corresponding to the first thread bundle on each execution unit 222; the stride offset represents the address spacing between adjacent warps in the VGPR space within the same execution unit 222, reflecting the step size of the register layout; and the number of warps on a single execution unit 222 defines the number of warps handled by each execution unit 22. Based on these three parameters, the scheduling unit 221 determines the starting address of the register corresponding to the i-th thread bundle in each execution unit 222 as the starting address of the contiguous storage space + the initial offset + (i-1). The interval offset, where i is the number of thread bundles allocated within the corresponding execution unit, guides the tensor processing unit 230 to accurately write the calculation results into the pre-allocated VGPR address space. Figure 5B This is a schematic diagram illustrating the allocation of vector general-purpose register resources within one or more thread bundles in a workgroup according to an embodiment of this disclosure. Figure 5BFor example, scheduling unit 221 distributes the thread bundles in the workgroup to execution units 0 to 3, ensuring that each execution unit 222 is allocated 2 thread bundles, thus guaranteeing workload balance among the execution units. The VGPR space allocated to execution units 0 to 3 can be considered as a contiguous address space. Let the starting address of this contiguous VGPR space be A. Then, the starting address of the register corresponding to the first thread bundle in execution unit 0 is A + initial offset 0; the starting address of the register corresponding to the first thread bundle in execution unit 1 is A + initial offset 1; the starting address of the register corresponding to the first thread bundle in execution unit 2 is A + initial offset 2; and the starting address of the register corresponding to the first thread bundle in execution unit 3 is A + initial offset 3. The starting address of the register corresponding to the second thread bundle in execution unit 0 is A + initial offset 0 + (2-1). The interval offset is as follows: the starting address of the register corresponding to the second thread bundle in execution unit 1 is A + initial offset 1 + (2-1) × interval offset; the starting address of the register corresponding to the second thread bundle in execution unit 2 is A + initial offset 2 + (2-1) × interval offset; and the starting address of the register corresponding to the second thread bundle in execution unit 3 is A + initial offset 3 + (2-1) × interval offset.

[0087] Furthermore, after completing the address mapping configuration of VGPR resources, the scheduling unit 221 further transmits the initial offset, interval offset, and number of thread bundles to the tensor processing unit 230. This allows the tensor processing unit 230 to automatically generate the result write address corresponding to each thread bundle during matrix multiplication and addition operations, based on the address calculation rules, without explicitly encoding the physical address of each target register in the instruction stream. This mechanism, where the scheduling unit 221 uniformly plans and the tensor processing unit addresses according to the rules, not only reduces the complexity of instruction encoding but also improves the flexibility and scalability of hardware execution. Especially when processing large-scale matrix operations, when the number of thread bundles in the workgroup changes dynamically or the allocation strategy of the execution unit is adjusted, the scheduling unit only needs to recalculate and issue new offset parameters to adapt to different resource layouts without modifying the underlying execution logic of the tensor processing unit. Meanwhile, this mechanism supports fine-grained register space partitioning, allowing computing tasks from different workgroups or at different stages to share the same VGPR address space, as long as they do not overlap in time and the scheduling unit 221 can correctly maintain the release and reuse status of resources, thereby improving the utilization of register resources and reducing memory fragmentation.

[0088] Figure 5C This is a schematic diagram illustrating an instruction execution method for a tensor processing unit according to an embodiment of the present disclosure. Figure 5CAs shown, during the workgroup initialization phase, the scheduling unit 221 needs to allocate necessary hardware resources to each thread bundle within the workgroup, including vector general-purpose registers (GGPRs) for storing intermediate data and operands required for vector operations. Since the calculation results of matrix multiplication-addition instructions will be written to the VGPR resources used by subsequent VECTOR instructions, it is essential to ensure that these registers are correctly allocated and writable before the matrix multiplication-addition instructions are executed. However, unlike traditional architectures that must wait for all thread bundles to complete VGPR resource allocation before data loading can begin, this embodiment allows the scheduling unit 221 to issue a loading instruction in advance, even before the VGPR resources of some thread bundles are ready. This loads the operands required for matrix multiplication-addition operations from shared memory 224 or other global storage media into the dedicated matrix data buffer associated with the tensor processing unit 230. Since this loading operation does not depend on the register allocation state of the execution unit 222, nor does it involve direct writing to the VGPR, matrix multiplication-addition instructions can be executed in parallel during resource initialization, fully utilizing idle periods of storage bandwidth. This mechanism breaks the strong dependence of traditional atomic instructions on resource readiness, freeing data prefetching from the initialization progress of the slowest thread bundle, thus effectively alleviating the overall latency problem caused by uneven resource allocation. Even if some thread bundles in the workgroup fail to complete VGPR allocation in time due to scheduling delays or resource contention, the data required by the remaining thread bundles can still be preloaded and cached, preparing for subsequent rapid computation. Furthermore, after each thread bundle has completed VGPR resource allocation, the scheduling unit 221 can trigger the execution of matrix multiplication and addition instructions on demand or in batches. At this time, the tensor processing unit 230 reads the prefetched matrix data from the dedicated matrix data buffer and loads it into the internal register for calculation, finally writing the result into the ready VGPR resources for subsequent vector operations. Since data loading has been completed in advance, the execution of matrix multiplication and addition instructions can be started almost immediately, significantly shortening the response latency between resource readiness and the start of actual computation. Furthermore, the scheduling unit 221 can implement fine-grained scheduling strategies based on the resource readiness status of each thread bundle, such as prioritizing the execution of MMA calculations for thread bundles whose data is ready and registers have been allocated, thereby achieving more flexible task scheduling and higher hardware utilization. By decoupling loading from computation, this disclosure not only improves the parallel efficiency of the workgroup initialization phase but also lays the foundation for pipeline optimization in subsequent iterations, enabling the tensor processing unit 230 to maintain high throughput and low latency continuous computation capabilities even in complex dependency and asynchronous resource allocation environments.

[0089] Figure 5D This is a schematic diagram illustrating an instruction execution method for a tensor processing unit according to an embodiment of the present disclosure. Figure 5DAs shown, in the iterative execution process involving multiple rounds of hybrid MMA and VECTOR computations, the MMA computation is performed by the tensor processing unit 230, acting as a "producer" to generate the data required for subsequent vector operations, while the VECTOR computation is performed by the execution unit 222, acting as a "consumer" to process the output results of the MMA. In the first round of computation, the scheduling unit 221 first issues a loading instruction to load the first batch of matrix data from shared memory 224 into a dedicated matrix data cache, and after the resources are ready, issues matrix multiplication and addition instructions to complete the first round of MMA computation, the results of which are written into the VGPR corresponding to each thread bundle. Subsequently, the execution unit 222 executes the VECTOR instructions to process the data in the VGPR. During this period, the traditional architecture, due to the binding of loading and computation, cannot prepare the data for the next round in advance, resulting in idle storage system. In this embodiment, the scheduling unit 221 can determine that the data required for the next round of MMA calculation is available while the first round of VECTOR calculation is still executing, and pre-fetch the data into a dedicated matrix data buffer, thus achieving parallel execution of data loading and the current calculation task. This pre-fetching operation does not need to wait for the completion of the current round of VECTOR calculation, nor is it restricted by the global synchronization mechanism; it only needs to ensure that the dedicated matrix data buffer has sufficient space and that the data source is valid to start. After the first round of VECTOR calculation ends and the global synchronization operation is performed, the relevant VGPR resources are released, and the scheduling unit 221 immediately issues the second round of matrix multiplication and addition operation instructions. The tensor processing unit 230 then directly reads the pre-fetched second batch of data from the dedicated matrix data buffer and quickly starts the matrix multiplication and addition operation calculation process. Since the data loading has been completed in the previous stage, the second round of MMA calculation does not need to undergo high-latency global memory access, significantly shortening the pause time between iterations. Especially in typical deep learning workloads, such as the self-attention mechanism of the Transformer model or the temporal propagation process of recurrent neural networks, such multi-round matrix operations occur frequently. The asynchronous loading and decoupled execution MMA computing mechanism provided in this disclosure can effectively eliminate the data supply wait between each round, thereby significantly improving the overall computing efficiency and hardware resource utilization.

[0090] Description of apparatus and devices according to embodiments of this disclosure

[0091] Reference Figure 6 , Figure 6 This is a schematic diagram of the structure of an instruction execution device 600 disclosed herein. The instruction execution device 600 is integrated into a computing unit and applied to a graphics processor. The graphics processor includes a scheduling unit, at least one tensor processing unit, and a dedicated matrix data cache associated with the tensor processing unit. The instruction execution device 600 includes:

[0092] The matrix data loading unit 610 is used to load the matrix data required for multiplication and addition operations from the storage medium into the dedicated matrix data cache area by issuing independent loading instructions through the scheduling unit.

[0093] The multiply-add operation execution unit 620 is used to control the tensor processing unit to obtain the matrix data from the dedicated matrix data buffer and perform multiply-add operations through the matrix multiply-add operation instructions issued by the scheduling unit; wherein the loading instruction and the matrix multiply-add operation instructions are executed asynchronously and independently by the scheduling unit.

[0094] The instruction execution apparatus disclosed herein is used to execute the instruction execution method as described in the above embodiments. Its specific processing procedure is the same as that of the instruction execution method described in the above embodiments, and will not be repeated here.

[0095] This disclosure also provides an electronic device 700, comprising:

[0096] At least one processor, and,

[0097] A memory that is communicatively connected to at least one processor; wherein,

[0098] The memory stores instructions that are executed by at least one processor to cause the at least one processor to perform the method as described in any of the above embodiments of this application when executing the instructions.

[0099] The following is combined Figure 7 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 710, a memory 720, an input / output interface 730, a communication interface 740, and a bus 750.

[0100] The processor 710 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0101] The memory 720 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 720 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 720 and is invoked by the processor 710 to execute the instruction execution method of the embodiments of this disclosure.

[0102] The input / output interface 730 is used to implement information input and output;

[0103] The communication interface 740 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0104] Bus 750 transmits information between various components of the device (e.g., processor 710, memory 720, input / output interface 730, and communication interface 740);

[0105] The processor 710, memory 720, input / output interface 730 and communication interface 740 are connected to each other within the device via bus 750.

[0106] This application also provides a computer-readable storage medium that stores one or more programs, which can be executed by one or more processors to implement the instruction execution method of the above embodiments, which will not be described again here.

[0107] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0108] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0109] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0110] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0112] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0113] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0114] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A method of instruction execution, applied to a graphics processor, wherein, The graphics processor includes a scheduling unit, at least one tensor processing unit, and a dedicated matrix data buffer associated with the tensor processing unit. The instruction execution method includes: The scheduling unit issues independent loading instructions to load the matrix data required for multiplication and addition operations from the storage medium into the dedicated matrix data cache area. The scheduling unit issues matrix multiplication and addition instructions to control the tensor processing unit to obtain the matrix data from the dedicated matrix data buffer and perform multiplication and addition operations. The loading instruction and the matrix multiplication and addition operation instruction are executed asynchronously and independently by the scheduling unit.

2. The instruction execution method of claim 1, wherein, The dedicated matrix data cache is a static random access memory located inside the tensor processing unit.

3. The instruction execution method of claim 1, wherein, The loading instruction issued by the scheduling unit includes: Determine whether the matrix multiplication and addition instruction cannot be executed immediately due to resource limitations; If so, the loading instruction is issued first to preload the matrix data required for multiplication and addition operations into the dedicated matrix data cache.

4. The instruction execution method of claim 3, wherein, The resource restrictions include at least one of the following situations: (a) One or more thread bundles within the workgroup of the graphics processor are undergoing resource initialization, resulting in the register resources used to store the calculation results of the matrix multiplication and addition operation instructions not being fully allocated; (b) The calculation result of the matrix multiplication and addition instruction is the operand of other subsequent instructions, forming a data dependency chain, and the subsequent instructions have not yet released the register resources they occupy; (c) Synchronization operations exist between the threads within the workgroup, causing the matrix multiplication and addition instructions to wait for the synchronization to complete before they can be executed.

5. The instruction execution method according to claim 1, characterized in that, The tensor processing unit executes the matrix multiplication and addition operation instructions including: Determine whether the matrix data required for the multiplication-addition operation has been pre-stored in the dedicated matrix data cache area; If so, the matrix data is directly read from the dedicated matrix data buffer into its internal register for calculation; If not, then the operation of loading matrix data from the storage medium into the dedicated matrix data cache is triggered.

6. The instruction execution method according to claim 5, characterized in that, The internal registers include a first register and a second register, and the tensor processing unit further includes the following when executing the matrix multiplication and addition operation instructions: The matrix data read from the dedicated matrix data buffer is loaded into the first register and the second register, respectively; The matrix data loaded into the first register and the second register are distributed to multiple multiply-accumulate units for parallel multiply-accumulate calculations.

7. The instruction execution method according to claim 1, characterized in that, The dedicated matrix data cache is divided into at least two logical partitions, which are used to cache operands from different sources in matrix multiplication operations.

8. The instruction execution method according to claim 1, characterized in that, The calculation result of the matrix multiplication and addition operation instruction needs to be written into a vector general-purpose register used by multiple execution units in the graphics processor. The scheduling unit allocates vector general-purpose register resources to one or more thread bundles within the workgroup, including: The scheduling unit provides key parameters for determining the specific register address to which the calculation result of the matrix multiplication and addition instruction should be written. These key parameters include: the initial offset, interval offset, and number of thread bundles on each execution unit. The vector general-purpose register space of the multiple execution units is a contiguous storage space. The scheduling unit determines the starting address of the register corresponding to the i-th thread bundle in each execution unit as the starting address of the contiguous storage space + initial offset + (i-1). Interval offset, where i is the number of thread bundles allocated within the corresponding execution unit.

9. The instruction execution method according to claim 1, characterized in that, After the loading instruction is executed, its state is recorded so that the scheduling unit or the tensor processing unit can determine the dependency relationship and schedule subsequent instructions.

10. A graphics processor, comprising: Tensor processing unit, used to perform multiplication and addition operations; The scheduling unit is configured to issue a loading instruction and a matrix multiplication and addition operation instruction according to the instruction execution method as described in any one of claims 1 to 9, so as to control the tensor processing unit to asynchronously execute the loading instruction and the matrix multiplication and addition operation instruction.

11. An instruction execution device, applied to a graphics processor, wherein, The graphics processor includes a scheduling unit, at least one tensor processing unit, and a dedicated matrix data buffer associated with the tensor processing unit. The instruction execution device includes: The matrix data loading unit is used to load the matrix data required for multiplication and addition operations from the storage medium into the dedicated matrix data cache area by issuing independent loading instructions through the scheduling unit. The multiply-add operation execution unit is used to control the tensor processing unit to obtain the matrix data from the dedicated matrix data buffer and perform multiply-add operations through the matrix multiply-add operation instructions issued by the scheduling unit; The loading instruction and the matrix multiplication and addition operation instruction are executed asynchronously and independently by the scheduling unit.

12. An electronic device comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory, wherein the program is executed by the processor to implement the instruction execution method as described in any one of claims 1 to 9.

13. A computer-readable storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the instruction execution method as described in any one of claims 1 to 9.

Images