Processors, chip products, computer equipment and instruction processing methods

Abstract

A processor, chip product, computer device, and instruction processing method are disclosed, relating to the field of processor technology. The processor includes a tensor computation engine and at least two memory processing elements. Each memory processing element runs n computation thread bundles, and each computation thread bundle corresponds to multiple synchronization entries. The memory processing element is used to invoke a first computation thread bundle running on it to generate computation instructions, which instruct the tensor computation engine to execute computation tasks. If any of the multiple synchronization entries corresponding to the first computation thread bundle are unused, the computation instructions are sent to the tensor computation engine. The tensor computation engine is used to record the synchronization status of the computation instructions through a first synchronization entry, which is any unused synchronization entry among the multiple synchronization entries. This application improves the processor's bandwidth utilization and computational resource utilization by setting multiple synchronization entries for each computation thread bundle.

Description

Technical Field

[0001] This application relates to the field of processor technology, and in particular to a processor, chip product, computer device, and instruction processing method. Background Technology

[0002] Currently, a processor may include a TCE (Tensor Compute Engine) and multiple MPEs (Memory Processing Elements). The TCE is responsible for processing tensor data to accelerate computation, while the MPE is responsible for memory-related management and processing tasks.

[0003] In related technologies, when the same computational task is divided into multiple computational wave bundles (waves), the processor allocates these wave bundles to multiple machine learning processors (MPEs). When the processor's execution center (TCE) processes this computational task, it needs to ensure data synchronization from the multiple MPEs. Therefore, each MPE needs to send an instruction to the TCE for its assigned computational wave bundle. The TCE will only prepare to read the source operands of the multiple instructions, i.e., the input data required for computation, after detecting that the instructions sent by the multiple MPEs have all arrived.

[0004] However, in related technologies, each computing thread bundle sends at most one instruction to the TCE at a time, resulting in low utilization of processor bandwidth resources. Summary of the Invention

[0005] This application provides a processor, a chip product, a computer device, and an instruction processing method. The technical solution provided by this application includes the following:

[0006] According to one aspect of the embodiments of this application, a processor is provided, the processor comprising: a tensor computation engine and at least two memory processing elements, wherein n computation thread bundles are running on the memory processing elements, and each computation thread bundle is configured with a plurality of synchronization entries, where n is a positive integer; The memory processing element is used to call the first computation thread bundle among the n computation thread bundles running on the memory processing element to generate a computation instruction, which is used to instruct the tensor computation engine to perform a computation task; if there is an unused synchronization entry among the multiple synchronization entries corresponding to the first computation thread bundle, the computation instruction is sent to the tensor computation engine. The tensor computation engine is used to record the synchronization status of the computation instructions through a first synchronization entry, wherein the first synchronization entry is any unused synchronization entry among the plurality of synchronization entries.

[0007] According to one aspect of the embodiments of this application, a chip product is provided, the chip product including the processor described above.

[0008] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including the processor described above.

[0009] According to one aspect of the embodiments of this application, an instruction processing method for a processor is provided, the processor including: a tensor calculation engine and at least two memory processing elements, wherein n calculation thread bundles are running on the memory processing elements, and each calculation thread bundle is configured with multiple synchronization entries, where n is a positive integer; The method includes: The memory processing element calls the first of the n computation thread bundles running on the memory processing element to generate computation instructions, which are used to instruct the tensor computation engine to perform computation tasks. If there is an unused synchronization entry among the multiple synchronization entries corresponding to the first computation thread bundle, the memory processing element sends the computation instruction to the tensor computation engine. The tensor computation engine records the synchronization status of the computation instructions through a first synchronization entry, which is any unused synchronization entry among the plurality of synchronization entries.

[0010] The technical solutions provided in this application embodiment may include the following beneficial effects.

[0011] By setting multiple synchronization entries for each computation thread bundle, the memory processing element can continuously send multiple computation instructions to the tensor computation engine based on the usage status of these synchronization entries. This improves bandwidth utilization between the memory processing element and the tensor computation engine. Furthermore, it allows the tensor computation engine to complete the execution of the current computation instruction without waiting for the transmission and synchronization of the next instruction, reducing idle time and thus improving the computational resource utilization of the tensor computation engine, which in turn improves the processor's bandwidth and computational resource utilization. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a processor provided in one possible implementation of this application; Figure 2 This is a schematic diagram of a memory processing element provided in one possible implementation of this application; Figure 3 This is a schematic diagram of an arithmetic logic unit provided in one possible implementation of this application; Figure 4 This is a schematic diagram illustrating the connection method of the arithmetic logic unit provided in one possible implementation of this application; Figure 5 This is a schematic diagram illustrating the connection method of the arithmetic logic unit provided in another possible implementation of this application; Figure 6 This is a schematic diagram of a processor provided in another possible implementation of this application; Figure 7 This is a schematic diagram illustrating the execution order of computational instructions that do not have data dependencies, provided in one possible implementation of this application. Figure 8 This is a schematic diagram illustrating the execution order of computational instructions with data dependencies provided in one possible implementation of this application; Figure 9 This is a schematic diagram showing the breakdown of computation instructions provided in one possible implementation of this application; Figure 10 This is a schematic diagram showing the distribution of execution results provided in one possible implementation of this application; Figure 11 This is a schematic diagram of a write-back cache device provided in one possible implementation of this application; Figure 12 This is a flowchart of an instruction processing method applied to a processor provided in one possible implementation of this application; Figure 13 This is a simplified structural block diagram of a computer device provided in one possible implementation of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0015] Please refer to Figure 1 The diagram illustrates a processor provided in one possible implementation of this application. The processor 100 may include a tensor computation engine 101 and a plurality of memory processing elements 102.

[0016] The processor 100 described above may be a processor that supports tensor processing, such as processor 100 supporting at least one of the following operations for tensors: transfer, storage, and computation. For example, the memory processing element 102 in processor 100 supports tensor transfer and tensor storage, and the tensor computation engine 101 in processor 100 supports tensor computation. Here, tensor transfer can refer to the transfer of tensor data corresponding to a tensor, tensor storage can refer to the storage of tensor data corresponding to a tensor, and tensor computation can refer to the computation of tensor data corresponding to a tensor.

[0017] Tensors are a general term and extension of concepts such as scalars, vectors, and matrices. They are a fundamental data structure in the field of machine learning and can be understood as multidimensional arrays containing at least one element. This application does not limit the dimension of tensors, allowing for formats such as one-dimensional, two-dimensional, and three-dimensional tensors. Tensor computation mainly refers to the calculations and processing performed on tensors in deep learning and machine learning. These calculations typically involve operations such as matrix multiplication and convolution, and are crucial steps in machine learning model training and inference.

[0018] The processor 100 described above can be a completely new design or an improvement upon an existing processor. Exemplarily, the processor 100 can be implemented as at least one of the following: a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a tensor processing unit (TPU), a field-programmable gate array (FPGA), a tensor core, or a neural processing unit (NPU). This application does not limit the processor 100.

[0019] The tensor computation engine 101 mentioned above refers to the computation engine in the processor 100 used to perform tensor computations. The tensor computation engine 101 supports at least one of the following operational logics: matrix multiplication, convolution, multiply-accumulate, and addition. For example, the tensor computation engine 101 can be used to implement multiply-accumulate operations on tensors to obtain the result of the tensor multiply-accumulate operation. This application embodiment does not limit the number of tensor computation engines 101 included in the processor 100, which can be set and adjusted according to actual usage requirements.

[0020] Tensor computation engine 101 can be used to read tensor data from memory processing element 102 to perform logical operations on the tensor data, thereby realizing tensor computation. Tensor data refers to the data used to describe a tensor, which can consist of encoded data generated for each element of the tensor.

[0021] The aforementioned memory processing element 102 is responsible for memory-related management and processing tasks, such as managing and processing memory operations to support the computing needs of the processor 100. For example, the memory processing element 102 is responsible for data transfer, address translation, and memory consistency maintenance to ensure efficient data supply required by the tensor computing engine 101. This application embodiment does not limit the number of memory processing elements 102 included in the processor 100; they can be set and adjusted according to actual usage requirements.

[0022] Optionally, each tensor calculation engine 101 may correspond to a memory processing element group, such as each tensor calculation engine 101 may correspond to a memory processing element group consisting of 4 memory processing elements 102. Each memory processing element 102 in the memory processing element group establishes a data output channel (such as a bus) with the tensor calculation engine 101 corresponding to the memory processing element group to realize operations such as tensor transmission and execution result storage.

[0023] The memory processing element 102 can autonomously generate computation instructions to inform the tensor computation engine 101 of the computation tasks to be performed, such as computation tasks for processing tensors. Exemplarily, the computation instructions may include at least one of the following: input data corresponding to the computation task, the storage address of the input data, the operational logic for manipulating the input data, and the storage address of the execution result of the computation instructions; this embodiment does not limit these. The computation instructions executed by the TCE may be referred to as TCE instructions, such as low-level hardware instructions that the TCE can directly understand and execute. Based on the computation instructions, the TCE reads source operands from the corresponding memory processing element 102 to perform tensor computation, obtains the execution result of the computation instructions, and stores the execution result in the memory processing element 102.

[0024] In one example, refer to Figure 2Each tensor computation engine 101 corresponds to four memory processing elements 102. Each memory processing element 102 may include multiple registers, such as 128kb registers, shared registers, xlot registers, etc. Registers can be used to store tensor data, such as matrix data, or to store the execution results of tensors, such as the execution results of computation instructions.

[0025] Each memory processing element 102 may further include: an FP (Floating Point Unit), an INT (Integer) unit, an SFU (Special Function Unit), an LSU (LoadStore Unit), and a TME (Tensor Memory Engine). The LSU is responsible for handling data read / write operations between the processor 100 and memory, while the TME refers to the computing engine in the processor 100 used for tensor transfer and tensor storage.

[0026] The tensor computation engine 101 may include a control logic unit and multiple ALUs (Arithmetic Logic Units). The number of ALUs may be the same as the number of memory processing elements 102, such as one ALU corresponding to each memory processing element 102. Data sharing is possible among the multiple ALUs. This application embodiment does not limit the number or connection method of the ALUs.

[0027] like Figure 2 As shown, each tensor computation engine 101 may include four ALUs. The control logic unit is used to schedule and manage tensor operations, such as receiving computation instructions from the memory processing element 102, decoding them into low-level micro-operation signals, and controlling the flow of data within the ALUs. The ALU is the core computational unit in the tensor computation engine 101 responsible for performing basic mathematical and logical operations. The ALU can receive low-level micro-operation signals from the control logic unit, as well as source operands from registers or caches, for computation.

[0028] refer to Figure 3Each arithmetic logic unit 1011 can be implemented as a processing unit array, which may include multiple processing units (DPs, or Data Processing Units) for performing logical operations. This application embodiment does not limit the number of processing units in the processing unit array, such as 8×8=64, 4×6=32, or 8×16=128. This application embodiment also does not limit the types of logical operations that the processing units can perform; the logical operations supported by the processing units include at least one of the following: matrix multiplication, convolution, multiply-accumulate, and addition. This application embodiment does not limit the structure of the processing units, which can be set and adjusted according to computing power requirements. For example, when a processing unit is used to perform multiply-accumulate operations, the processing unit may include a multiplier, an adder, etc. The structures of each processing unit can be identical.

[0029] Optionally, the control logic unit can connect to each processing unit in each arithmetic logic unit 1011, and can be used to assign underlying micro-operation signals and tensors to the processing units.

[0030] Optionally, each tensor computation engine 101 may also be provided with a corresponding shared memory unit 103. The shared memory unit 103 can be used to temporarily store data to be processed, and this data can be shared by multiple ALUs. For example, the shared memory unit 103 can be implemented as an on-chip cache, referred to as local memory, to temporarily store tensor data, thereby improving the speed at which the tensor computation engine 101 retrieves tensor data. The shared memory unit 103 can also be implemented as a register to temporarily store tensor data, thereby further improving the speed at which the tensor computation engine 101 retrieves tensor data (the data read speed of a register is much faster than the data read speed of memory). The shared memory unit 103 can also be implemented as a Level 1 cache in the processor 100, i.e., an L1 cache, which is the cache layer closest to the tensor computation engine 101. It typically has a small storage space but a very fast data read speed. For example, as... Figure 2 As shown, the shared storage unit 103 may include LMS (Large Model Support) and L1 cache.

[0031] For example, refer to Figure 4The four ALUs are connected in a 1×4 configuration. Each of the four MPEs (MPEs) from MPE0 to MPE3 stores matrix A. The control logic unit of the TCE can read matrix A from these four MPEs into the corresponding ALU. Taking ALU0 as an example, the TCE can read matrix A from the register of the MPE0 corresponding to ALU0 into ALU0. The control logic unit of the TCE can read matrix B from the LMS of the shared memory unit 103 and broadcast matrix B to ALU0 through ALU3. After the calculations are completed by these four ALUs, the TCE stores the calculation results (i.e., matrix D) of these four ALUs into the corresponding MPEs, thereby achieving data sharing.

[0032] For example, refer to Figure 5 The four ALUs are connected in a 2×2 configuration. MPE0 stores matrix A0, and MPE1 stores matrix A1. The TCE's control logic unit can read matrix A0 from MPE0 into ALU0 and ALU1, and read matrix A1 from MPE1 into ALU2 and ALU3. MPE2 stores matrix B0, and MPE3 stores matrix B1. The TCE's control logic unit can read matrix B0 from MPE2 into ALU1 and ALU3, and read matrix B1 from MPE3 into ALU1 and ALU3, thus achieving data sharing.

[0033] In one example, a tensor computation engine 101, four memory processing elements 102, and a shared memory unit 103 can constitute a processing unit (P) 110. In this case, as... Figure 6 As shown, processor 100 may include multiple processor clusters (PCs) 130 (e.g., 4), each processor cluster 130 may include multiple processor execution engines (PXs) 120 (e.g., 4), and each processor execution engine 120 may include multiple processing units 110 (e.g., 2). The multiple processor execution engines 120 in each processor cluster 130 share an L2 cache 140. The storage space of the L2 cache is larger than that of the L1 cache, but the data read speed of the L2 cache is slower than that of the L1 cache. The main memory of processor 100 can be implemented as an L3 cache 150.

[0034] In this embodiment, each memory processing element 102 runs n computational thread bundles (waves or warps), where n is a positive integer. For example, each memory processing element 102 may run one or more computational thread bundles. A computational thread bundle is the basic execution unit scheduled by the processor 100, and it may include multiple threads, such as 32 threads, 64 threads, etc. Multiple threads in a computational thread bundle execute the same instructions synchronously but process different data. Optionally, the memory processing element 102 generates TCE instructions for the computational thread bundle to invoke TCE to accelerate computation.

[0035] Optionally, the computation thread bundles on the multiple memory processing elements 102 are associated. For example, the processor 100 includes n thread bundle groups, each thread bundle group including one computation thread bundle on the multiple memory processing elements 102. The multiple computation thread bundles in the thread bundle group can be referred to as associated computation thread bundles. Exemplarily, each thread bundle group corresponds to a computation task, which is divided into multiple specific computational tasks (hereinafter referred to as subtasks), such as processing a specific region in a feature map. Each computation thread bundle in the thread bundle group is used to execute one of the multiple subtasks to jointly complete the computation task, thereby accelerating the execution of the computation task.

[0036] To ensure that multiple computational threads in a thread bundle can work together, the tensor computation engine 101 uses synchronization units to manage the dependencies between multiple computational threads in the thread bundle group, ensuring that multiple computational threads only continue to the next step after each thread has reached a certain point.

[0037] For example, the synchronization unit in the tensor computation engine 101 uses synchronization entries to record the synchronization status of computation instructions sent by multiple computation thread bundles in the same thread bundle group. Resources are released and execution continues only after all computation instructions of all associated computation thread bundles have been executed. Here, a synchronization entry is a data structure that coordinates the execution status of multiple computation thread bundles. It can be used to record the instruction completion status of each computation thread bundle to ensure data consistency and execution order between parallel tasks.

[0038] Optionally, each thread bundle is configured with an independent synchronization entry. The tensor computation engine 101 determines whether synchronization has ended by monitoring the status of the synchronization entries of all associated computation thread bundles. All associated computation thread bundles may also share the same synchronization entry. The tensor computation engine 101 determines whether synchronization has ended by monitoring the status of this synchronization entry. This embodiment of the application does not limit this approach.

[0039] In related technologies, each computation thread bundle has only one corresponding synchronization entry. This means that the memory processing element 102 can only continue to send computation instructions to the tensor computation engine 101 for that computation thread bundle after the synchronization entry corresponding to the computation thread bundle has been released. Therefore, at most one computation instruction is sent to the tensor computation engine 101 for each computation thread bundle, resulting in low bandwidth utilization between the memory processing element 102 and the tensor computation engine 101.

[0040] In this embodiment, each computation thread bundle is configured with multiple synchronization entries. This allows the memory processing element 102 to continuously send multiple computation instructions to the tensor computation engine 101 based on the usage status of the multiple synchronization entries. This improves the bandwidth utilization between the memory processing element 102 and the tensor computation engine 101. Furthermore, it allows the tensor computation engine 101 to complete the execution of the current computation instruction without waiting for the transmission and synchronization of the next computation instruction. This reduces the idle time of the tensor computation engine 101, thereby improving the utilization of the computational resources of the tensor computation engine 101, and consequently improving the bandwidth and computational resource utilization of the processor 100.

[0041] The technical solutions provided in the embodiments of this application will be described in detail below. For the contents not described in the embodiments of this application, please refer to the above embodiments, and they will not be repeated here.

[0042] For ease of explanation and illustration, this application uses one of the at least two memory processing elements as an example to illustrate the management process of multiple synchronization entries and computation instructions. The aforementioned memory processing element can refer to any one of the at least two memory processing elements.

[0043] S1, the memory processing element, is used to call the first of the n computation thread bundles running on the memory processing element to generate computation instructions. The computation instructions are used to instruct the tensor computation engine to perform computation tasks.

[0044] The first computation thread bundle can refer to any one of the n computation thread bundles. Each computation thread bundle can participate in different global computation tasks. A global computation task can be completed collaboratively by one computation thread bundle (i.e., a thread bundle group) in each memory processing element. For example, if a global computation task is divided into multiple local computation tasks, each computation thread bundle in the thread bundle group is responsible for handling one of the multiple local computation tasks. Each local computation task can be further broken down into multiple computation tasks, forming a computation task flow. The memory processing element can generate a computation instruction for each computation task to call the tensor computation engine to complete the execution of multiple computation tasks.

[0045] Optionally, during the execution of the corresponding local computation task by the first computation thread bundle, the memory processing element may also generate computation instructions for the first computation thread bundle when it is necessary to call the tensor computation engine. For example, the computation instructions may include at least one of the following: input data corresponding to the computation task, the storage address of the input data, the operational logic for operating the input data, and the storage address of the execution result of the computation instructions; this embodiment does not limit this. The above-mentioned computation instructions may also be generated by the memory processing element; this embodiment does not limit this as well.

[0046] For example, refer to Figure 2 When the tensor computation engine 101 is configured with four memory processing elements 102, if the processor 100 receives one global computation task, each memory processing element 102 can activate one computation thread bundle to handle one of the local computation tasks (e.g., divided into four local computation tasks) of the global computation task, thereby coordinating the processing of the global computation task. If the processor 100 receives two global computation tasks, each memory processing element 102 can activate two computation thread bundles to coordinating the processing of the two global computation tasks respectively. Each global computation task corresponds to one computation thread bundle in the four memory processing elements 102, and these four computation thread bundles form a thread bundle group. Each computation thread bundle in the same thread bundle group synchronously processes its respective local computation task.

[0047] Optionally, at the same stage, the tensor computation engine must wait for all computation instructions sent by each computation thread bundle within the same thread bundle group to arrive before it can begin executing those instructions. This is to ensure data consistency and execution order between local computation tasks. For example, at the same stage, for each computation thread bundle within the same thread bundle group, the information included in the computation instructions is the same except for the input data.

[0048] S2, the memory processing element, is used to send computation instructions to the tensor computation engine when there are unused synchronization entries among the multiple synchronization entries corresponding to the first computation thread bundle.

[0049] Optionally, each computation thread bundle can have at least two synchronization entries. Each synchronization entry grants the same computation thread bundle the permission to send a computation instruction to the tensor computation engine. For example, if a synchronization entry is in use, the memory processing element cannot send a computation instruction to the tensor computation engine for the computation thread bundle through that synchronization entry; if a synchronization entry is in unused, the memory processing element can send a computation instruction to the tensor computation engine for the computation thread bundle through that synchronization entry. The tensor computation engine mentioned above corresponds to at least two memory processing elements.

[0050] Synchronization entries can be used to record the synchronization status of computation instructions, such as whether the computation instruction has completed synchronization. For example, if a computation instruction has completed execution and its result has been written to the target location (such as a register of a memory processing element), and its associated computation instructions have also completed execution and their results have been written to the target location, it can be determined that the computation instruction has completed synchronization. Associated computation instructions can refer to computation instructions sent by associated computation thread bundles; that is, if all computation instructions corresponding to the same thread bundle group have completed execution and their results have been written to the target location, it can be determined that all computation instructions have completed synchronization. This application does not limit the method for determining synchronization completion.

[0051] When the computation instructions are synchronized, the corresponding synchronization entry is released and switches from the used state to the unused state. When the computation instructions are not synchronized, the corresponding synchronization entry is occupied, i.e., it is in the used state.

[0052] For example, refer to Figure 2 The memory processing element 102 runs a first computation thread bundle, and when it is determined that there is a synchronization entry in the first computation thread bundle that is in an unused state, it sends a computation instruction to the tensor computation engine 101.

[0053] Optionally, the memory processing element is further configured to cache the computation instruction and not send the computation instruction to the tensor computation engine if there are no unused synchronization entries among the multiple synchronization entries corresponding to the first computation thread bundle, until it is determined that there are unused synchronization entries in the first computation thread bundle.

[0054] For example, with a synchronization count of 4, the tensor computation engine can receive up to 4 computation instructions from each computation thread bundle simultaneously. This helps reduce the idle time of bandwidth between the memory processing element and the tensor computation engine, and effectively reduces the idle time of the tensor computation engine due to waiting for the transmission and synchronization of computation instructions.

[0055] In one example, a throttling counter is set for the first computation thread bundle. The throttling counter is used to record the number of synchronization entries in the used state among the multiple synchronization entries corresponding to the first computation thread bundle. The number of multiple synchronization entries is m.

[0056] The value of the throttling counter is the number of synchronization entries in the used state corresponding to the computation thread bundle, that is, the number of computation instructions that the computation thread bundle has sent to the tensor computation engine but have not yet been completed. For example, when the value of the throttling counter is 4, the number of synchronization entries in the used state corresponding to the computation thread bundle is 4. Optionally, the throttling counter can be implemented as hardware, such as a counter added to the memory processing element. The throttling counter can also be implemented as software, such as a data structure in the memory processing element used to record the number of synchronization entries in the used state; this embodiment of the application does not limit this.

[0057] m is an integer greater than 1, such as 2, 3, and 4. For example, each computation thread bundle corresponds to 4 synchronization entries.

[0058] For example, the memory processing element is also configured to: send a calculation instruction to the tensor calculation engine when the value of the throttling counter is less than m; and increment the value of the throttling counter.

[0059] If the value of the throttling counter is less than m, it can be determined that there is still an unused synchronization entry corresponding to the first computation thread bundle. For example, if m=4 and the value of the throttling counter is 3, it can be determined that there is still one unused synchronization entry corresponding to the first computation thread bundle. After sending the computation instruction, the memory processing element controls the value of the throttling counter to increment by 1.

[0060] Optionally, the memory processing element also includes a control unit, which is responsible for fetching and decoding upper-level instructions (such as instructions corresponding to local computation tasks) and issuing computation instructions. The aforementioned throttling counter can be located within the control unit. The control unit determines whether the first computation thread bundle can be selected to send computation instructions by detecting the value of the throttling counter, and increments the value of the throttling counter by 1 for each computation instruction sent.

[0061] This application embodiment sets up a throttling calculator in the memory processing element and uses the throttling calculator to count the synchronization entries in the used state. This enables the memory processing element to accurately control the number of calculation instructions sent based on the value of the throttling calculator, thereby improving the accuracy of calculation instruction management.

[0062] In one example, the memory processing element is also used to: not send computation instructions to the tensor computation engine when the value of the throttling counter is equal to m.

[0063] If the value of the throttling counter is equal to m, it can be determined that there are no unused synchronization entries in the first computation thread bundle, and the memory processing element does not have the authority to send computation instructions to the memory processing element.

[0064] For example, if the value of the throttling counter is 4 when m=4, it can be determined that there are no unused synchronization entries in the first computation thread bundle, and the memory processing element does not send computation instructions to the tensor computation engine until the value of the throttling counter is detected to be less than 4.

[0065] This application embodiment controls the memory processing element to send excessive calculation instructions to the tensor calculation engine discontinuously by using the value of the throttling calculator. This helps to reduce the storage pressure on the tensor calculation engine and reduce the probability of the execution process of calculation instructions becoming chaotic.

[0066] In one example, the memory processing element selects one computation thread from n computation thread bundles each clock cycle to send a computation instruction. When the computation instruction is a TCE instruction, the memory processing element checks whether the value of the throttling counter is less than m. Exemplarily, the memory processing element selects one computation thread from the n computation thread bundles sequentially; alternatively, the memory processing element may randomly select one computation thread bundle from the n computation thread bundles. This embodiment of the application does not limit the choice in this respect.

[0067] To avoid memory processing elements selecting computation thread bundles that do not have synchronization entries in an unused state, embodiments of this application also provide a first queue, such as n computation thread bundles forming the first queue.

[0068] For example, the memory processing element is further configured to remove the first computation thread bundle from the first queue when the value of the throttling counter is equal to m, to obtain an updated first queue; select a computation thread bundle from the updated first queue, and send computation instructions to the tensor computation engine.

[0069] For example, the memory processing element is used to select a computation thread bundle from the updated first queue in sequence. Alternatively, the memory processing element may randomly select a computation thread bundle from the updated first queue. This application embodiment does not limit this.

[0070] This application embodiment selects a computation thread bundle from those computation thread bundles that still have unused synchronization entries to send computation instructions, thereby avoiding the memory processing element from selecting a computation thread bundle that does not have unused synchronization entries. This avoids the latency introduced by the comparison throttling counter, which is beneficial to improving the management efficiency of computation instructions.

[0071] Optionally, the memory processing element is also configured to: add the first computation thread bundle to the updated first queue when the value of the throttling counter is restored to less than m, thereby obtaining the restored first queue; select a computation thread bundle from the restored first queue and send computation instructions to the tensor computation engine.

[0072] For example, the memory processing element is used to select a computation thread bundle from the restored first queue in sequence. Alternatively, the memory processing element may randomly select a computation thread bundle from the restored first queue. This application embodiment does not limit this.

[0073] When the memory processing element determines that the synchronization entry corresponding to the first computation thread bundle has been released, it decreases the value of the throttling counter and adds the first computation thread bundle to the updated first queue.

[0074] This application embodiment restores the computation thread bundle with unused synchronization entries and adds it back to the first queue, so that the computation thread bundle can be selected to send computation instructions, thereby ensuring the continuous transmission of computation instructions.

[0075] S3, the tensor computation engine, is used to record the synchronization status of computation instructions through a first synchronization entry, which is any unused synchronization entry among multiple synchronization entries.

[0076] The tensor computation engine is the tensor computation engine corresponding to the memory processing element group to which the aforementioned memory processing element belongs. When at least one synchronization entry is unused among multiple synchronization entries, the first synchronization entry is the synchronization entry selected by the tensor computation engine from the at least one unused synchronization entry. For example, the tensor computation engine may select one synchronization entry as the first synchronization entry from unused synchronization entries (i.e., synchronization entries in an unused state) according to the order in which the computation instructions are sent. The tensor computation engine may also randomly select one unused synchronization entry as the first synchronization entry. This application embodiment does not limit the method for selecting the first synchronization entry. The tensor computation engine selects a synchronization entry through a synchronization unit to record computation instructions.

[0077] Optionally, an identifier can be used to indicate the state of a synchronization entry. For example, if the identifier is 1, the synchronization entry is in a used state; if the identifier is 0, the synchronization entry is in an unused state.

[0078] In one example, the above calculation instructions may further include: identification information of the memory processing element, identification information of the first thread bundle group, and identification information of the first calculation thread bundle. The first thread bundle group is the thread bundle group to which the first calculation thread bundle belongs. The identification information of the memory processing element may include at least one of the following: identification information of the memory processing element itself, and identification information of the first storage unit in the memory processing element. The first storage unit may be a register in the memory processing element.

[0079] The tensor computation engine can use a first synchronization entry to record the identification information of the first thread bundle group and the first computation thread bundle, as well as the synchronization status of the computation instructions. The synchronization status of the computation instructions can also be indicated by identifiers, such as 1 indicating that the computation instruction has completed synchronization and 0 indicating that the computation instruction has not completed synchronization.

[0080] Optionally, the tensor computation engine is used to wait for the associated instructions of the computation instruction after receiving the computation instruction, and to prepare to read the source operands of each computation instruction after all the computation instructions corresponding to the first thread bundle have arrived.

[0081] For example, the tensor computation engine determines whether all computation instructions corresponding to the first thread bundle have arrived based on the synchronization entries that are in the used state. The memory processing elements send computation instructions sequentially to the computation thread bundles. The tensor computation engine, based on the synchronization entries that are in the used state, can determine all computation instructions corresponding to the first thread bundle. For each computation thread bundle in the first thread bundle, associated computation instructions are determined according to the sending time of the computation instructions. For example, for each computation thread bundle, at least one computation instruction corresponding to the computation thread bundle is sorted according to the order of sending time to obtain the sequence number of the computation instructions. The computation instructions with the same sequence number in each computation thread bundle are determined as associated computation instructions. If the number of associated computation instructions reaches the number of memory processing elements, it can be determined that all associated computation instructions have arrived.

[0082] This application uses the first computation thread bundle as an example for illustration. The synchronization method for the other computation thread bundles among the n computation thread bundles is the same as that for the first computation thread bundle. For example, when the number of synchronization entries is 4, the tensor computation engine needs to process a maximum of 4 sets of associated computation instructions simultaneously for the same computation thread bundle. For the same memory processing element, the tensor computation engine needs to process the computation instructions corresponding to the n computation thread bundles simultaneously, which greatly improves the resource utilization of the tensor computation engine.

[0083] For example, refer to Figure 2Taking one computation thread bundle running on each memory processing element 102 as an example, when the TCE instructions corresponding to the four computation thread bundles in the same thread bundle group have all arrived at the tensor computation engine 101, the tensor computation engine 101 prepares to read the source operands of the four TCE instructions, such as extracting the storage address of the input data from the TCE instructions, in order to prepare to read the input data (i.e., the source operands) from the corresponding memory processing element 102 or shared memory unit 103. For the same computation thread bundle, if there is no data dependency between the current TCE instruction and the previous TCE instruction, the tensor computation engine 101 directly reads the input data and performs computation directly after the previous TCE instruction has been executed. If there is a data dependency between the current TCE instruction and the previous TCE instruction, the tensor computation engine 101 reads the input data for computation after the previous TCE instruction has been executed. The specific execution process of the computation instructions will be explained below, and will not be repeated here.

[0084] Tensor calculation engine 101 determines that the four TCE instructions have been synchronized after all four TCE instructions have been executed and the execution results have been stored in their respective memory processing elements 102.

[0085] In one example, the tensor computation engine is also used to mark the first synchronization entry as used after recording the synchronization state of the computation instruction through the first synchronization entry, and to mark the first synchronization entry as unused when the computation instruction and the computation instruction corresponding to the related computation thread bundle have completed synchronization.

[0086] The aforementioned related computation thread bundles are computation thread bundles related to the first computation thread bundle among other memory processing elements. These other memory processing elements are memory processing elements other than those mentioned above, among at least two memory processing elements. For example, computation thread bundles belonging to the same thread bundle group as the first computation thread bundle are all related computation thread bundles of the first computation thread bundle.

[0087] Once the computation instruction and all related computation instructions have been synchronized, the tensor computation engine releases the first synchronization entry and notifies at least two memory processing elements that the first synchronization entry has been released. The memory processing elements, upon determining that the first synchronization entry has been re-marked as unused, decrease the value of the throttling counter, such as decrementing the stream counter by 1. Optionally, the memory processing elements can also actively determine whether a synchronization entry has been released, for example, by periodically querying the tensor computation engine for the status of the synchronization entry to determine whether it has been re-marked as unused.

[0088] This embodiment of the application, by releasing the synchronization entry after the calculation instructions have been synchronized, ensures that the calculation instructions are executed and sent correctly, which is beneficial to improving the stability of the processor. Furthermore, after any synchronization entry corresponding to the first calculation thread bundle is released, the memory processing element can continue to send calculation instructions for the first calculation thread bundle, which is beneficial to improving the processor's bandwidth utilization.

[0089] In summary, the technical solution provided in this application, by setting multiple synchronization entries for each computation thread bundle, allows the memory processing element to continuously send multiple computation instructions to the tensor computation engine based on the usage status of these synchronization entries. This improves the bandwidth utilization between the memory processing element and the tensor computation engine. Furthermore, it allows the tensor computation engine to complete the execution of the current computation instruction without waiting for the transmission and synchronization of the next computation instruction, reducing the idle time of the tensor computation engine and thus improving its computational resource utilization, thereby increasing the processor's bandwidth and computational resource utilization.

[0090] The following section will describe the process of the tensor computation engine processing computation instructions. For content not described in the embodiments of this application, please refer to the above embodiments; further details will not be repeated here. For ease of explanation and illustration, the embodiments of this application will use the aforementioned memory processing element as an example.

[0091] In one example, the memory processing element is also used to send multiple computation instructions to the tensor computation engine for the first computation thread bundle.

[0092] Optionally, the memory processing element sequentially sends multiple computation instructions to the tensor computation engine. In this embodiment, the multiple computation instructions can be all computation instructions corresponding to the entire execution of the first computation thread bundle, or they can be a portion of the computation instructions corresponding to the entire execution of the first computation thread bundle, such as at least two consecutive computation instructions. This embodiment does not limit this. It should be noted that the number of computation instructions that have reached the tensor computation engine but have not been executed is less than or equal to m.

[0093] Optionally, the tensor computation engine is also used to read the source operand of the second computation instruction during the execution of the computation task of the first computation instruction, when there is no data dependency between the first and second computation instructions in a series of consecutive first and second computation instructions.

[0094] The first and second computation instructions can be any two consecutive computation instructions from a plurality of computation instructions. There is no data dependency between the first and second computation instructions. A data dependency refers to a dependency in the execution order of multiple computation instructions. For example, in RAW (Read after Write) type: the result of the preceding computation instruction needs to be used as the input of the next computation instruction; in WAR (Write after Read) type: the input of the preceding computation instruction is the same as the output of the subsequent computation instruction; in WAW (Write after Write) type: the output of the preceding computation instruction is the same as the output of the subsequent computation instruction. This application does not limit this.

[0095] For example, in a global matrix multiplication task, the matrices involved in the multiplication are typically very large. The memory processing element then splits the matrix into multiple sub-matrices. Therefore, each associated computation thread bundle needs to send multiple computation instructions. The execution result of the preceding computation instruction serves as the input data for the next computation instruction. Consequently, multiple computation instructions need to establish data dependencies. Optionally, if the second computation instruction does not require the execution result of the first computation instruction as input data, then no data dependency needs to be established between the second and first computation instructions.

[0096] The process of executing the computation task of the first computation instruction is as follows: after reading the source operands of the first computation instruction, the process of processing the source operands, such as performing logical operations on tensor data. For example, the tensor computation engine can execute multiple computation instructions corresponding to the first computation thread bundle through the ALU corresponding to the memory processing element.

[0097] During the execution of the first computation instruction, the tensor computation engine reads the source operands of the second computation instruction when it detects that the second computation instruction and its associated computation instructions have been completed. After the first computation instruction is executed, the tensor computation engine directly executes the computation task of the second computation instruction. That is, while the execution result of the first computation instruction is being written back to the memory processing element, the tensor computation engine executes the computation task of the second computation instruction in parallel.

[0098] For example, refer to Figure 7 Taking the first computation instruction 701 and the second computation instruction 702 sent by MPE0 to the first computation thread bundle as an example, during the execution of the computation task of the first computation instruction 701, if the second computation instruction 702 and its associated computation instructions both reach the tensor computation engine, the source operands of the second computation instruction 702 can be read in parallel. During the process of writing the execution result of the first computation instruction 701 back to MPE0, the tensor computation engine can execute the computation task of the second computation instruction 702 in parallel.

[0099] Because there are no data dependencies between the computation instructions of different computation thread bundles, the tensor computation engine can execute the computation instructions corresponding to each computation thread bundle in the same thread bundle group in parallel. For example, for the same computation thread bundle, the tensor computation engine can process m computation instructions sent consecutively by that computation thread bundle in parallel, which greatly improves the computational efficiency of the tensor computation engine.

[0100] In this embodiment, for multiple computation instructions without data dependencies, the execution process of the preceding computation instructions and the reading of the source operands of the subsequent computation instructions are processed in parallel, as are the writing back process of the execution results of the preceding computation instructions and the execution process of the subsequent computation instructions. This is beneficial to improving the execution efficiency of computation instructions, thereby improving the computation efficiency of the tensor computation engine.

[0101] Optionally, if there are data dependencies between multiple computation instructions corresponding to the first computation thread bundle, the tensor computation engine needs to ensure that the multiple computation instructions are executed in the correct order in order to guarantee the correct execution result.

[0102] In this embodiment, each computation instruction is equipped with a first scoreboard and a second scoreboard. A scoreboard is a hardware mechanism for dynamically tracking data dependencies between computation instructions. It manages the execution order of computation instructions by recording their execution status (e.g., sent, executed, result ready). For example, the first scoreboard indicates whether the execution result of the computation instruction has been written back, and the second scoreboard indicates whether the source operands of the computation instruction are ready. Based on the first and second scoreboards, it can be ensured that multiple computation instructions are executed in the correct order.

[0103] For example, the memory processing element is also configured to, when there is a data dependency between the first calculation instruction and the second calculation instruction, update the first scoreboard corresponding to the first calculation instruction after sending the first calculation instruction to the tensor calculation engine, and obtain the updated first scoreboard.

[0104] Optionally, the memory processing element increments the value of the first scoreboard to obtain an updated first scoreboard. For example, the updated first scoreboard is 1 to indicate that the first calculation instruction has been sent, but the execution result of the first calculation instruction has not been written back. The initial value of the first scoreboard can be 0 to indicate that the execution result of the first calculation instruction has been written back. Since the second calculation instruction is sent after the first calculation instruction, it can be determined whether the execution result of the first calculation instruction has been written back based on the first scoreboard of the first calculation instruction.

[0105] Optionally, the memory processing element may simultaneously increment the value of the second scoreboard of the first calculation instruction to obtain an updated second scoreboard, indicating that the first calculation instruction has been sent but the source operand has not been acquired.

[0106] The memory processing element is also used to restore the first scoreboard of the first calculation instruction and obtain the restored first scoreboard after the first calculation instruction and the associated calculation instruction of the first calculation instruction have been synchronized.

[0107] That is, when the execution result of the first calculation instruction is completely written back to the memory processing element, the memory processing element restores the updated first scoreboard of the first calculation instruction to the initial value, and obtains the restored first scoreboard to indicate that the execution result of the first calculation instruction has been written back.

[0108] For example, the data dependency between two computation instructions can be represented as follows: inst0 dst0, src0, wr=sb1, rd=sb0; inst1 dst1, dst0, wr=sb2, rd=sb1; In this context, `inst0` represents computation instruction 0, `dst0` represents the storage location of the execution result of computation instruction 0, `src0` represents the storage location of the source operands of computation instruction 0, `sb1` is the first scoreboard (`wr` scoreboard) of computation instruction 0 (if its value is 0), and `sb0` is the second scoreboard (`rd` scoreboard) of computation instruction 0 (if its value is 1). Similarly, `inst1` represents computation instruction 1, `dst1` represents the storage location of the execution result of computation instruction 1, `dst0` represents the storage location of the source operands of computation instruction 1, and `sb2` is the first scoreboard of computation instruction 1 (if its value is 0).

[0109] Calculation instruction 1 needs to read the execution result of calculation instruction 0 from dst0 for calculation. Therefore, the second scoreboard (rd scoreboard) of calculation instruction 1 is recorded with the same value as the first scoreboard of calculation instruction 0, namely sb1. When calculation instruction 0 is sent, the memory processing element controls sb1 to increment by 1. When the tensor calculation engine is preparing to read the source operand of calculation instruction 1, calculation instruction 1 is blocked because sb1 is not 0. It continues until calculation instruction 0 is executed and its execution result is stored in dst0. Then, the memory processing element controls sb1 to decrement by 1 to 0, and the tensor calculation engine begins to read the source operand of calculation instruction 1 to ensure that the source operand of calculation instruction 1 is correctly obtained.

[0110] In other words, when the tensor computation engine detects a second computation instruction and all its associated computation instructions have been reached, it does not immediately read the source operands of the second computation instruction. Instead, it obtains the source operands corresponding to the second computation instruction only if it detects that the second scoreboard of the second computation instruction matches the restored first scoreboard. If it detects that the second scoreboard of the second computation instruction is the same as the restored first scoreboard, it obtains the source operands corresponding to the second computation instruction, thereby ensuring that the first and second computation instructions are executed in the correct order.

[0111] For example, refer to Figure 8 Once the execution result of the first calculation instruction 801 has been completely written back to the memory processing element, the tensor calculation engine begins to read the source operands of the second calculation instruction 802 and execute the calculation task of the second calculation instruction 802.

[0112] In one feasible example, the interval between the start time of the first calculation instruction and the start time of the second calculation instruction can be set to be longer than the execution time of the first calculation instruction, which can also ensure that the first and second calculation instructions are executed in the correct order. For example, when the execution time of the calculation is substantially fixed, a certain margin can be set for the execution time to obtain the interval duration, and this application embodiment does not limit this.

[0113] In one example, the tensor computation engine is also used to send a first message to the memory processing element, the first message instructing the memory processing element to restore the second scoreboard of the second computation instruction.

[0114] Optionally, the memory processing element decrements the value of the second scoreboard to obtain a restored second scoreboard. For example, the restored second scoreboard is 0 to indicate that the source operand of the second calculation instruction is ready. The initial value of the second scoreboard can be 1 to indicate that the source operand of the second calculation instruction is not ready. When the second calculation instruction is sent, its second scoreboard is incremented by the memory processing element.

[0115] The scoreboards associated with the second calculation instruction are updated synchronously with the scoreboard of the second calculation instruction. The update method for the scoreboard of the associated calculation instruction is the same as that for the scoreboard of the second calculation instruction, and will not be repeated here.

[0116] After obtaining the source operands of the second computation instruction, the tensor computation engine executes the computation task of the second computation instruction.

[0117] In embodiments of this application, when there are data dependencies between multiple computation instructions, the first scoreboard of the computation instructions is updated to ensure that multiple computation instructions are executed in the correct order, which helps to improve the computational security of the tensor computation engine.

[0118] In one example, taking the first computation instruction as an example, the tensor computation engine is also used to implement at least one of the following steps.

[0119] 1. When the source operand of the first calculation instruction is obtained, the first calculation instruction is decomposed into multiple micro-operation instructions, and the source operand of the first calculation instruction is decomposed into multiple sub-data, with each micro-operation instruction corresponding to one of the multiple sub-data.

[0120] Micro-operation instructions are the aforementioned underlying micro-operation signals, which can be understood and executed by the ALU. The tensor computation engine can generate a micro-operation instruction for each sub-data to instruct the ALU to process that sub-data and obtain the execution result of that sub-data.

[0121] For example, the TCE instruction supports different MNK sizes, and the tensor computation engine can decompose different MNKs into the smallest computational units (i.e., sub-data), such as 32×32×16. Here, MNK can be represented as a matrix multiplication operation between an M×k matrix and a k×N matrix.

[0122] For example, refer to Figure 9 When the size of the TCE instruction is 64×64×16, the TCE instruction is used to instruct a matrix multiplication operation on a 64×16 matrix 901 and a 16×64 matrix 902. The TCE instruction can be broken down into four micro-operation instructions, each corresponding to a 32×16 matrix and a 16×32 matrix. For example, this micro-operation instruction is used to instruct a matrix multiplication operation on a 32×16 matrix and a 16×32 matrix.

[0123] 2. For the first micro-operation instruction among multiple micro-operation instructions, determine the tag information of the first micro-operation instruction. The tag information of the first micro-operation instruction includes: the identification information of the first storage unit and the identification information of the first thread bundle group. The first storage unit is used to store the execution result of the first micro-operation instruction.

[0124] The first micro-operation instruction can be any one of multiple micro-operation instructions. The label information of the first micro-operation instruction is used to identify the first micro-operation instruction.

[0125] The first computation instruction belongs to the first instruction group. The first instruction group includes computation instructions related to the first computation instruction among the various computation instructions corresponding to the first thread bundle group. The first thread bundle group is the thread bundle group to which the first computation thread bundle belongs. The first thread bundle group includes computation thread bundles related to the first computation thread bundle among at least two memory processing elements.

[0126] Optionally, the first storage unit may be a register in the memory processing element, and the identification information of the first storage unit may be implemented as the identification information of the register to which the execution result of the first micro-operation instruction is stored.

[0127] For example, refer to Figure 10 Taking register file 1001 in MPE0 as an example, given that the size of the TCE instruction sent by MPE0 to Wave0 is determined, the tensor calculation engine can calculate the distribution of the execution result of the TCE instruction in register file 1001. Thus, it can determine the register (i.e. the first storage unit) corresponding to each micro-operation instruction in register file 1001, and determine the identification information of the register as the identification information of the first storage unit, such as Ra, Ra+1, etc.

[0128] 3. If a write-back entry containing tag information of the first micro-operation instruction is detected in the write-back cache of the tensor computation engine, the allocation of new write-back entries to the first micro-operation instruction is stopped, and the new write-back entries of the first micro-operation instruction are used to cache the execution result of the first micro-operation instruction.

[0129] The write-back cache is used to cache the latest execution results of the tensor computation engine. That is, the execution results are first stored in the write-back cache instead of being directly written back to the memory processing element, and the cached execution results are deleted only after the execution results have been completely written back to the memory processing element. Write-back entries are used to record execution results and can be implemented as cache lines or data blocks, etc.

[0130] Before executing the first micro-operation instruction, the tensor computation engine allocates a write-back entry for it to store the execution result. The engine only executes the first micro-operation instruction after a write-back entry has been allocated; otherwise, it blocks the instruction.

[0131] In this embodiment of the application, the write-back entry includes the tag information of the micro-operation instruction, namely the identification information of the storage unit and the identification information of the thread bundle group.

[0132] For example, refer to Figure 11 The write-back cache device 1101 includes multiple write-back entries, which include allocated and unallocated states. In the allocated state, the write-back entry records a Squad id and a reg id, where the Squad id is the identifier of the thread bundle group and the reg id is the identifier of the register. In the unallocated state, the write-back entry is empty.

[0133] When the tensor computation engine allocates a write-back entry for the first micro-operation instruction, it first checks whether there is a target synchronization entry in the write-back cache device 1101. The target synchronization entry is a synchronization entry that records the tag information of the first micro-operation instruction. If the synchronization entry includes both the Squad id and reg id of the first micro-operation instruction, the synchronization entry is determined as the target synchronization entry.

[0134] If a target write-back entry is detected in the write-back buffer device 1101, the allocation of a new write-back entry to the first micro-operation instruction is stopped. That is, if the execution results of other micro-operation instructions need to be stored in the register corresponding to the first micro-operation instruction, it is necessary to wait for the other micro-operation instructions to be executed and for the execution results of the other micro-operation instructions to be completely stored in the corresponding register.

[0135] 4. If it is detected that there is no write-back entry containing the tag information of the first micro-operation instruction in the write-back buffer, a new write-back entry is allocated to the first micro-operation instruction, and the tag information of the first micro-operation instruction is recorded through the new write-back entry of the first micro-operation instruction.

[0136] In other words, there are no other micro-operation instructions whose execution results need to be stored in the register corresponding to the first micro-operation instruction. The tensor calculation engine can directly allocate a new write-back entry to the first micro-operation instruction, such as any write-back entry in an unallocated state.

[0137] This application embodiment splits computation instructions into multiple micro-operation instructions and allocates write-back entries to micro-operation instructions based on the tag information of the micro-operation instructions, so that micro-operation instructions without storage conflicts can be executed in parallel, which is beneficial to improving the processing efficiency of computation instructions.

[0138] In one example, the tensor computation engine is also used for: 5. When a new write-back entry is assigned to the first micro-operation instruction, the sub-data corresponding to the first micro-operation instruction is processed according to the first micro-operation instruction to obtain the execution result of the first micro-operation instruction.

[0139] like Figure 11 As shown, when a new write-back entry is assigned to the first micro-operation instruction, the tensor computation engine sends the first micro-operation instruction and the corresponding sub-data to the corresponding ALU 1102. The ALU 1102 executes the first micro-operation instruction to process the sub-data corresponding to the first micro-operation instruction and obtains the execution result of the first micro-operation instruction.

[0140] 6. Store the execution result of the first micro-operation instruction into the newly written-back entry.

[0141] The tensor computation engine caches the execution result of the first micro-operation instruction into the newly assigned write-back entry.

[0142] For each micro-operation instruction, the tensor computation engine caches the execution result of the micro-operation instruction into the newly assigned write-back entry.

[0143] 7. Send the execution result of the first micro-operation instruction to the memory processing element so as to store it in the first storage unit in the memory processing element.

[0144] After the execution result of the first micro-operation instruction is cached in the new write-back entry, the tensor computation engine sends the execution result of the first micro-operation instruction to the memory processing element to store it in the corresponding register.

[0145] For each micro-operation instruction, the tensor computation engine stores the execution result of the micro-operation instruction into the corresponding register.

[0146] 8. Release the newly written-back entry after the execution result of the first micro-operation instruction has been completely stored in the first memory unit.

[0147] Once the execution result of the first micro-operation instruction is completely stored in the first storage unit, the tensor computation engine clears the data in the newly written-back entry, releases the newly written-back entry, and switches the newly written-back entry to an unallocated state.

[0148] In this process, the first calculation instruction is executed after multiple micro-operation instructions have been completed. The execution results of the multiple micro-operation instructions form the execution result of the first calculation instruction.

[0149] In the case of data dependencies between computation instructions, this application embodiment splits computation instructions into multiple micro-operation instructions and assigns write-back entries to micro-operation instructions based on their tag information. This allows micro-operation instructions without storage conflicts to be executed in parallel, which helps improve the processing efficiency of computation instructions.

[0150] In addition, by writing back to the cache to cache the execution results of micro-operation instructions, each memory processing element can obtain the correct data, which helps to improve the security of the processor.

[0151] In some embodiments, the technical solutions provided in the embodiments of this application are described using four memory processing elements and one computing thread bundle running on each memory processing element as an example.

[0152] 1. The task scheduling unit in the processor sends four consecutive waves (i.e., computation thread bundles) to four MPEs (i.e., memory processing elements).

[0153] These 4 MPEs form a squad, and each wave corresponds to the same squad ID.

[0154] 2. For any of the four MPEs, the MPE sends a calculation instruction for the assigned wave every clock cycle.

[0155] Optionally, when the computation is executed as a TCE instruction, the MPE checks the value of the throttling counter assigned to the wave. If the value of the throttling counter is less than 4, the MPE sends a TCE instruction to the TCE (Tensor Calculation Engine) and increments the value of the throttling counter by 1; if the value of the throttling counter is greater than or equal to 4, the MPE does not send a TCE instruction to the TCE.

[0156] The first and second scoreboards of the MPE incrementing TCE instruction.

[0157] Optionally, each wave is configured with 4 synchronization entries so that each wave can continuously send multiple TCE commands to the TCE, such as up to 4 TCE commands to the TCE.

[0158] 3. When TCE receives a TCE instruction from MPE, it selects an unused synchronization entry to record the synchronization status, square id, and wave id of the TCE instruction.

[0159] 4. When TCE detects that all four associated TCE instructions have arrived, it prepares to read the source operands of the four associated TCE instructions.

[0160] For example, the first TCE command sent by each of the four waves forms an associated TCE command group, the second TCE command sent by each of the four waves forms an associated TCE command group, and so on.

[0161] Optionally, for any of the four associated TCE instructions, if there is no data dependency between the TCE instruction and the preceding TCE instruction, the source operand corresponding to the TCE instruction can be directly read from the corresponding MPE.

[0162] When there is a data dependency between a TCE instruction and its preceding TCE instruction, the TCE waits for the preceding TCE instruction to be executed and for the execution result to be completely stored in the corresponding MPE before reading the source operand of the TCE instruction and executing the TCE instruction.

[0163] For any of the four associated TCE instructions, after the source operand is read, the corresponding MPE is decremented by the second scoreboard.

[0164] 5. When there is no data dependency between the TCE instruction and the preceding TCE instruction, the TCE reads the source operand of the preceding TCE instruction during the execution of the preceding TCE instruction, and executes the TCE instruction during the write-back of the execution result of the preceding TCE instruction.

[0165] 6. When there is a data dependency between a TCE instruction and its preceding TCE instruction, the TCE instruction is split into multiple micro-operation instructions, and the source operation data of the TCE instruction is correspondingly split into multiple sub-data.

[0166] 7. For any micro-operation instruction among multiple micro-operation instructions, TCE checks the write-back entry in the write-back buffer. If a write-back entry is available that includes the squad id and reg id of the micro-operation instruction, TCE does not allocate a write-back entry to the micro-operation instruction and postpones the execution of the micro-operation instruction.

[0167] 8. In the absence of a squad id and reg id for the micro-operation instruction, the TCE assigns a new write-back entry to the micro-operation instruction and records the squad id and reg id of the micro-operation instruction using the new write-back entry.

[0168] 9. The TCE will assign the micro-operation instructions with newly written back entries to the ALU. The ALU will then process the corresponding sub-data according to the micro-operation instructions to obtain the execution result of the micro-operation instructions.

[0169] 10. TCE stores the execution result of the micro-operation command into the newly assigned write-back entry, and releases the write-back entry when the execution result of the micro-operation command has been completely stored in the corresponding MPE.

[0170] 11. After the execution results of the four associated TCE instructions are completely stored in the corresponding MPE, the TCE releases the synchronization entries corresponding to the four associated TCE instructions.

[0171] 12. Each of the four MPEs decrements its respective throttling counter by 1.

[0172] In summary, the technical solution provided in this application, by setting multiple synchronization entries for each computation thread bundle, allows the memory processing element to continuously send multiple computation instructions to the tensor computation engine based on the usage status of these synchronization entries. This improves the bandwidth utilization between the memory processing element and the tensor computation engine. Furthermore, it allows the tensor computation engine to complete the execution of the current computation instruction without waiting for the transmission and synchronization of the next computation instruction, reducing the idle time of the tensor computation engine and thus improving its computational resource utilization, thereby increasing the processor's bandwidth and computational resource utilization.

[0173] Furthermore, when there are data dependencies between computation instructions, by splitting the computation instructions into multiple micro-operation instructions and allocating write-back entries to the micro-operation instructions based on their tag information, micro-operation instructions that do not have memory conflicts can be executed in parallel, which helps to improve the processing efficiency of computation instructions.

[0174] Furthermore, in the absence of data dependencies between computation instructions, parallel processing of the execution of preceding computation instructions and the reading of source operands for subsequent computation instructions, as well as parallel processing of the writing back of the execution results of preceding computation instructions and the execution of subsequent computation instructions, can improve the execution efficiency of computation instructions, thereby improving the computational efficiency of the tensor computation engine.

[0175] The following are embodiments of the method of this application. For details not disclosed in the embodiments of the method of this application, please refer to the embodiments above.

[0176] Please refer to Figure 12 This illustrates a flowchart of an instruction processing method applied to a processor, provided in one possible implementation of this application. The components of the processor can be found in the description of the embodiments above. Figure 12 As shown, the method may include the following steps: Step 1201: The memory processing element generates a computation instruction for the first computation thread bundle among the n computation thread bundles running on the memory processing element. The computation instruction is used to instruct the tensor computation engine to perform the computation task.

[0177] Step 1202: If there are unused synchronization entries among the multiple synchronization entries corresponding to the first computation thread bundle, the memory processing element sends a computation instruction to the tensor computation engine.

[0178] In one example, the first computation thread bundle is equipped with a throttling counter, which is used to record the number of synchronization entries in the used state among the plurality of synchronization entries. The number of the plurality of synchronization entries is m, where m is an integer greater than 1. When the value of the throttling counter is less than m, the memory processing element sends the calculation instruction to the tensor calculation engine and increases the value of the throttling counter.

[0179] In one example, the memory processing element does not send the computation instruction to the tensor computation engine when the value of the throttling counter is equal to m.

[0180] In one example, after the tensor computation engine records the synchronization status of the computation instruction through the first synchronization entry, it marks the first synchronization entry as the used state. Furthermore, when the computation instruction and the computation instruction corresponding to the relevant computation thread bundle complete synchronization, the engine marks the first synchronization entry as the unused state. The relevant computation thread bundle is a computation thread bundle associated with the first computation thread bundle among other memory processing elements. The other memory processing elements are memory processing elements other than the memory processing element itself among the at least two memory processing elements. The memory processing element is used to decrease the value of the throttling counter when it is determined that the first synchronization entry has been re-marked as the unused state.

[0181] In one example, when n is greater than 1, the n computation thread bundles form a first queue; when the value of the throttling counter is equal to m, the memory processing element removes the first computation thread bundle from the first queue to obtain an updated first queue; and selects a computation thread bundle from the updated first queue to send the computation instruction to the tensor computation engine.

[0182] In one example, when the value of the throttling counter recovers to less than m, the memory processing element adds the first computation thread bundle to the updated first queue to obtain the recovered first queue; selects a computation thread bundle from the recovered first queue and sends the computation instruction to the tensor computation engine.

[0183] Step 1203: The tensor computation engine records the synchronization status of the computation instruction through the first synchronization entry, which is any unused synchronization entry among multiple synchronization entries.

[0184] In one example, the memory processing element sequentially sends multiple computation instructions to the tensor computation engine for the first computation thread bundle; For consecutive first and second computation instructions among the plurality of computation instructions, if there is no data dependency between the first and second computation instructions, the tensor computation engine reads the source operand of the second computation instruction during the execution of the computation task of the first computation instruction.

[0185] In one example, when there is a data dependency between the first calculation instruction and the second calculation instruction, the memory processing element updates the first scoreboard corresponding to the first calculation instruction after sending the first calculation instruction to the tensor calculation engine, and obtains the updated first scoreboard. The first scoreboard is used to indicate whether the execution result of the calculation instruction has been written back.

[0186] In one example, the memory processing element restores the first scoreboard of the first calculation instruction after the first calculation instruction and the associated calculation instruction of the first calculation instruction have been synchronized, thus obtaining the restored first scoreboard; When the tensor computation engine detects that the second scoreboard of the second computation instruction matches the restored first scoreboard, it obtains the source operands corresponding to the second computation instruction. The second scoreboard is used to indicate whether the source operands of the computation instruction are ready. The tensor computation engine sends a first message to the memory processing element, the first message being used to instruct the memory processing element to restore the second scoreboard of the second computation instruction.

[0187] In one example, the first calculation instruction belongs to a first instruction group, the first instruction group includes calculation instructions related to the first calculation instruction among the various calculation instructions corresponding to the first thread bundle group, the first thread bundle group is the thread bundle group to which the first calculation thread bundle belongs, and the first thread bundle group includes calculation thread bundles related to the first calculation thread bundle among the at least two memory processing elements. When the tensor computation engine obtains the source operand of the first computation instruction, it decomposes the first computation instruction into multiple micro-operation instructions and decomposes the source operand of the first computation instruction into multiple sub-data, with each micro-operation instruction corresponding to one sub-data among the multiple sub-data. For the first micro-operation instruction among the plurality of micro-operation instructions, the tensor computation engine determines the tag information of the first micro-operation instruction. The tag information of the first micro-operation instruction includes: the identification information of the first storage unit and the identification information of the first thread bundle group. The first storage unit is used to store the execution result of the first micro-operation instruction. When the tensor computation engine detects that there is a write-back entry containing the tag information of the first micro-operation instruction in the write-back cache device of the tensor computation engine, it stops allocating new write-back entries to the first micro-operation instruction. The new write-back entries of the first micro-operation instruction are used to cache the execution result of the first micro-operation instruction. When the tensor computation engine detects that there is no write-back entry containing the tag information of the first micro-operation instruction in the write-back cache device, it allocates the new write-back entry to the first micro-operation instruction and records the tag information of the first micro-operation instruction through the new write-back entry of the first micro-operation instruction.

[0188] In one example, when the first micro-operation instruction is assigned the new write-back entry, the tensor computation engine processes the sub-data corresponding to the first micro-operation instruction according to the first micro-operation instruction to obtain the execution result of the first micro-operation instruction. The tensor computation engine stores the execution result of the first micro-operation instruction into the newly written-back entry; The tensor computation engine sends the execution result of the first micro-operation instruction to the memory processing element, so as to store it in the first storage unit in the memory processing element; The tensor computation engine releases the newly written-back entry when the execution result of the first micro-operation instruction is completely stored in the first storage unit; wherein, the first computation instruction is executed when the plurality of micro-operation instructions are executed.

[0189] In summary, the technical solution provided in this application, by setting multiple synchronization entries for each computation thread bundle, allows the memory processing element to continuously send multiple computation instructions to the tensor computation engine based on the usage status of these synchronization entries. This improves the bandwidth utilization between the memory processing element and the tensor computation engine. Furthermore, it allows the tensor computation engine to complete the execution of the current computation instruction without waiting for the transmission and synchronization of the next computation instruction, reducing the idle time of the tensor computation engine and thus improving its computational resource utilization, thereby increasing the processor's bandwidth and computational resource utilization.

[0190] Please refer to Figure 13 This diagram illustrates a simplified structural block diagram of a computer device provided in one embodiment of this application. The computer device 1300 can be any electronic device capable of data calculation, processing, and storage.

[0191] The computer device 1300 includes a processor 1301 and a memory 1302. The processor 1301 may be implemented as the processor described in the above embodiments.

[0192] Processor 1301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1301 may also include an AI processor for handling computational operations related to machine learning.

[0193] Memory 1302 may include one or more computer-readable storage media, which may be non-transitory. Memory 1302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 1302 is used to store a computer program configured to be executed by one or more processors.

[0194] Those skilled in the art will understand that Figure 13 The structure shown does not constitute a limitation on the computer device 1300, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0195] In some embodiments, a chip product is also provided, which includes the processor described above. Optionally, the chip product may be a GPU chip product, where the processor described above is a GPU, and the GPU chip product includes the GPU described above. Optionally, the above chip product may be implemented as a graphics card, which includes the processor described above, such as a GPU.

[0196] In some embodiments, a computer device is also provided, which includes the processor described above. Optionally, the computer device may be a personal computer, workstation, game console, and some mobile devices (such as tablet computers, smartphones, etc.), or it may be an in-vehicle terminal device, smart home device, smart TV, smart robot, etc., or it may be a server, server cluster, artificial intelligence computing cluster, cloud computing cluster, etc., wherein the artificial intelligence computing cluster may also be simply referred to as an intelligent computing cluster or smart computing cluster, and this application does not limit it in this way.

[0197] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0198] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A processor, characterized in that, The processor includes a tensor computation engine and at least two memory processing elements, wherein n computation thread bundles run on the memory processing elements, and each computation thread bundle is configured with multiple synchronization entries, where n is a positive integer. The memory processing element is used to call the first computation thread bundle among the n computation thread bundles running on the memory processing element to generate a computation instruction, which is used to instruct the tensor computation engine to perform a computation task; if there is an unused synchronization entry among the multiple synchronization entries corresponding to the first computation thread bundle, the computation instruction is sent to the tensor computation engine. The tensor computation engine is used to record the synchronization status of the computation instructions through a first synchronization entry, wherein the first synchronization entry is any unused synchronization entry among the plurality of synchronization entries.

2. The processor according to claim 1, characterized in that, The first computing thread bundle is equipped with a throttling counter, which is used to record the number of synchronization entries in the multiple synchronization entries that are in the used state. The number of multiple synchronization entries is m, where m is an integer greater than 1. The memory processing element is also used for: If the value of the throttling counter is less than m, the calculation instruction is sent to the tensor calculation engine; Increase the value of the throttling counter.

3. The processor according to claim 2, characterized in that, The tensor computation engine is further configured to mark the first synchronization entry as used after recording the synchronization status of the computation instruction through the first synchronization entry, and to mark the first synchronization entry as unused when the computation instruction and the computation instruction corresponding to the related computation thread bundle have completed synchronization. The related computation thread bundle is the computation thread bundle related to the first computation thread bundle in other memory processing elements, and the other memory processing elements are memory processing elements other than the memory processing element in the at least two memory processing elements. The memory processing element is configured to decrease the value of the throttling counter when it is determined that the first synchronization entry has been re-recorded as the unused state.

4. The processor according to claim 2, characterized in that, When n is greater than 1, the n computational thread bundles form a first queue; the memory processing element is used for: When the value of the throttling counter is equal to m, the first calculation thread bundle is removed from the first queue to obtain the updated first queue; Select a computation thread bundle from the updated first queue and send the computation instructions to the tensor computation engine.

5. The processor according to claim 4, characterized in that, The memory processing element is used for: If the value of the throttling counter recovers to less than m, the first calculation thread bundle is added to the updated first queue to obtain the recovered first queue; Select a computation thread bundle from the restored first queue and send the computation instructions to the tensor computation engine.

6. The processor according to any one of claims 1 to 5, characterized in that, The memory processing element is also used to send multiple computation instructions to the tensor computation engine for the first computation thread bundle; The tensor computation engine is further configured to, for consecutive first and second computation instructions among the plurality of computation instructions, read the source operands of the second computation instruction during the execution of the computation task of the first computation instruction, when there is no data dependency between the first and second computation instructions.

7. The processor according to claim 6, characterized in that, The memory processing element is also used for: In the case where there is a data dependency between the first calculation instruction and the second calculation instruction, after sending the first calculation instruction to the tensor calculation engine, the first scoreboard corresponding to the first calculation instruction is updated to obtain the updated first scoreboard. The first scoreboard is used to indicate whether the execution result of the calculation instruction has been written back.

8. The processor according to claim 7, characterized in that, The memory processing element is further configured to restore the first scoreboard of the first calculation instruction and obtain the restored first scoreboard when the first calculation instruction and the associated calculation instruction of the first calculation instruction are synchronized. The tensor computation engine is further configured to obtain the source operands corresponding to the second computation instruction when the second scoreboard of the second computation instruction is detected to match the recovered first scoreboard. The second scoreboard is used to indicate whether the source operands of the computation instruction are ready. The tensor calculation engine is also used to send first information to the memory processing element, the first information being used to instruct the memory processing element to restore the second scoreboard of the second calculation instruction.

9. The processor according to claim 7, characterized in that, The first calculation instruction belongs to the first instruction group. The first instruction group includes calculation instructions related to the first calculation instruction among the various calculation instructions corresponding to the first thread bundle group. The first thread bundle group is the thread bundle group to which the first calculation thread bundle belongs. The first thread bundle group includes calculation thread bundles related to the first calculation thread bundle among the at least two memory processing elements. The tensor computation engine is used for: When the source operand of the first calculation instruction is obtained, the first calculation instruction is decomposed into multiple micro-operation instructions, and the source operand of the first calculation instruction is decomposed into multiple sub-data, each of the micro-operation instructions corresponding to one of the multiple sub-data; For the first micro-operation instruction among the plurality of micro-operation instructions, the tag information of the first micro-operation instruction is determined. The tag information of the first micro-operation instruction includes: the identification information of the first storage unit and the identification information of the first thread bundle group. The first storage unit is used to store the execution result of the first micro-operation instruction. If a write-back entry containing tag information of the first micro-operation instruction is detected in the write-back cache device of the tensor computation engine, the allocation of new write-back entries to the first micro-operation instruction is stopped, and the new write-back entries of the first micro-operation instruction are used to cache the execution result of the first micro-operation instruction. If it is detected that there is no write-back entry including the tag information of the first micro-operation instruction in the write-back cache device, the new write-back entry is allocated to the first micro-operation instruction, and the tag information of the first micro-operation instruction is recorded through the new write-back entry of the first micro-operation instruction.

10. The processor according to claim 9, characterized in that, The tensor computation engine is used for: When the first micro-operation instruction is assigned the new write-back entry, the sub-data corresponding to the first micro-operation instruction is processed according to the first micro-operation instruction to obtain the execution result of the first micro-operation instruction; The execution result of the first micro-operation instruction is stored in the newly written-back entry; The execution result of the first micro-operation instruction is sent to the memory processing element to be stored in the first storage unit in the memory processing element; If the execution result of the first micro-operation instruction is completely stored in the first storage unit, the newly written-back entry is released; Among them, when the plurality of micro-operation instructions are executed, the first calculation instruction is executed.

11. A chip product, characterized in that, The chip product includes the processor as described in any one of claims 1 to 10.

12. A computer device, characterized in that, The computer device includes a processor as described in any one of claims 1 to 10.

13. An instruction processing method applied to a processor, characterized in that, The processor includes a tensor computation engine and at least two memory processing elements, wherein n computation thread bundles run on the memory processing elements, and each computation thread bundle is configured with multiple synchronization entries, where n is a positive integer. The method includes: The memory processing element calls the first of the n computation thread bundles running on the memory processing element to generate computation instructions, which are used to instruct the tensor computation engine to perform computation tasks. If there is an unused synchronization entry among the multiple synchronization entries corresponding to the first computation thread bundle, the memory processing element sends the computation instruction to the tensor computation engine. The tensor computation engine records the synchronization status of the computation instructions through a first synchronization entry, which is any unused synchronization entry among the plurality of synchronization entries.

Images