Processing unit, dynamic binning and pipelining method

Abstract

The present application provides a processing unit and a dynamic block and pipeline method. The processing unit includes a memory and an operation unit electrically connected to the memory. The operation unit dynamically calculates a block size according to a capacity of the memory, divides source data into a plurality of blocks according to the block size, loads one of the blocks into the memory, and performs an operation on the loaded block.

Description

Technical Field

[0001] This invention relates to a processing unit, and more particularly to a processing unit and a dynamic segmentation and pipeline method using a dynamic segmentation and pipeline method. Background Technology

[0002] In processing units (such as Neural Processing Units, NPUs), due to the limited internal memory capacity for computation, large external tensor data is typically tiled before being loaded into internal memory in batches for computation. However, current processing units often employ fixed block sizes and a single instruction queue for block operations. In this design, loading, computation, and storage operations must be performed sequentially, requiring different hardware processing units to wait for each other. Because of the single instruction queue, loading and storage operations are serialized, preventing parallel execution even if different blocks do not overlap in memory addresses.

[0003] Furthermore, traditional block sizes often need to be manually adjusted based on hardware specifications. Whenever hardware specifications change, the mismatch between the block size and the current specifications can lead to wasted memory, buffer overflow, or performance degradation. Memory can be divided into multiple subspaces; the number of these subspaces is called the buffer factor. Fixed buffer factors also present similar problems: a buffer factor that is too small cannot fully utilize computational resources, while a buffer factor that is too large may lead to over-provisioning and wasted memory space in scenarios with small tensor data or limited memory capacity. Summary of the Invention

[0004] This invention proposes a processing unit comprising a memory and an arithmetic unit. The arithmetic unit is electrically connected to the memory and is used to dynamically calculate the block size based on the memory capacity. The arithmetic unit is used to divide source data into multiple blocks according to the block size, load one of these blocks into the memory, and perform operations on the loaded block.

[0005] In one embodiment of the present invention, the above-mentioned computation unit is further used to calculate the block size based on the number of tensors, buffer factor, element size, and vector unit width.

[0006] In one embodiment of the present invention, the above-mentioned arithmetic unit is used to divide the capacity of the memory by the product of the number of tensors, the buffer factor and the element size, and then align it down to the width of the vector unit to calculate the block size.

[0007] In one embodiment of the present invention, the above-described arithmetic unit is used to calculate the block size during the compilation stage or the execution stage.

[0008] In one embodiment of the present invention, the aforementioned arithmetic unit is used to set multiple candidate buffer factors and calculate the block size corresponding to each candidate buffer factor. The arithmetic unit determines whether the block size corresponding to each candidate buffer factor is greater than the width of the vector unit to select a candidate buffer factor as the current buffer factor.

[0009] In one embodiment of the present invention, the processing unit further includes multiple arithmetic modules. The arithmetic unit divides the memory space into multiple buffer spaces according to the current buffer factor, and loads multiple first blocks into the buffer spaces respectively. The arithmetic unit is also used to allocate the first blocks to the arithmetic modules, which are used to process the first blocks in parallel.

[0010] In one embodiment of the present invention, the aforementioned arithmetic unit is used to manage multiple instruction queues, decompose the aforementioned operations into multiple stages, and add each stage to a corresponding instruction queue. The arithmetic unit is used to establish the dependencies between the stages of the first block, and retrieve a stage from the instruction queue according to the dependencies and allocate it to one of the arithmetic modules.

[0011] In one embodiment of the present invention, the aforementioned stages include a loading stage, a computation stage, and a storage stage. The aforementioned instruction queue includes a loading queue, a computation queue, and a storage queue. The arithmetic unit is used to add the loading stage to the loading queue, the computation stage to the computation queue, and the storage stage to the storage queue.

[0012] In one embodiment of the present invention, the plurality of blocks described above includes a first block and a second block. The size of the first block is different from the size of the second block. The arithmetic unit is used to perform operations on the first block and the second block simultaneously.

[0013] This invention also proposes a dynamic partitioning and pipeline method for a memory. This dynamic partitioning and pipeline method includes: dynamically calculating the block size based on the memory capacity; dividing the source data into multiple blocks according to the block size; and loading one of the blocks into the memory and performing operations on the loaded block.

[0014] In the aforementioned processing unit and dynamic partitioning and pipeline method, the appropriate block size can be dynamically derived based on the memory capacity, and the buffer factor can be dynamically determined. This improves memory utilization efficiency and overall computational throughput. Attached Figure Description

[0015] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings.

[0016] Figure 1This is a schematic diagram of a host system and input / output (I / O) devices according to an exemplary embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of a host system, a memory storage device, and an I / O device according to an exemplary embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of a memory storage device according to an exemplary embodiment of the present invention;

[0019] Figure 4 This is a schematic diagram of a processing unit according to an exemplary embodiment of the present invention dividing source data into blocks and loading it into a memory;

[0020] Figure 5 This is a schematic diagram of a vector unit according to an exemplary embodiment of the present invention;

[0021] Figure 6 This is a schematic diagram of multiple buffer spaces in a memory as shown in an exemplary embodiment of the present invention;

[0022] Figure 7 This is a schematic diagram of multiple instruction queues as shown in an exemplary embodiment of the present invention;

[0023] Figure 8 This is a schematic diagram illustrating the execution of multiple stages of multiple blocks over time, as shown in an exemplary embodiment of the present invention;

[0024] Figure 9 This is a schematic diagram illustrating the execution of different stages at different times by each buffer space when the buffer factor is 3, as shown in an exemplary embodiment of the present invention.

[0025] Figure 10 This is a schematic diagram illustrating the execution of different stages at different times by each buffer space when the buffer factor is 2, as shown in an exemplary embodiment of the present invention.

[0026] Figure 11 This is a flowchart illustrating a dynamic segmentation and pipeline method according to an exemplary embodiment of the present invention. Detailed Implementation

[0027] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered identical or similar when they appear in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of systems and methods within the scope of the claims of the present invention.

[0028] The terms "first," "second," etc., used in this article do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms.

[0029] Figure 1 This is a schematic diagram of a host system and input / output (I / O) devices according to an exemplary embodiment of the present invention. Figure 2 This is a schematic diagram of a host system, a memory storage device, and an I / O device according to an exemplary embodiment of the present invention.

[0030] Please refer to Figure 1 and Figure 2 The host system 11 can be a desktop computer, server, distributed system, laptop computer, etc., and this invention is not limited thereto. The host system 11 may include a processor 111, random access memory (RAM) 112, read-only memory (ROM) 113, data transfer interface 114, and processing unit 115. The processor 111, RAM 112, ROM 113, data transfer interface 114, and processing unit 115 may be coupled to a system bus 110. The processor 111 may be a central processing unit. The processing unit 115 may be a graphics processing unit (GPU), tensor processing unit (TPU), neural processing unit (NPR), etc. The processing unit 115 includes a memory 116, which may be static random access memory (SRAM) or video random access memory (VRAM), etc.

[0031] In one exemplary embodiment, processor 111 can be coupled to memory storage device 10 via data transfer interface 114. For example, processor 111 can store data to or read data from memory storage device 10 via data transfer interface 114. Furthermore, host system 11 can be coupled to I / O device 12 via system bus 110. For example, host system 11 can transmit output signals to I / O device 12 or receive input signals from I / O device 12 via system bus 110. In this embodiment, multiple components within host system 11 are coupled to each other via system bus 110, but in other embodiments, any two components in host system 11 can be coupled to each other in a point-to-point manner.

[0032] In one exemplary embodiment, the processor 111, random access memory 112, read-only memory 113, and data transfer interface 114 may be disposed on the motherboard 20 of the host system 11. The number of data transfer interfaces 114 may be one or more. Through the data transfer interface 114, the motherboard 20 may be coupled to the memory storage device 10 via wired or wireless means.

[0033] In one exemplary embodiment, the memory storage device 10 may be, for example, a USB flash drive 201, a memory card 202, or a solid state drive (SSD) 203. In some embodiments, the memory storage device 10 may be located externally to the host system 11 as a wireless memory storage device 204. The wireless memory storage device 204 may be, for example, a Near Field Communication (NFC) memory storage device, a WiFi wireless fax memory storage device, a Bluetooth memory storage device, or a Bluetooth Low Energy (BLE) memory storage device (e.g., iBeacon), or other memory storage devices based on various wireless communication technologies. Furthermore, the motherboard 20 may also be coupled to various I / O devices such as a Global Positioning System (GPS) module 205, a network interface card 206, a wireless transmission device 207, a keyboard 208, a screen 209, and a speaker 210 via the system bus 110. For example, in one exemplary embodiment, the motherboard 20 may access the wireless memory storage device 204 via the wireless transmission device 207.

[0034] Figure 3 This is a schematic diagram of a memory storage device according to an exemplary embodiment of the present invention. Please refer to... Figure 3 The memory storage device 10 includes a connection interface unit 31, a memory control circuit unit 32, and a rewritable non-volatile memory module 33.

[0035] The interface unit 31 is used to couple to the processor 111. The memory storage device 10 can communicate with the processor 111 via the interface unit 31. In one exemplary embodiment, the interface unit 31 is compatible with the Peripheral Component Interconnect Express (PCI Express) standard. In one exemplary embodiment, the connection interface unit 31 may also conform to the Serial Advanced Technology Attachment (SATA) standard, the Parallel Advanced Technology Attachment (PATA) standard, the Institute of Electrical and Electronics Engineers (IEEE) 1394 standard, the Universal Serial Bus (USB) standard, the SD interface standard, the Ultra High Speed-I (UHS-I) interface standard, the Ultra High Speed-II (UHS-II) interface standard, the Memory Stick (MS) interface standard, the MCP interface standard, the MMC interface standard, the eMMC interface standard, the Universal Flash Storage (UFS) interface standard, the eMCP interface standard, the CF interface standard, the Integrated Device Electronics (IDE) standard, or other suitable standards. The connection interface unit 31 may be packaged in the same chip as the memory control circuit unit 32, or the connection interface unit 31 may be disposed outside the chip containing the memory control circuit unit 32.

[0036] The memory control circuit unit 32 is coupled to the connection interface unit 31 and the rewritable non-volatile memory module 33. The memory control circuit unit 32 is used to execute multiple logic gates or control instructions implemented in hardware or firmware, and to perform operations such as writing, reading and erasing data in the rewritable non-volatile memory module 33 according to the instructions of the processor 111.

[0037] The rewritable non-volatile memory module 33 is used to store data written by the processor 111. The rewritable non-volatile memory module 33 may include a single-level cell (SLC) NAND flash memory module (i.e., a flash memory module that can store 1 bit in one cell), a multi-level cell (MLC) NAND flash memory module (i.e., a flash memory module that can store 2 bits in one cell), a triple-level cell (TLC) NAND flash memory module (i.e., a flash memory module that can store 3 bits in one cell), a quadruple-level cell (QLC) NAND flash memory module (i.e., a flash memory module that can store 4 bits in one cell), other flash memory modules, or other memory modules with the same characteristics.

[0038] Each memory cell in the rewritable nonvolatile memory module 33 stores one or more bits by changing a voltage (hereinafter also referred to as the threshold voltage). Specifically, each memory cell has a charge trapping layer between its control gate and channel. By applying a write voltage to the control gate, the amount of electrons in the charge trapping layer can be changed, thereby changing the threshold voltage of the memory cell. This operation of changing the threshold voltage of the memory cell is also called "writing data to the memory cell" or "programming the memory cell". As the threshold voltage changes, each memory cell in the rewritable nonvolatile memory module 33 has multiple storage states. By applying a read voltage, it can be determined which storage state a memory cell belongs to, and thus the one or more bits stored in that memory cell can be retrieved.

[0039] In one exemplary embodiment, the memory cells of the rewritable non-volatile memory module 33 can constitute multiple physical programming units, and these physical programming units can constitute multiple physical erase units. Specifically, memory cells on the same word line can form one or more physical programming units. If each memory cell can store more than two bits, then physical programming units on the same word line can be classified into at least lower physical programming units and upper physical programming units. For example, the least significant bit (LSB) of a memory cell belongs to the lower physical programming unit, and the most significant bit (MSB) of a memory cell belongs to the upper physical programming unit. Generally, in MLC NAND flash memory, the write speed of the lower physical programming unit is greater than that of the upper physical programming unit, and / or the reliability of the lower physical programming unit is higher than that of the upper physical programming unit.

[0040] In some embodiments, the processor 111 or the memory control circuit unit 32 may determine the programming mode for writing data to the rewritable non-volatile memory module 33. When using a single-level memory cell programming mode, data is written to the lower physical programming unit. When using a multi-level (including second-level, third-level, fourth-level, etc.) memory cell programming mode, data is written to at least the lower physical programming unit and the upper physical programming unit.

[0041] Figure 4 This is a schematic diagram illustrating how a processing unit, according to an exemplary embodiment of the present invention, divides source data into blocks and loads it into memory. Please refer to... Figure 4 The processing unit 115 includes a memory 116 and an arithmetic unit 430. The arithmetic unit 430 is electrically connected to the memory 116 and is used to perform operations on blocks loaded into the memory 116.

[0042] In the field of artificial intelligence, tile partitioning refers to dividing a large tensor into multiple smaller sub-data blocks, allowing each sub-data block to be loaded into a limited high-speed memory for computation. Tiling avoids memory overflow caused by loading the entire large tensor at once, or performance bottlenecks caused by frequent access to slow main memory. Simultaneously, tile partitioning allows the computation unit 430 to continuously perform vectorized operations on consecutive and adjacent sub-data blocks, thereby improving data locality and the utilization rate of the computation unit.

[0043] like Figure 4As shown, the operation unit 430 divides the source data 410 into multiple blocks 420 according to the block size, loads at least one block into the memory 116, and then performs operations on the loaded blocks. The source data 410 can be a weight matrix, feature map, activation value tensor, input feature vector, or other large tensor data in a neural network.

[0044] In some embodiments, the arithmetic unit 430 can dynamically calculate the block size during the compilation or execution phase. The compilation phase, for example, refers to the stage where, before the neural network model is deployed to the processing unit 115, the software compiler converts the high-order code into instructions executable by the processing unit 115. The execution phase, for example, refers to the stage after the neural network model is deployed, where the processing unit 115 actually receives the arithmetic instructions and prepares to perform operations on the source data 410. By dynamically calculating the block size, the block size can be adaptively adjusted according to various information such as the capacity of the memory 116, thereby maintaining better memory utilization and computational efficiency across different hardware specifications or different network structures.

[0045] In some embodiments, the block sizes of the multiple blocks 420 may be different. For example, these blocks 420 may include a first block and a second block, which have different block sizes. In other words, the aforementioned "dynamically calculating the block size" can also refer to calculating an appropriate block size for each operation in the operation of the artificial intelligence model.

[0046] Figure 5 This is a schematic diagram of a vector unit according to an exemplary embodiment of the present invention. Please refer to... Figure 5 The computation unit 430 includes a vector unit 510. The vector unit 510 employs a Single Instruction Multiple Data (SIMD) architecture. The vector unit 510 can store more than one data element, enabling the hardware to process multiple data elements in parallel within a single computation cycle. In the inference and training processes of neural networks, operations such as matrix multiplication, convolution, and activation functions exhibit significant parallelism, which can be accelerated using the SIMD architecture.

[0047] exist Figure 5 In the example, the bit width of vector unit 510 is W = 1024 bits. When the element size dtype of each data element is 32 bits, vector unit 510 can process 1024 / 32 = 32 data elements simultaneously in one operation cycle, that is, the vector unit width W... simdThe number of elements is 32. In other embodiments, the bit width W of the vector unit 510 can also be 128 bits, 256 bits, 512 bits, 2048 bits, or other bit widths, and the present invention does not limit this. The element size dtype depends on the data type, and can be, for example, 8 bits, 16 bits, 32 bits, or 64 bits.

[0048] In some embodiments, the operation unit 430 is further configured to calculate the block size based on the number of tensors, the buffer factor, the element size, and the vector cell width. Specifically, the operation unit 430 divides the capacity of the memory 116 by the product of the number of tensors, the buffer factor, and the element size, and then aligns it down to the vector cell width to calculate the block size. This calculation can be expressed as: Block size = Downalignment (S / (K × D × dtype), W) simd Where S represents the capacity of memory 116 available for block loading (e.g., in bytes). K represents the number of tensors, that is, the number of tensors that need to be loaded into memory 116 simultaneously in one operation (e.g., for matrix multiplication, two input tensors plus one output tensor, K=3). D represents the buffer factor, which determines how many buffer spaces memory 116 will be divided into to support pipeline execution. dtype represents the element size, that is, the number of bytes occupied by a single data element. W simd This represents the width of the vector unit. Down alignment refers to adjusting the quotient (S / (K × D × dtype)) to W. simd The value is an integer multiple of the quotient and not greater than the maximum value of the quotient. Therefore, it is ensured that each block can completely correspond to the width of one operation of the vector unit 510, avoiding the appearance of non-divisible residual elements at the block boundaries, which would increase the additional processing cost.

[0049] For example, when the memory capacity S of memory 116 is 65536 bytes (i.e., 64Kbytes), the number of tensors K is 3, the buffer factor D is 3, the element size dtype is 4 bytes, and the vector cell width W... simd When the value is 32, the quotient 65536 / (3 × 3 × 4) ≈ 1820 can be calculated first, and then aligned downwards to an integer multiple of 32 to obtain a block size of 1792. Using this formulaic derivation method, the operation unit 430 can automatically calculate the block size that can be fully loaded into the memory 116 and aligned with the width of the vector unit 510 under different hardware specifications and different tensor configurations.

[0050] Figure 6 This is a schematic diagram illustrating multiple buffer spaces in a memory according to an exemplary embodiment of the present invention. Please refer to... Figure 6The arithmetic unit 430 can divide the space of the memory 116 into multiple buffer spaces. Each buffer space is used to hold a block of data and temporary data required during the block execution operation. The number of buffer spaces divided into the memory 116 is equal to the aforementioned buffer factor D.

[0051] Figure 6 Taking two buffer factors as examples: when the buffer factor D=2, the memory 116 is divided into two buffer spaces 611 and 612; when the buffer factor D=3, the memory 116 is divided into three buffer spaces 601, 602, and 603. By setting multiple buffer spaces, the arithmetic unit 430 can allocate different blocks to different buffer spaces for simultaneous processing. A larger buffer factor D means that the memory 116 can accommodate more blocks simultaneously, but the available capacity of each buffer space will be relatively reduced; a smaller buffer factor D means that the available capacity of each buffer space is larger.

[0052] In some embodiments, the operation unit 430 is used to select one from a plurality of candidate buffer factors as the current buffer factor. Specifically, the operation unit 430 first sets a plurality of candidate buffer factors (e.g., including 3 and 2), and then selects the current buffer factor according to the aforementioned formula (block size = downalignment (S / (K × D × dtype), W...). simd The block size corresponding to each candidate buffer factor is calculated separately. Then, the computation unit 430 determines whether the block size corresponding to each candidate buffer factor is greater than the vector unit width W. simd Therefore, it is necessary to identify which candidate buffer factors meet the criteria. The block size is greater than the vector unit width W. simd It is a necessary condition for the block size to be effectively utilized: if the block size corresponding to a certain candidate buffer factor is insufficient to fill the number of elements required for one operation of the vector unit 510, then the vector unit 510 will have idle computing resources when processing the block, so the candidate buffer factor should not be selected.

[0053] exist Figure 6 In the example, the arithmetic unit 430 prioritizes a larger buffer factor, i.e., it prefers D=3. When the condition D=3 is met (i.e., the block size calculated when D=3 is still greater than the vector unit width), the arithmetic unit 430 uses D=3 as the current buffer factor and also outputs the corresponding block size; if the block size calculated when D=3 is less than the vector unit width, the arithmetic unit 430 then chooses D=2 as the current buffer factor, in which case the block size will be relatively larger. In this way, if the memory 116 has sufficient capacity, three buffer spaces can be configured as much as possible.

[0054] In other embodiments, the operation unit 430 may also select the current buffer factor from the eligible candidate buffer factors in other ways. For example, the operation unit 430 may perform performance simulations for each eligible candidate buffer factor and select the best performing one as the current buffer factor based on indicators such as execution time, throughput, or memory utilization under expected load. Alternatively, the operation unit 430 may directly obtain the corresponding buffer factor by using a pre-established lookup table with parameters such as the capacity of memory 116, the number of tensors, and the element size as indexes. In some embodiments, the candidate buffer factors are not limited to 2 or 3, and may include 4, 5, or more; the present invention does not impose any limitations on this.

[0055] Figure 7 This is a schematic diagram of multiple instruction queues according to an exemplary embodiment of the present invention. In some embodiments, the processing unit 115 further includes multiple arithmetic modules 701-703. The arithmetic unit 430 divides the space of the memory 116 into multiple buffer spaces according to the current buffer factor D, loads multiple first blocks into these buffer spaces respectively, and then allocates these first blocks to the arithmetic modules 701-703, which process these first blocks in parallel. The first blocks are, for example, subsets of the multiple blocks 420 currently prepared for operation, and their number roughly corresponds to the current buffer factor.

[0056] Furthermore, the arithmetic unit 430 manages multiple instruction queues, decomposing the operations performed on each block into multiple stages and adding each stage to a corresponding instruction queue. In some embodiments, these stages include a loading stage, a computation stage, and a storage stage, and the aforementioned multiple instruction queues correspondingly include a loading queue, a computation queue, and a storage queue. Please refer to... Figure 7 The arithmetic unit 430 adds the loading stage of each block to the loading queue, the calculation stage of each block to the calculation queue, and the storage stage of each block to the storage queue. Specifically, arithmetic module 701 corresponds to the loading queue and executes the loading stage therein; arithmetic module 702 corresponds to the calculation queue and executes the calculation stage therein; and arithmetic module 703 corresponds to the storage queue and executes the storage stage therein. Arithmetic modules 701-703 can also be referred to as arithmetic circuits.

[0057] Figure 7Let's take four blocks as an example: the loading queue has four loading stages in sequence: L0, L1, L2, and L3; the computation queue has four computation stages in sequence: C0, C1, C2, and C3; and the storage queue has four storage stages in sequence: S0, S1, S2, and S3. Specifically, L0, C0, and S0 correspond to the loading, computation, and storage stages of the same block (which can be considered the 0th block); L1, C1, and S1 correspond to the loading, computation, and storage stages of the next block (which can be considered the 1st block); and so on. L2, C2, and S2 correspond to the 2nd block, and L3, C3, and S3 correspond to the 3rd block.

[0058] The processing unit 430 retrieves stages from the instruction queue sequentially based on the dependencies between these blocks at each stage, and assigns the retrieved stages to the corresponding processing modules 701-703. By diverting different stages to different instruction queues and assigning them to different processing modules, multiple stages that originally had to be executed sequentially can be executed in an overlapping manner between different blocks in a pipeline manner.

[0059] Figure 8 This is a schematic diagram illustrating the execution of multiple stages of multiple blocks over time, as shown in an exemplary embodiment of the present invention. Please refer to... Figure 7 and Figure 8 , Figure 8 The horizontal axis represents time, and the vertical axis, from top to bottom, sequentially lists the loading queue, the calculation queue, and the storage queue. These queues contain the stages executed by the operation modules 701, 702, and 703, respectively.

[0060] For the same block, there are strict dependencies between its stages. For example, for the 0th block, its computation stage C0 can only begin after the loading stage L0 is completed, and its storage stage S0 can only begin after the computation stage C0 is completed; this is because the computation stage C0 needs to use the data loaded by the loading stage L0, and the storage stage S0 needs to write the results produced by the computation stage C0. For different blocks, since each block uses a different buffer space, there are no data dependencies between different stages belonging to different blocks, and they can be executed simultaneously by different computation modules.

[0061] Under the scheduling of the arithmetic unit 430, the three arithmetic modules 701, 702, and 703 can obtain stages from their respective instruction queues and execute them simultaneously. This allows, for example, the storage stage S0 of the 0th block, the computation stage C1 of the 1st block, and the loading stage L2 of the 2nd block to be performed in parallel within the same time period, achieving the effect of overlapping the three types of arithmetic pipelines: loading, computation, and storage.

[0062] In this embodiment, Figure 8All blocks in the process have the same block size. As mentioned above, in some embodiments, the block size of each block can be determined independently and therefore is not the same as the others. It is worth noting that the loading, computation, and storage operations corresponding to the same block must use the same block size, but the block sizes of different blocks can be different. For example, the block size of the 0th block is 4x4, and the block size of the 1st block is 2x2, and the operations of these two blocks can be executed simultaneously.

[0063] Figure 9 This is a schematic diagram illustrating the execution of different stages at different times by each buffer space when the buffer factor is 3, as shown in an exemplary embodiment of the present invention. Please refer to... Figure 9 In the figure, the horizontal axis from left to right represents the three buffer spaces 601, 602 and 603 configured in memory 116, and the vertical axis represents time. Figure 9 This is illustrated using a scenario where the buffer factor D=3.

[0064] For the same block, its loading, computation, and storage phases are executed sequentially in the same buffer space. That is, the computation module 701 first loads the block into the buffer space, the computation module 702 then performs computations on the data in the buffer space, and finally the computation module 703 reads the computation result from the same buffer space and writes it back. When the buffer factor D=3, the three buffer spaces 601, 602, and 603 can respectively carry different phases of three different blocks in the same time period, so that the three types of operations of loading, computation, and storage completely overlap on the time axis. When the buffer space configured in the memory 116 is used to carry the loading phase of a new block, the block previously carried by the buffer space must have completed the storage phase; otherwise, the buffer space still contains data that has not been written back and may be overwritten by the newly loaded data.

[0065] For example, after the system reaches a steady state, during a certain period, buffer space 601 may carry the storage stage Si of the i-th block, buffer space 602 may carry the computation stage Ci+1 of the (i+1)-th block, and buffer space 603 may carry the loading stage Li+2 of the (i+2)-th block. In the next period, the blocks carried by each buffer space advance one position, buffer space 601 carries the loading stage Li+3 of the new block, buffer space 602 carries the storage stage Si+1, and buffer space 603 carries the computation stage Ci+2. Thus, when the buffer factor D=3, the three buffer spaces take turns undertaking the roles of loading, computation, and storage in a rotating manner, enabling the three operation modules 701, 702, and 703 to execute at full capacity.

[0066] Figure 10This is a schematic diagram illustrating the execution of different stages at different times by each buffer space when the buffer factor is 2, as shown in an exemplary embodiment of the present invention. Please refer to... Figure 10 In the figure, the horizontal axis from left to right represents the two buffer spaces 611 and 612 configured in memory 116, and the vertical axis represents time. Figure 10 This is illustrated using a scenario where the buffer factor D=2.

[0067] Similar to Figure 9 For the same block, its loading phase, computation phase, and storage phase are still executed sequentially on the same buffer space. When the buffer factor D=2, only two buffer spaces 611 and 612 are configured in memory 116 for use in turn: when one buffer space is executing the loading phase, the other buffer space can execute the computation phase or the storage phase at the same time; after the i-th block completes the storage phase in buffer space 611, buffer space 611 can be released to carry the loading phase of the (i+2)-th block.

[0068] Compared to the scenario where the buffer factor D=3, the buffer factor D=2 allows for a larger block size per buffer space (because the memory 116 is divided into fewer parts). However, only the different stages on two blocks can overlap; for example, the loading stage and the computation stage can overlap, as can the computation stage and the storage stage. It is not possible to achieve simultaneous parallel processing of loading, computation, and storage operations. Therefore, D=2 is suitable for scenarios where the memory 116 has limited capacity and it is difficult to further divide it into three buffer spaces.

[0069] Figure 11 This is a flowchart illustrating a dynamic segmentation and pipeline method according to an exemplary embodiment of the present invention. Figure 11 The method shown can be executed by the aforementioned processing unit 115, for example, by the arithmetic unit 430 therein. Please refer to... Figure 11 The dynamic segmentation and pipeline method in this embodiment includes the following steps.

[0070] In step 1101, the arithmetic unit 430 dynamically calculates the block size based at least on the capacity of the memory 116. That is, the arithmetic unit 430 uses the currently available capacity of the memory 116 as input and dynamically derives a suitable block size based on the current hardware and tensor configuration, avoiding the use of a pre-set fixed block size.

[0071] In step 1102, the arithmetic unit 430 divides the source data 410 into multiple blocks 420 according to the block size calculated in step 1101. That is, the arithmetic unit 430 logically divides the large source data 410 into multiple blocks 420 of the same size according to the block size determined in the previous step, so that they can be loaded into the memory 116 in batches for subsequent processing.

[0072] In step 1103, the arithmetic unit 430 loads one of the blocks 420 into the memory 116 and performs operations on the loaded block. In some embodiments, the above-described dynamic block and pipeline method may further include the following additional steps, which may be used individually or in combination to further improve performance. In some embodiments, the step of dynamically calculating the block size based on the capacity of the memory 116 in step 1101 may further include: calculating the block size based on the number of tensors, the buffer factor, the element size, and the vector cell width; specifically, the capacity of the memory 116 may be divided by the product of the number of tensors, the buffer factor, and the element size, and then aligned down to the vector cell width to calculate the block size.

[0073] In summary, the processing unit and dynamic block partitioning and pipeline method of this invention can automatically calculate appropriate block sizes under different hardware specifications and tensor configurations, avoiding memory waste, buffer overflow, and performance degradation caused by manual adjustments. In this invention, the calculated block size not only fits completely into memory but also precisely aligns with the computational width of the vector units, ensuring that each computation fully utilizes the parallel computational capabilities of the vector units. Furthermore, this invention dynamically determines the amount of buffer space, enabling parallel processing with multiple computation modules and instruction queues, thereby improving overall throughput and vector unit utilization.

[0074] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A processing unit, characterized in that, Include: Memory; and An arithmetic unit, electrically connected to the memory, is used to dynamically calculate the block size based on the capacity of the memory. The arithmetic unit is used to divide the source data into multiple blocks according to the block size, load one of the multiple blocks into the memory, and perform operations on the loaded block.

2. The processing unit according to claim 1, characterized in that, The computation unit is also used to calculate the block size based on the number of tensors, buffer factor, element size, and vector unit width.

3. The processing unit according to claim 2, characterized in that, The arithmetic unit is used to divide the capacity of the memory by the product of the number of tensors, the buffer factor, and the element size, and then align it down to the width of the vector unit to calculate the block size.

4. The processing unit according to claim 1, characterized in that, The arithmetic unit is used to calculate the block size during the compilation or execution phase.

5. The processing unit according to claim 1, characterized in that, The computation unit is used to set multiple candidate buffer factors and calculate the block size corresponding to each of the multiple candidate buffer factors. The operation unit is used to determine whether the block size corresponding to each of the plurality of candidate buffer factors is greater than the width of the vector unit in order to select one of the plurality of candidate buffer factors as the current buffer factor.

6. The processing unit according to claim 5, characterized in that, It also includes multiple processing modules, wherein the processing unit divides the memory space into multiple buffer spaces according to the current buffer factor, and loads multiple first blocks of the multiple blocks into the multiple buffer spaces respectively. The processing unit is used to allocate the plurality of first blocks to the plurality of processing modules, and the plurality of processing modules are used to process the plurality of first blocks in parallel.

7. The processing unit according to claim 6, characterized in that, The arithmetic unit is used to manage multiple instruction queues, decompose the operation into multiple stages, and add each of the multiple stages to a corresponding one of the multiple instruction queues. The computing unit is used to establish the dependencies between the multiple stages of the multiple first blocks, and to obtain one of the multiple stages from the multiple instruction queues according to the dependencies and assign it to one of the multiple computing modules.

8. The processing unit according to claim 7, characterized in that, The plurality of stages include a loading stage, a computation stage, and a storage stage; the plurality of instruction queues include a loading queue, a computation queue, and a storage queue; the arithmetic unit is used to add the loading stage to the loading queue, the computation stage to the computation queue, and the storage stage to the storage queue.

9. The processing unit according to claim 1, characterized in that, The plurality of blocks include a first block and a second block, wherein the size of the first block is different from the size of the second block, and the arithmetic unit is used to execute the arithmetic operations of the first block and the second block simultaneously.

10. A dynamic segmentation and pipeline method, characterized in that, For memory, the dynamic block partitioning method includes: The block size is dynamically calculated based on the capacity of the memory; The source data is divided into multiple blocks based on the block size; One of the plurality of blocks is loaded into the memory and operations are performed on the loaded block.

11. The dynamic segmentation and pipeline method according to claim 10, characterized in that, The step of dynamically calculating the block size based on the capacity of the memory includes: The block size is calculated based on the number of tensors, buffer factor, element size, and vector cell width.

12. The dynamic segmentation and pipeline method according to claim 11, characterized in that, The step of dynamically calculating the block size based on the capacity of the memory includes: The block size is calculated by dividing the capacity of the memory by the product of the number of tensors, the buffer factor, and the element size, and then aligning it down to the width of the vector cell.

13. The dynamic segmentation and pipeline method according to claim 10, characterized in that, The step of dynamically calculating the block size based on the capacity of the memory includes: The block size is calculated during the compilation or execution phase.

14. The dynamic segmentation and pipeline method according to claim 10, characterized in that, Also includes: Multiple candidate buffer factors are set, and the block size corresponding to each of the multiple candidate buffer factors is calculated; as well as Determine whether the block size corresponding to each of the plurality of candidate buffer factors is greater than the vector unit width in order to select one of the plurality of candidate buffer factors as the current buffer factor.

15. The dynamic segmentation and pipeline method according to claim 14, characterized in that, Also includes: The memory space is divided into multiple buffer spaces according to the current buffer factor, and multiple first blocks of the multiple blocks are loaded into the multiple buffer spaces respectively; as well as The plurality of first blocks are assigned to a plurality of processing modules, which are used to process the plurality of first blocks in parallel.

16. The dynamic segmentation and pipeline method according to claim 15, characterized in that, Also includes: Manage multiple instruction queues; The operation is decomposed into multiple stages, and each of the multiple stages is added to a corresponding one of the multiple instruction queues; as well as Establish the dependencies between the multiple stages of the multiple first blocks, and obtain one of the multiple stages from the multiple instruction queues according to the dependencies and assign it to one of the multiple operation modules.

17. The dynamic segmentation and pipeline method according to claim 16, characterized in that, The plurality of stages includes a loading stage, a computation stage, and a storage stage; the plurality of instruction queues includes a loading queue, a computation queue, and a storage queue; and the dynamic block partitioning method further includes: The loading phase is added to the loading queue, the computation phase is added to the computation queue, and the storage phase is added to the storage queue.

18. The dynamic segmentation and pipeline method according to claim 10, characterized in that, The plurality of blocks includes a first block and a second block, wherein the size of the first block is different from the size of the second block, and the dynamic block and pipeline method further includes: The operations of the first block and the second block are performed simultaneously.

Images