A tensor storage management method and device

Abstract

The present disclosure provides a tensor storage management method and device, wherein the method comprises: obtaining a memory access request of tensor data; selectively performing at least one of an address configuration operation and an access scheduling operation according to a memory access feature indicated by the memory access request; wherein the address configuration operation is used for: distributing the storage space of the tensor data according to the number of Banks required by the tensor data; and / or converting the logical address of the tensor data into a physical Bank address, so that the physical Bank addresses of different memory access requests are distributed on different Banks; the access scheduling operation is used for: when multiple memory access requests compete for the same Bank, temporarily delaying the processing of at least part of the requests, and / or decomposing one memory access request into multiple sub-requests and processing them in parallel with the computing operation. The present disclosure can adaptively cope with different efficiency bottlenecks, and significantly improve the actual memory bandwidth utilization rate of TMEM under different deep learning operators.

Description

Technical Field

[0001] This disclosure relates to the field of tensor storage technology, and more specifically, to a tensor storage management method and apparatus. Background Technology

[0002] Tensor cores, as the core computing components of modern AI accelerators, graphics processing units (GPUs), and neural network processors (NPUs), are used to efficiently execute deep learning operators such as convolution and general matrix multiplication (GEMM). To match the extremely high access bandwidth of tensor cores, tensor memory (TMEM), as a dedicated on-chip storage unit, typically employs a multi-bank parallel organization structure. Each bank can independently respond to read and write requests within the same clock cycle, thus ideally achieving a linear scaling of bandwidth with the number of banks.

[0003] However, in practical engineering applications, the actual memory access efficiency of TMEM is often far lower than its theoretical bandwidth. Therefore, there is an urgent need for a flexible organization method and access scheduling strategy for configuring storage resources to comprehensively improve the actual memory access efficiency of TMEM. Summary of the Invention

[0004] This disclosure provides at least one tensor storage management method and apparatus to solve the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this disclosure provide a tensor storage management method applied to a tensor memory (TMEM) containing multiple storage banks, comprising:

[0006] A memory access request to retrieve tensor data; Based on the memory access characteristics indicated by the memory access request, at least one of the address configuration operation and the access scheduling operation is selectively performed; The address configuration operation is used to: distribute the storage space of the tensor data according to the number of Banks required by the tensor data; and / or convert the logical address of the tensor data into a physical Bank address so that the physical Bank addresses of different memory access requests are distributed on different Banks; The access scheduling operation is used to: when multiple memory access requests compete for the same Bank, temporarily delay at least some of the requests, and / or decompose a memory access request into multiple sub-requests and process them in parallel with the computation operation.

[0007] In one possible implementation, distributing the storage space for the tensor data includes: Obtain the offset parameter and the number of banks parameter K, wherein the value of the number of banks parameter K corresponds to the target memory access bandwidth requirement of the tensor data, and the offset parameter is used to specify the starting storage location of the tensor data in the TMEM; Based on the offset parameter and the number of banks parameter K, the storage space of the tensor data is allocated to K banks in an interleaved manner.

[0008] In one possible implementation, if the offset parameter and the number of banks are not specified, the system falls back to a global interleaving or non-interleaving bank allocation method. Global interleaving means that the space allocated is interleaved across all banks, while non-interleaving means that the allocated space is stored sequentially into the corresponding banks.

[0009] In one possible implementation, the step of converting the logical address of the tensor data to a physical bank address includes: A non-linear physical Bank index is generated by performing an XOR operation on the row index bits and column index bits in the logical address of the tensor data using an XOR mask. Here, a logical row with a first granularity is defined as a scheduling envelope, which contains multiple data blocks with a second granularity. When the hardware execution unit accesses the scheduling envelope with the second granularity, the XOR operation causes different data blocks within the same scheduling envelope to be mapped to different physical Bank groups.

[0010] In one possible implementation, the first granularity size is 32, the second granularity size is 8, and one of the scheduling envelopes contains four data blocks, each mapped to one of four different physical bank groups.

[0011] In one possible implementation, the temporary delay processing of at least a portion of the requests includes: Configure a write-back cache for each Bank of TMEM; When read and write requests occur concurrently for the same Bank, the write data is temporarily stored in the corresponding write-back cache without blocking the read request. The TMEM controller monitors the read request queue of each Bank, and when it detects that the same Bank is in an idle period without read requests, the write data in the corresponding write-back cache is written back to that Bank.

[0012] In one possible implementation, the write-back cache is a register file or a static random access memory (SRAM) with a capacity smaller than the corresponding bank capacity; the write-back operation has a lower priority than the read request and is initiated after idle time is detected.

[0013] In one possible implementation, decomposing a memory access request into multiple sub-requests and processing them in parallel with computational operations includes: The output matrix corresponding to the memory access request is divided into multiple logical segments, and the size of each logical segment is configured such that the computation time can cover the maximum memory access latency required from the completion of segment computation to the completion of write-back. Once the computation result of a logical shard is generated, a write-back operation is initiated for that shard without waiting for other shards to complete their computations, allowing the write-back operation of that shard to be executed in parallel with the computation operations of subsequent shards.

[0014] In one possible implementation, the number of logical partitions is N, where N≥2; the method further includes: The size of the logical slice is set by a configurable register, and the slice size is dynamically adjusted according to the access delay parameter of TMEM and the calculation delay parameter of the computing unit.

[0015] Secondly, this disclosure also provides a tensor memory management device for use in a tensor memory (TMEM) comprising multiple storage banks, including: The acquisition module is used to acquire memory access requests for tensor data; An execution module is configured to selectively perform at least one of an address configuration operation and an access scheduling operation based on the memory access characteristics indicated by the memory access request. The address configuration operation is used to: distribute the storage space of the tensor data according to the number of Banks required by the tensor data; and / or convert the logical address of the tensor data into a physical Bank address so that the physical Bank addresses of different memory access requests are distributed on different Banks; The access scheduling operation is used to: when multiple memory access requests compete for the same Bank, temporarily delay at least some of the requests, and / or decompose a memory access request into multiple sub-requests and process them in parallel with the computation operation.

[0016] The tensor memory management method and apparatus described above are applied to a tensor memory memory (TMEM) containing multiple storage banks. When a memory access request for tensor data is received, at least one of an address configuration operation and an access scheduling operation is selectively executed based on the memory access characteristics indicated by the request. The address configuration operation is used to: distribute the storage space of the tensor data according to the number of banks required by the tensor data; and / or convert the logical address of the tensor data into a physical bank address, so that the physical bank addresses of different memory access requests are distributed across different banks. The access scheduling operation is used to: when multiple memory access requests compete for the same bank, perform temporary delay processing on at least some of the requests; and / or decompose a memory access request into multiple sub-requests and process them in parallel with computational operations. This disclosure, by using address configuration and access scheduling as optional and combinable technical means, can adaptively address different efficiency bottlenecks, avoid the problem of a single optimization scheme failing in other scenarios, and significantly improve the actual memory access bandwidth utilization of TMEM under different deep learning operators.

[0017] Other advantages of this disclosure will be explained in more detail in conjunction with the following description and accompanying drawings.

[0018] It should be understood that the above description is merely an overview of the technical solutions disclosed herein, so as to provide a general understanding of the technical means of this disclosure and to enable implementation in accordance with the contents of the specification. To make the above and other objects, features, and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. The accompanying drawings are incorporated in and constitute a part of this specification. These drawings illustrate embodiments conforming to this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure. It should be understood that the drawings only illustrate certain embodiments of this disclosure and should not be considered as a limitation on the scope of protection. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. Furthermore, the same reference numerals denote the same components throughout the drawings. In the drawings: Figure 1 A flowchart of a tensor storage management method provided by an embodiment of this disclosure is shown; Figure 2 This diagram illustrates tensor storage with specified offsets and number of banks in the tensor storage management method provided in this embodiment of the present disclosure. Figure 3This diagram illustrates the mapping from the M32 logical row to the M8 execution unit based on address substitution in the tensor storage management method provided in this embodiment of the present disclosure.

[0020] Figures 4(a) to 4(b) show schematic diagrams of the traditional full-quantity pingpong buffer principle; Figures 5(a) to 5(e) show schematic diagrams of the fragmented pipelined caching mechanism in the tensor storage management method provided in the embodiments of this disclosure; Figure 6 A schematic diagram of a tensor storage management device provided in an embodiment of this disclosure is shown; Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0021] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0022] In the description of embodiments disclosed herein, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of the disclosed features, figures, steps, behaviors, components, portions or combinations thereof in this specification, and do not exclude the possibility of the presence of one or more other features, figures, steps, behaviors, components, portions or combinations thereof.

[0023] Unless otherwise stated, " / " means "or". For example, A / B can mean A or B. In this article, "and / or" is merely a way of describing the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A alone, A and B at the same time, and B alone.

[0024] The terms "first," "second," etc., are used only for ease of description to distinguish identical or similar technical features and should not be construed as indicating or implying the relative importance or number of these technical features. Therefore, a feature defined by "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, the term "multiple" means two or more.

[0025] Research has revealed that in practical engineering applications, the actual memory access efficiency of TMEM is often far lower than its theoretical bandwidth. The root cause of this efficiency loss is not a single factor, but rather a series of interconnected systemic problems, including: When multiple memory access requests simultaneously hit the same bank, the requests must be processed serially, leading to a sharp drop in effective bandwidth. Existing solutions employ static address mapping, such as the Swizzling technique in Compute Unified Device Architecture (CUDA), which distributes logical addresses to different banks through predefined address transformation rules. However, static solutions struggle to adapt to the dynamically changing tensor shapes, strides, and merging access patterns in deep learning operators, and conflicts are difficult to completely avoid when multiple tensors of different sizes and dimensions reside in the TMEM simultaneously.

[0026] In a typical in-situ accumulation computation D = A × B + D, matrix D serves as both the input (read) and output (write-back) object. When read and write requests to the same bank occur concurrently, traditional read-before-write or double-buffered schemes either introduce pipeline bubbles or require double the storage space, resulting in decreased area efficiency.

[0027] In large-dimensional matrix block computation, the write-back of the output matrix or the prefetching of the input matrix for the next layer typically requires waiting for the entire tile computation to complete before it can begin. During this period, the computation unit is idle, waiting for memory access to complete, causing computation and memory access to be serialized. The traditional pingpong buffer scheme achieves a certain degree of overlap through double buffering, but it requires twice the storage area. When on-chip storage resources are limited, this overhead restricts the choice of tile size, thus affecting the overall computational efficiency.

[0028] In modern tensor processors, the logical organization of TMEM (Tension Metrics and Memory Access Modules) is typically in large-granularity rows (e.g., M=32) to facilitate software programming and data management. However, the actual memory access granularity of hardware execution units is often smaller (e.g., M=8). If linear address mapping is used, four consecutive small-granularity memory access instructions will cycle through the same physical bank set, causing the pipeline to pause every four cycles, severely reducing bandwidth utilization. While existing Swizzle technology can partially alleviate this, its design goal is mostly general address scattering and it is not optimized for this specific granularity mismatch scenario.

[0029] In summary, existing technologies lack a systematic solution that can collaboratively optimize the actual memory access efficiency of tensor memories from multiple levels, including address mapping, resource allocation, conflict resolution, and latency hiding. Most solutions only address a single bottleneck while ignoring the coupled effects of other factors, resulting in limited overall efficiency improvements.

[0030] To at least partially address one or more of the aforementioned problems and other potential issues, this disclosure provides at least one tensor memory management scheme that comprehensively considers bank conflict avoidance, read / write contention resolution, memory access latency concealment, and granularity matching optimization, thereby significantly improving the actual memory access bandwidth utilization of TMEM, reducing pipeline stalls, and minimizing area overhead.

[0031] To facilitate understanding of this embodiment, a tensor storage management method disclosed in this disclosure will first be described in detail. The execution subject of the tensor storage management method provided in this disclosure is generally an electronic device with certain computing capabilities, such as a tensor data processor or other processing device. In some possible implementations, the tensor storage management method can be implemented by the processor calling computer-readable instructions stored in the memory.

[0032] See Figure 1 The flowchart illustrates a tensor storage management method provided in an embodiment of this disclosure, the method comprising the following steps S101-S102: S101: Memory access request to obtain tensor data; S102: Based on the memory access characteristics indicated by the memory access request, selectively perform at least one of the address configuration operation and the access scheduling operation; The address configuration operation is used to: distribute the storage space of the tensor data according to the number of banks required by the tensor data; and / or to convert the logical address of the tensor data into a physical bank address so that the physical bank addresses of different memory access requests are distributed on different banks. Access scheduling operations are used to: temporarily delay at least some of the requests when multiple memory access requests compete for the same Bank, and / or decompose a memory access request into multiple sub-requests and process them in parallel with computational operations.

[0033] According to the embodiments of this disclosure, the address configuration operation and / or access scheduling operation are adaptively selected to be performed based on the memory access characteristics indicated by the memory access request.

[0034] Specifically, when TMEM receives a memory access request for tensor data, it first parses the characteristic information carried by the request, such as: the target memory access bandwidth requirement of the tensor data (how many banks are needed in parallel to meet its throughput requirements), the access address pattern (whether it is a linear continuous address row offset distribution), whether there is a possibility of concurrent reading and writing to the same address (such as the D matrix in in-situ accumulation calculation), and the matching status of the memory access granularity of the hardware execution unit with the TMEM logic row granularity, etc.

[0035] Based on these characteristics, determine which of the following types of operations(s) needs to be performed: If the feature shows that different tensor data have significantly different bandwidth requirements, then select the "Distribute according to the number of Banks" method in the address configuration operation.

[0036] If the feature indicates a mismatch between the access granularity and the TMEM logical row granularity (e.g., M32 logical row but the hardware can only read M8 at a time), then select the "address translation" method in the address configuration operation.

[0037] If the characteristic is that there is concurrent read and write competition for the same Bank (e.g., in-situ accumulation of matrix D), then the "temporary storage delay processing" method in the access scheduling operation is selected.

[0038] If the characteristic is that the write-back delay is long, causing the computing unit to idle, then the "decompose parallel processing" method in the access scheduling operation is selected.

[0039] All of the above operations can be performed individually or in any combination. For example, in the scenario of matrix in-situ accumulation, address configuration (to avoid conflicts within the Bank of the D matrix) and access scheduling (to reduce read / write concurrency and hide write-back latency) are often required simultaneously.

[0040] Through the aforementioned selective execution mechanism, TMEM can flexibly select the most suitable optimization method according to the actual efficiency bottlenecks it faces, thereby achieving actual memory access bandwidth close to the theoretical peak under different deep learning operators (convolution, GEMM, etc.), effectively avoiding the problem of a single optimization scheme failing in other scenarios.

[0041] The following sections will discuss different implementation methods using several specific examples: Example 1 The embodiments of this disclosure can distribute the storage space of tensor data through the following steps to achieve flexible bank allocation based on bandwidth requirements: Step 1: Obtain the offset parameter and the number of banks parameter K. The value of the number of banks parameter K corresponds to the target memory access bandwidth requirement of the tensor data, and the offset parameter is used to specify the starting storage location of the tensor data in TMEM. Step 2: Based on the offset parameter and the number of banks parameter K, allocate the storage space of the tensor data to K banks in an interleaved manner.

[0042] like Figure 2As shown, TMEM contains 4 banks (Bank0 to Bank3), each providing independent bandwidth capacity. Assuming the target memory access bandwidth requirement of tensor data A (e.g., matrix A) is the parallel capability of 2 banks, the number of banks is determined to be K=2, and the offset parameter is set to offset=1 (in units of bank capacity). Based on these two parameters, the storage space for data A starts from Bank1 and is interleaved between Bank1 and Bank2 in address order.

[0043] Specifically, consecutive address units 0 are mapped to Bank1, address 1 to Bank2, address 2 to Bank1, address 3 to Bank2, and so on. In this way, linear access to data A will not cause Bank collisions, and the number of Banks occupied will just meet the bandwidth requirements. The unoccupied Bank0 and Bank3 can be reserved for other tensor data.

[0044] Furthermore, the offset parameter can be set in lines or bytes within the Bank for finer control over the starting position. By setting different offsets for different tensor data, they can be placed in different regions of the Bank space, achieving physical isolation and avoiding conflicts.

[0045] In this way, Bank resources can be flexibly allocated according to bandwidth demand, avoiding the waste of Bank resources for low-bandwidth data and ensuring that high-bandwidth data obtains sufficient parallelism; address interleaving ensures that linear access is free of Bank conflicts; and offsets achieve the isolation of different tensors in the Bank space.

[0046] In practical applications, if the offset parameter and the number of banks are not specified, the bank allocation method will fall back to either global interleaving or non-interleaving. Global interleaving means that the space allocated by the instruction is interleaved across all banks, while non-interleaving means that the allocated space is stored sequentially on the corresponding banks to accommodate different bank allocation methods.

[0047] Example 2 The embodiments of this disclosure can convert the logical address of tensor data into a physical bank address through the following steps to achieve granular matching based on address permutation: Step 1: Perform an XOR operation on the row index bits and column index bits in the logical address of the tensor data using an XOR mask to generate a non-linear physical bank index. Step 2: Define the logical row with the first granularity as the scheduling envelope. The scheduling envelope contains multiple data blocks with the second granularity. When the hardware execution unit accesses the scheduling envelope with the second granularity, the XOR operation makes different data blocks within the same scheduling envelope map to different physical Bank groups.

[0048] In a typical configuration, the TMEM logical row size is 32 data elements (i.e., first granularity size = 32), while the hardware execution unit (e.g., the memory access interface of a tensor processor) can only process 8 elements at a time (second granularity size = 8). If linear address mapping is used, four consecutive M8 memory access instructions will access the same physical bank group, causing a pipeline stall every four clock cycles.

[0049] Based on this, the embodiments of this disclosure employ an XOR mask to perform an XOR operation on the row index bits and column index bits in the logical address to generate a non-linear physical bank index. Specifically, let the row address be R and the column address (offset within the row) be C, then the physical bank index = (C XOR R) mod the total number of banks. When the first M8 instruction (C=0~7) is executed, the bank index is 0~7 XOR R; when the second M8 instruction (C=8~15) is executed, the bank index is 8~15 XOR R; and so on. By reasonably designing the XOR mask (for example, taking the lower 2 bits of C and XORing them with the lower 2 bits of R), four consecutive M8 instructions can be mapped to four different physical bank groups: Bank0, Bank1, Bank2, and Bank3, respectively.

[0050] In this embodiment, a logical row of size 32 is defined as a scheduling envelope, which contains four data blocks of size 8. The XOR operation ensures that different data blocks within the same scheduling envelope are mapped to different physical bank groups. For example... Figure 3 As shown, the storage of the M32 matrix in TMEM allows the four data blocks to be read simultaneously when read at the M8 granularity, as they are located in different banks, thus maintaining the maximum bandwidth of TMEM.

[0051] Thus, the embodiments of this disclosure specifically address the periodic Bank conflicts between large-granularity logic rows and small-granularity execution units. Due to the spatially interleaved distribution of physical addresses, consecutive M8 memory access requests that might logically cause collisions are transformed into concurrent accesses to different Bank groups or different physical arbitration domains at the physical level. This allows the TMEM bus controller to maintain 32 Banks simultaneously in an active working state in each cycle, even when the hardware execution granularity is small, so that consecutive small-granularity memory access instructions hit different Bank groups, and the effective bandwidth can be improved by about 4 times compared to linear mapping.

[0052] It should be noted that the first granularity size of 32 and the second granularity size of 8 are only specific examples. In practical applications, different granularity sizes and proportions can be determined according to the actual application requirements. No specific restrictions are imposed here. Similarly, no specific restrictions are imposed here on the different mapping groups resulting from different granularity sizes and proportions.

[0053] Example 3 The embodiments of this disclosure can perform temporary delay processing on at least a portion of the requests through the following steps to achieve concurrent read / write operations with independent write-back cached by bank: Step 1: Configure a write-back cache for each Bank of TMEM; Step 2: When read and write requests occur concurrently for the same Bank, the write data is temporarily stored in the corresponding write-back cache without blocking the read request. The TMEM controller monitors the read request queue of each Bank. When it detects that the same Bank is in an idle period without read requests, the write data in the corresponding write-back cache is written back to that Bank.

[0054] The write-back cache is either a register file or a static random access memory (SRAM) with a capacity smaller than the corresponding bank capacity. Write-back operations have a lower priority than read requests and are initiated after idle time is detected.

[0055] In practical applications, to address the read-write conflict caused by the simultaneous reading (as input) and writing back (as output) of matrix D in the in-situ accumulation calculation D = A × B + D, this disclosure embodiment configures a dedicated writeback buffer for each storage bank (Bank0 to Bank3) of TMEM, forming a one-to-one correspondence between "Bank-Buffer".

[0056] During computation, when the computation result needs to be written back to TMEM, if there is no Bank conflict, it can be written directly back to the Bank; otherwise, it is first written to the write-back cache. Meanwhile, read requests can read data from the Bank normally without any blocking. The TMEM controller continuously monitors the read request queue of each Bank. When it detects that the read request queue of a Bank is empty (i.e., in an idle period), the controller writes the temporary data in the corresponding write-back cache back to that Bank.

[0057] Write-back caches can be implemented using register files or small-capacity static random access memory (SRAM), with a capacity much smaller than that of the corresponding bank (e.g., a depth of only 2 to 4 entries), which results in lower hardware overhead.

[0058] This embodiment employs a bank-based independent writeback buffer hardware architecture. Each storage bank of the TMEM is configured with a dedicated, small-capacity writeback buffer, forming a one-to-one "bank-buffer" structure. This decouples the physical paths of write and read data, preventing read / write conflicts within the same bank from a hardware perspective. A read / write bandwidth decoupling and asynchronous write-back mechanism are used, ensuring that the number of banks naturally guarantees sufficient read bandwidth for computational needs. When bank conflicts occur, the data is first written to the corresponding writeback buffer for temporary storage, without occupying read ports or blocking read requests, thus achieving continuous, uninterrupted reading and non-preemptive writing. An idle-cycle insertion write-back scheduling strategy is used. The storage controller (i.e., the TMEM controller) monitors the idle cycles of each bank and, during gaps without read access, writes the D matrix result in the writeback buffer back to the original TMEM address with low priority, without conflict, and without waiting, inserting bubbles, or altering the computational pipeline timing. Its in-situ accumulation has no additional storage overhead, does not require doubling the storage area for D, and eliminates the need for data migration and address remapping. It supports in-situ updates, significantly reducing on-chip storage usage and data transport power consumption.

[0059] In this way, bank conflicts are completely eliminated, read and write paths are separated, there is no concurrent competition within the same bank, and the conflict rate is reduced to 0; bandwidth utilization is maximized, read bandwidth runs at full capacity, and write operations do not preempt or block; the computation pipeline is uninterrupted, without waiting or pauses, and the effective computing power of tensor computation units is improved; area and power consumption are optimized, requiring only a small-capacity buffer, with extremely low area cost, significantly reducing data transport power consumption; it has greater versatility, is compatible with various matrix multiplication and accumulation, in-situ update, and recursive accumulation operators, and is adapted to various architectures such as AI accelerators / NPUs / GPUs.

[0060] Example 4 The embodiments of this disclosure can decompose a memory access request into multiple sub-requests and process them in parallel with computational operations through the following steps, so as to achieve hidden write-back latency in fragmented pipelined caching: Step 1: Divide the output matrix corresponding to the memory access request into multiple logical slices. The size of each logical slice is configured such that the computation time can cover the maximum memory access latency required from the completion of slice computation to the completion of write-back. Step 2: Once the calculation result of a logical shard is generated, the write-back operation for that shard is initiated without waiting for the calculation of other shards to be completed, so that the write-back operation of that shard is executed in parallel with the calculation operation of subsequent shards.

[0061] The number of logical slices is N, where N≥2; the size of the logical slices is set through configurable registers, and the slice size is dynamically adjusted according to the access delay parameters of TMEM and the calculation delay parameters of the computing unit.

[0062] To address the issue of write-back operation delays causing idle computation units in large-dimensional matrix block computation, this embodiment divides the output matrix to be written back into multiple logical slices. After the matrix multiply-accumulate (MMA) computation unit generates partial results, asynchronous write-back is initiated immediately. If there is an initial D matrix input, the writing of a new matrix D can be initiated after a certain logical slice is output, without having to wait for the entire slice computation to complete.

[0063] Before discussing the improvement scheme provided by the embodiments of this disclosure, we first clarify the working principle of the existing full pingpong buffer. As shown in Figure 5, assuming that a full amount of data is 4KB, the complete pingpong buffer uses two 4KB buffers to perform pingpong operations. Assuming that it starts with writing ping and reading pong, as shown in Figure 4(a), it will then become writing pong and reading ping, as shown in Figure 4(b), alternating between the two.

[0064] To reduce chip area, this embodiment does not use the traditional "full pingpong buffer," but instead establishes a segment-level pipelined buffer. When a portion of the accumulated result in the register or TMEM reaches a preset physical segment size (e.g., 1 / 4 or 1 / 8 of the D matrix space), an access request is immediately initiated via TMEM read / write instructions.

[0065] Here, we still assume the output matrix D is 4KB in size, which is divided into four logical partitions, each 1KB. The computation unit processes partition 0, partition 1, partition 2, and partition 3 sequentially. Whenever the computation result of one partition is generated, a write-back operation is immediately initiated for that partition without waiting for the computation of other partitions to complete. Thus, the write-back operation of partition 0 is executed in parallel with the computation operation of partition 1, the write-back operation of partition 1 is executed in parallel with the computation of partition 2, and so on.

[0066] Examples are shown in Figures 5(a) to 5(e). In Figure 5(a), fragment 0 is written when reading fragments 1 / 2 / 3 / 4; in Figure 5(b), fragment 1 is written when reading fragments 0 / 2 / 3 / 4; in Figure 5(c), fragment 2 is written when reading fragments 0 / 1 / 3 / 4; in Figure 5(d), fragment 3 is written when reading fragments 0 / 1 / 2 / 4; and in Figure 5(e), fragment 4 is written when reading fragments 0 / 1 / 2 / 3. There is always a write space to receive the prefetched data. The size of this space (i.e., the fragment size) needs to be able to cover the data access latency. That is, a full amount of data is 4KB, divided into 4 fragments, each fragment is 1KB. Only an additional 1KB of storage space is needed to hide the data access latency.

[0067] The key is to configure the size of each logical slice so that its computation time covers the maximum memory access latency required from the completion of slice computation to the completion of write-back. If the time required to compute a slice is T_calc, and the maximum latency required to write back a slice is T_write, then as long as T_calc ≥ T_write, the write-back latency can be completely hidden. The slice size can be set through configurable registers and dynamically adjusted according to the access latency parameters of TMEM and the computation latency parameters of the computation unit.

[0068] Thus, compared to a full pingpong buffer that occupies twice the space, the sharding pipeline mechanism not only reduces storage space requirements by about 50%, enabling a single computing core to support larger-dimensional block operations, but also distributes the originally instantaneous high-bandwidth pulse pressure throughout the entire computing cycle through a "peak shaving and valley smoothing" strategy. This significantly eliminates pipeline suspension cycles caused by physical bank conflicts, and while reducing the risk of register overflow, it completely hides the latency of output write-back or input loading, resulting in a significant improvement in effective computing power. Its configurable sharding parameters allow the solution to adapt to different hardware platforms and different operators.

[0069] Example 5 In practical deep learning operators, multiple efficiency bottlenecks often coexist. This disclosure demonstrates how to combine these various techniques to maximize the actual memory access efficiency of TMEM. Specifically, this embodiment reflects the "multiple combinations" scenario within the phrase "selectively performing at least one of address configuration operations and access scheduling operations."

[0070] Taking the matrix in-situ accumulation D = A × B + D as an example: First, using the method of Example 1, allocate an appropriate number of banks (e.g., K=4) to the D matrix according to its target bandwidth requirements, and set an offset to isolate it from other matrices.

[0071] Meanwhile, the address replacement method of Example 2 is used to eliminate internal bank conflicts that may be caused by granularity mismatch.

[0072] During the calculation process, the write-back caching mechanism of Example 3 is used to mitigate concurrent conflicts between writing to and from the same Bank.

[0073] Meanwhile, using the fragmented pipeline mechanism of Example 4, the D matrix is ​​divided into multiple fragments to achieve pipeline overlap between write-back and computation.

[0074] The four technologies are independent of each other, do not interfere with each other, and can work together. Address configuration avoids bank conflicts within the D matrix; address substitution eliminates granularity mismatch; write-back caching decouples read and write paths; and fragmented pipelines hide write-back latency. Ultimately, TMEM's actual memory access bandwidth utilization is close to the theoretical peak, and it does not require an additional pingpong storage area, achieving optimal area efficiency.

[0075] In the description of this specification, references to terms such as "some possible implementations," "some implementations," "example," "specific example," or "some examples" indicate that a specific feature, structure, material, or characteristic described in connection with that implementation or example is included in at least one implementation or example of this disclosure, and the aforementioned terms do not necessarily refer to the same implementation or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more implementations or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different implementations or examples described in this specification, as well as the features of different implementations or examples.

[0076] Regarding the method flowcharts of embodiments of this disclosure, certain operations are described as different steps performed in a certain order. Such flowcharts are illustrative and not restrictive. Some steps described herein may be grouped together and performed in a single operation, or some steps may be divided into multiple sub-steps, and some steps may be performed in an order different from that shown herein. The various steps shown in the flowcharts may be implemented in any way by any circuit structure and / or tangible mechanism (e.g., software running on a computer device, hardware (e.g., logic functions implemented by a processor or chip), and / or any combination thereof).

[0077] Those skilled in the art will understand that in the methods described in the above specific embodiments, the order in which the steps are written does not imply a strict execution order, and the specific execution order of each step should be determined by its function and possible internal logic.

[0078] Based on the same inventive concept, this disclosure also provides a tensor storage management device corresponding to the tensor storage management method. Since the principle of the device in this disclosure for solving the problem is similar to that of the tensor storage management method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0079] Reference Figure 6 The diagram shown is a schematic of a tensor storage management device provided in an embodiment of this disclosure. The device includes: an acquisition module 201 and an execution module 202; wherein, Module 201 is used to obtain memory access requests for tensor data; The execution module 202 is configured to selectively execute at least one of an address configuration operation and an access scheduling operation based on the memory access characteristics indicated by the memory access request. The address configuration operation is used to: distribute the storage space of the tensor data according to the number of banks required by the tensor data; and / or to convert the logical address of the tensor data into a physical bank address so that the physical bank addresses of different memory access requests are distributed on different banks. Access scheduling operations are used to: temporarily delay at least some of the requests when multiple memory access requests compete for the same Bank, and / or decompose a memory access request into multiple sub-requests and process them in parallel with computational operations.

[0080] The aforementioned tensor memory management device, applied to a tensor memory memory (TMEM) containing multiple storage banks, selectively executes at least one of address configuration and access scheduling operations when a memory access request for tensor data is received, based on the memory access characteristics indicated by the request. The address configuration operation distributes the storage space of the tensor data according to the number of banks required by the tensor data, and / or converts the logical address of the tensor data into a physical bank address, so that the physical bank addresses of different memory access requests are distributed across different banks. The access scheduling operation temporarily delays at least some of the requests when multiple memory access requests compete for the same bank, and / or decomposes a memory access request into multiple sub-requests and processes them in parallel with computational operations. This disclosure, by using address configuration and access scheduling as optional and combinable technical means, can adaptively address different efficiency bottlenecks, avoid the failure of a single optimization scheme in other scenarios, and significantly improve the actual memory access bandwidth utilization of TMEM under different deep learning operators.

[0081] In one possible implementation, execution module 202 is specifically configured to distribute the storage space of tensor data according to the following steps: Obtain the offset parameter and the number of banks parameter K. The value of the number of banks parameter K corresponds to the target memory access bandwidth requirement of the tensor data, and the offset parameter is used to specify the starting storage location of the tensor data in TMEM. Based on the offset parameter and the number of banks parameter K, the storage space of the tensor data is allocated to K banks in an interleaved manner.

[0082] In one possible implementation, the execution module 202 is further configured to: fall back to a global interleaving or non-interleaving bank allocation method if no offset parameter and bank number parameter are specified. Global interleaving means that the space allocated by the instruction is interleaved on all banks, while non-interleaving means that the allocated space is stored sequentially on the corresponding banks.

[0083] In one possible implementation, execution module 202 is specifically configured to convert the logical address of tensor data into a physical bank address according to the following steps: A non-linear physical bank index is generated by performing an XOR operation on the row index bits and column index bits in the logical address of the tensor data using an XOR mask. Here, a logical row with a first granularity is defined as a scheduling envelope, which contains multiple data blocks with a second granularity. When the hardware execution unit accesses the scheduling envelope with the second granularity, the XOR operation causes different data blocks within the same scheduling envelope to be mapped to different physical Bank groups.

[0084] In one possible implementation, the first granularity size is 32 and the second granularity size is 8; a scheduling envelope contains four data blocks, each mapped to one of four different physical bank groups.

[0085] In one possible implementation, execution module 202 is specifically configured to perform temporary delay processing on at least a portion of the requests according to the following steps: Configure a write-back cache for each Bank of TMEM; When read and write requests occur concurrently for the same Bank, the write data is temporarily stored in the corresponding write-back cache without blocking the read request. The TMEM controller monitors the read request queue of each Bank, and when it detects that the same Bank is in an idle period without read requests, it writes the write data in the corresponding write-back cache back to that Bank.

[0086] In one possible implementation, the write-back cache is a register file or a static random access memory (SRAM) with a capacity smaller than the corresponding bank capacity; the write-back operation has a lower priority than read requests and is initiated after idle time is detected.

[0087] In one possible implementation, execution module 202 is specifically configured to decompose a memory access request into multiple sub-requests and process them in parallel with computational operations according to the following steps: The output matrix corresponding to the memory access request is divided into multiple logical slices. The size of each logical slice is configured such that the computation time can cover the maximum memory access latency required from the completion of slice computation to the completion of write-back. Once the computation result of a logical shard is generated, a write-back operation is initiated for that shard without waiting for other shards to complete their computations, allowing the write-back operation of that shard to be executed in parallel with the computation operations of subsequent shards.

[0088] In one possible implementation, the number of logical slices is N, where N≥2; the execution module 202 is further configured to: The size of the logical slice can be set through configurable registers, and the slice size can be dynamically adjusted according to the access delay parameters of TMEM and the calculation delay parameters of the computing unit.

[0089] It should be noted that the apparatus in this embodiment can implement the various processes of the aforementioned method and achieve the same effects and functions, which will not be elaborated here.

[0090] This disclosure also provides an electronic device, such as... Figure 7 The diagram shown is a schematic representation of an electronic device structure provided in this embodiment of the present disclosure, including: a processor 301, a memory 302, and a bus 303. The memory 302 stores machine-readable instructions executable by the processor 301 (e.g., ...). Figure 6 The device acquires the corresponding execution instructions (such as the acquisition module 201 and the execution module 202) in the device. When the electronic device is running, the processor 301 and the memory 302 communicate via the bus 303. When the machine-readable instructions are executed by the processor 301, the following processing is performed: A memory access request to retrieve tensor data; Based on the memory access characteristics indicated by the memory access request, selectively perform at least one of the address configuration operation and the access scheduling operation; The address configuration operation is used to: distribute the storage space of the tensor data according to the number of banks required by the tensor data; and / or to convert the logical address of the tensor data into a physical bank address so that the physical bank addresses of different memory access requests are distributed on different banks. Access scheduling operations are used to: temporarily delay at least some of the requests when multiple memory access requests compete for the same Bank, and / or decompose a memory access request into multiple sub-requests and process them in parallel with computational operations.

[0091] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the tensor storage management method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0092] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the tensor storage management method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0093] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0094] The various embodiments in this disclosure are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments. In particular, the description of the apparatus, device, and computer-readable storage medium embodiments is simplified because they are basically similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments.

[0095] The apparatus, device, and computer-readable storage medium provided in this disclosure correspond one-to-one with the method. Therefore, the apparatus, device, and computer-readable storage medium also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the apparatus, device, and computer-readable storage medium will not be repeated here.

[0096] Those skilled in the art will understand that embodiments of this disclosure can be implemented as methods and apparatus (devices or systems), or as computer-readable storage media. Therefore, this disclosure can be implemented entirely in hardware, entirely in software, or in a combination of software and hardware. Furthermore, this disclosure can be implemented as a computer-readable storage medium on one or more computer-readable storage media containing computer-usable program code (including, but not limited to, disk storage, read-only optical disc storage (CD-ROM), optical storage, etc.).

[0097] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices or systems), and computer-readable storage media according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article including instruction means, wherein the instruction means implement the functions specified in one or more flowcharts and / or one or more blocks in a block diagram.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0100] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0101] Memory can include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0102] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory, read-only memory, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. Furthermore, although the operations of the methods of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally, certain steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple sub-steps.

[0103] While the spirit and principles of this disclosure have been described above with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A tensor storage management method, applied to a tensor memory (TMEM) containing multiple storage banks, characterized in that, include: A memory access request to retrieve tensor data; Based on the memory access characteristics indicated by the memory access request, at least one of the address configuration operation and the access scheduling operation is selectively performed; The address configuration operation is used to: distribute the storage space of the tensor data according to the number of Banks required by the tensor data; and / or convert the logical address of the tensor data into a physical Bank address so that the physical Bank addresses of different memory access requests are distributed on different Banks; The access scheduling operation is used to: when multiple memory access requests compete for the same Bank, temporarily delay at least some of the requests, and / or decompose a memory access request into multiple sub-requests and process them in parallel with the computation operation.

2. The method according to claim 1, characterized in that, The distribution of the storage space for the tensor data includes: Obtain the offset parameter and the number of banks parameter K, wherein the value of the number of banks parameter K corresponds to the target memory access bandwidth requirement of the tensor data, and the offset parameter is used to specify the starting storage location of the tensor data in the TMEM; Based on the offset parameter and the number of banks parameter K, the storage space of the tensor data is allocated to K banks in an interleaved manner.

3. The method according to claim 2, characterized in that, The method further includes: If the offset parameter and the number of banks are not specified, the system will fall back to the global interleaving or non-interleaving bank allocation method. Global interleaving means that the space allocated by the instruction is interleaved across all banks, while non-interleaving means that the allocated space is stored sequentially into the corresponding banks.

4. The method according to claim 1, characterized in that, The step of converting the logical address of the tensor data into a physical bank address includes: A non-linear physical Bank index is generated by performing an XOR operation on the row index bits and column index bits in the logical address of the tensor data using an XOR mask. Here, a logical row with a first granularity is defined as a scheduling envelope, which contains multiple data blocks with a second granularity. When the hardware execution unit accesses the scheduling envelope with the second granularity, the XOR operation causes different data blocks within the same scheduling envelope to be mapped to different physical Bank groups.

5. The method according to claim 4, characterized in that, The first granularity size is 32, and the second granularity size is 8; one of the scheduling envelopes contains four data blocks, which are respectively mapped to four different physical bank groups.

6. The method according to claim 1, characterized in that, The provisional delay of at least some requests includes: Configure a write-back cache for each Bank of TMEM; When read and write requests occur concurrently for the same Bank, the write data is temporarily stored in the corresponding write-back cache without blocking the read request. The TMEM controller monitors the read request queue of each Bank, and when it detects that the same Bank is in an idle period without read requests, the write data in the corresponding write-back cache is written back to that Bank.

7. The method according to claim 6, characterized in that, The write-back cache is a register file or a static random access memory (SRAM) with a capacity smaller than the corresponding bank capacity; the write-back operation has a lower priority than the read request and is initiated after idle time is detected.

8. The method according to claim 1, characterized in that, The process of decomposing a memory access request into multiple sub-requests and processing them in parallel with computational operations includes: The output matrix corresponding to the memory access request is divided into multiple logical segments, and the size of each logical segment is configured such that the computation time can cover the maximum memory access latency required from the completion of segment computation to the completion of write-back. Once the computation result of a logical shard is generated, a write-back operation is initiated for that shard without waiting for other shards to complete their computations, allowing the write-back operation of that shard to be executed in parallel with the computation operations of subsequent shards.

9. The method according to claim 8, characterized in that, The number of logical partitions is N, where N≥2; the method further includes: The size of the logical slice is set by a configurable register, and the slice size is dynamically adjusted according to the access delay parameter of TMEM and the calculation delay parameter of the computing unit.

10. A tensor memory management device, applied to a tensor memory (TMEM) comprising multiple storage banks, characterized in that, include: The acquisition module is used to acquire memory access requests for tensor data; An execution module is configured to selectively perform at least one of an address configuration operation and an access scheduling operation based on the memory access characteristics indicated by the memory access request. The address configuration operation is used to: distribute the storage space of the tensor data according to the number of Banks required by the tensor data; and / or convert the logical address of the tensor data into a physical Bank address so that the physical Bank addresses of different memory access requests are distributed on different Banks; The access scheduling operation is used to: when multiple memory access requests compete for the same Bank, temporarily delay at least some of the requests, and / or decompose a memory access request into multiple sub-requests and process them in parallel with the computation operation.

Images