Tensor transpose processor

Abstract

The invention relates to a tensor transpose processor. A processor is designed to optimize memory bandwidth utilization for tensor transpose in machine learning. An example processor includes an input tensor shift buffer, a temporary buffer, and an output tensor shift buffer. The input tensor shift buffer reads the input tensor from the input memory and performs a plurality of input tensor shift cycles. The shifted tensor data is then written to a temporary buffer. The output tensor shift buffer reads the shifted tensor data from the temporary buffer and performs a plurality of output tensor shift cycles. Finally, the result is written to an output memory. This configuration facilitates efficient processing and transformation of tensor data, thereby optimizing the computational process required for machine learning tasks.

Description

[0001] Cross Reference to Related Applications

[0002] This application is a continuation-in-part (CIP) of U.S. Patent Application No. 18 / 798,035, filed August 8, 2024, entitled “PROCESSOR, METHOD, AND SYSTEM FOR ACCELERATING TENSOR TRANSPOSE FOR MACHINE LEARNING,” the entirety of which is incorporated by reference herein. TECHNICAL FIELD

[0003] The present disclosure relates generally to hardware design for accelerating tensor transpose in machine learning, and more particularly, to a processor with a built-in engine for simplifying tensor transpose operations in neural networks (NNs). BACKGROUND

[0004] In the field of machine learning and deep learning, tensor transpose is critical for data manipulation, playing a key role in pre-processing, data augmentation, and adjusting data to fit neural network architectures. This process requires rearranging multi-dimensional tensor dimensions, which is crucial for improving the training and inference efficiency of models. However, current computing infrastructure, including central processing units (CPUs) and graphics processing units (GPUs), falls short in effectively handling the complex memory access and reorganization demands posed by tensor transpose.

[0005] This deficiency in hardware capabilities leads to significant inefficiencies, especially as tensor complexity and dimensions grow. Hardware struggles to proficiently handle the non-linear data access patterns required for tensor transpose, resulting in suboptimal cache memory utilization and increased memory bandwidth demands. Such inefficiencies not only slow down the computation process but also exacerbate the gap between existing hardware designs and the complex demands of contemporary machine learning tasks. SUMMARY

[0006] Various embodiments of the present specification can include processors and systems for improving memory access efficiency during tensor transpose operations.

[0007] In one aspect, the disclosure describes a processor for accelerating tensor transpose operations in machine learning. The processor can include: an input tensor shift buffer; a staging buffer; and an output tensor shift buffer. The input tensor shift buffer is configured to: read an input tensor from an input memory, perform a plurality of input tensor shift cycles on the input tensor read from the input memory, and write results of the plurality of input tensor shift cycles to the staging buffer. The output tensor shift buffer is configured to: read the results of the plurality of input tensor shift cycles from the staging buffer, perform a plurality of output tensor shift cycles on the results read from the staging buffer, and write results of the plurality of output tensor shift cycles to an output memory.

[0008] In some embodiments, the staging buffer includes a plurality of memory banks.

[0009] In some embodiments, the input tensor shift buffer is further configured to, during each cycle of the plurality of input tensor shift cycles: read a sub-tensor of the input tensor from the input memory, and write the sub-tensor of the input tensor across the plurality of memory banks of the staging buffer in a first shift direction.

[0010] In some embodiments, to read the sub-tensor of the input tensor, the input tensor shift buffer is further configured to read a row of data of the input tensor from the input memory, and to write the sub-tensor of the input tensor across the plurality of memory banks of the staging buffer in the first shift direction, the input tensor shift buffer is further configured to write the row of data read from the input tensor across the plurality of memory banks of the staging buffer in a column direction.

[0011] In some embodiments, the output tensor shift buffer is further configured to, during each cycle of the plurality of output tensor shift cycles: read a sub-tensor from the staging buffer, and write the sub-tensor to the output memory in a second shift direction.

[0012] In some embodiments, to read the sub-tensor from the staging buffer, the output tensor shift buffer is further configured to: read a row of data from the staging buffer; and write the row of data from the staging buffer by shifting the row of data in the second shift direction.

[0013] In some embodiments, a direction of the input tensor shift is opposite to a direction of the output tensor shift.

[0014] In some embodiments, the staging buffer includes a pair of buffers that are the same size.

[0015] In some embodiments, the pair of buffers includes a first buffer and a second buffer that support parallel processing, and the output tensor shift buffer is reading from the second buffer while the input tensor shift buffer is writing to the first buffer.

[0016] In some embodiments, the pair of buffers includes a first buffer and a second buffer, and after the input tensor shift buffer writes the first result obtained by performing multiple input tensor shift cycles on the first input tensor into the first buffer, the output tensor shift buffer begins to read the first result from the first buffer to perform multiple output tensor shift cycles; and while the output tensor shift buffer is reading from the first buffer, the input tensor shift buffer begins to write the second result obtained by performing multiple input tensor shift cycles on the second input tensor into the second buffer, so that the input tensor shift buffer and the output tensor shift buffer avoid idle time.

[0017] In another aspect, this disclosure describes a method for accelerating tensor transpose operations in machine learning. The method may include: performing a plurality of input tensor shift loops on an input tensor read from an input memory using an input tensor shift buffer; during each loop of the plurality of input tensor shift loops, writing the result of the input tensor shift loop to a temporary buffer using the input tensor shift buffer; performing a plurality of output tensor shift loops on the result read from the temporary buffer using an output tensor shift buffer; and during each loop of the plurality of output tensor shift loops, writing the result of the output tensor shift loop to an output memory using the output tensor shift buffer.

[0018] These and other features of the systems, methods, and non-transitory computer-readable media disclosed herein, as well as the operation and function of related structural elements, combinations of components, and economic benefits of manufacture, will become clearer upon consideration of the following description and appended claims with reference to the accompanying drawings (all drawings form a part of this specification, wherein like reference numerals denote corresponding parts in the respective drawings). However, it should be clearly understood that the drawings are for illustration and description only and are not intended to define the scope of the invention. Attached Figure Description

[0019] Figure 1A This shows an example of tensor transpose in machine learning.

[0020] Figure 1B This illustrates the inefficient use of memory bandwidth by existing processors for tensor transpose.

[0021] Figure 2 Exemplary architecture diagrams of tensor transpose processors according to various embodiments are shown.

[0022] Figure 3AAn exemplary tensor transpose using a tensor transpose processor is shown, in accordance with various embodiments.

[0023] Figure 3B An example load and store procedure to implement an internal transpose in a tensor transpose processor is shown, in accordance with various embodiments.

[0024] Figure 3C An example architecture diagram of an internal transpose engine is shown, in accordance with various embodiments.

[0025] Figure 3D A parallel processing pipeline in an internal transpose engine is shown, in accordance with various embodiments.

[0026] Figure 4 Another exemplary tensor transpose using a tensor transpose processor is shown, in accordance with various embodiments.

[0027] Figure 5A An example high-dimensional tensor representation is shown, in accordance with various embodiments. Figures 5B-5D Facilitating the description of

[0028] Figure 5B An exemplary tensor transpose using a tensor transpose processor when the contiguous units of a tensor are less than the memory bandwidth is shown, in accordance with various embodiments.

[0029] Figure 5C Another exemplary tensor transpose using a tensor transpose processor when the contiguous units of a tensor are less than the memory bandwidth is shown, in accordance with various embodiments.

[0030] Figure 5D An exemplary tensor transpose using a tensor transpose processor when the contiguous units of a tensor are equal to or greater than the memory bandwidth is shown, in accordance with various embodiments.

[0031] Figure 6A An exemplary method of tensor transposition using a tensor transpose processor is shown, in accordance with various embodiments.

[0032] Figure 6B Another exemplary method of tensor transposition using a tensor transpose processor is shown, in accordance with various embodiments.

[0033] Figure 7 An example computer system that can implement any of the embodiments described herein is shown. DETAILED DESCRIPTION

[0034] In machine learning, tensor operations such as transpose are crucial but are often hampered by inefficiencies in existing solutions. Traditional methods typically rely on sequential processing, where tensors are read, transposed, and written back element by element or row by row. This leads to significant latency and suboptimal performance, especially for large tensors. Single-buffer systems for intermediate storage can become bottlenecks, resulting in idle time and underutilization of processing resources. Furthermore, many methods lack sufficient parallelism to leverage modern hardware architectures that excel at parallel operations. Inefficient memory usage is another key drawback, as frequent and redundant memory accesses increase bandwidth consumption and contention, particularly problematic for high-dimensional tensors. Additionally, traditional methods often fail to consider directional constraints on tensor data, leading to complex and inefficient data movement patterns. These issues result in high latency and limited throughput, making the transpose process a bottleneck in machine learning pipelines. This disclosure describes an efficient hardware design and method to improve the performance of tensor transpose operations.

[0035] The concept of transpose is not limited to two-dimensional matrices but extends to tensors (i.e., multidimensional numerical arrays). Tensors are crucial in various scientific fields, including physics, computer science, and engineering. The dimensions of a tensor are often referred to as "axes" or "modes." While transposing a two-dimensional matrix involves flipping its main diagonal elements, transposing a tensor with dimensions exceeding two requires rearranging those dimensions, a process that can significantly alter the tensor's structure and the relationships between its elements.

[0036] The transpose of a high-dimensional tensor is defined by specifying the permutation of its dimensions. Figure 1A This shows an example of tensor transpose in machine learning. Figure 1A The multidimensional tensor shown has three dimensions, represented by (X, Y, Z), where X, Y, and Z represent the size of the tensor along each axis. In practice, each dimension X, Y, or Z can include more than one dimension, meaning that multidimensional tensors can be larger than... Figure 1A The three dimensions shown are more than that.

[0037] right Figure 1A Transposing a tensor may involve replacing these dimensions with (Z,X,Y), (Y,Z,X), or any other permutation, depending on the specific requirements of the operation being performed. This permutation changes how elements are indexed and accessed, effectively reorganizing the tensor's data.

[0038] Tensor transposition is of paramount importance in fields such as deep learning and computer graphics, where tensors are used to represent complex datasets, such as images, videos, and multidimensional signal data. In these applications, tensor transposition is crucial for: aligning data into the format desired by specific algorithms, optimizing memory layout for faster computation, or visualizing multidimensional data in a more interpretable way.

[0039] A common application involves transposing tensors representing images. In computer memory, images are typically stored as three-dimensional tensors, with dimensions corresponding to height, width, and color channels (e.g., RGB). Depending on processing requirements, this tensor may need to be transposed to align with the input format expected by image processing libraries or machine learning models (which may expect color channels as the first dimension rather than the last).

[0040] The flexibility of reordering tensor dimensions during transpose allows for diverse operations on multidimensional data, enabling more efficient data processing, analysis, and visualization. The specific way tensor dimensions are permuted during transpose depends on the application's needs, highlighting the adaptability and broad applicability of this operation across various scientific and engineering disciplines.

[0041] Figure 1B This illustrates the inefficiency of existing processors in utilizing memory bandwidth for tensor transposition. As previously mentioned, tensor transposition involves reordering multidimensional tensors widely used in machine learning applications. When transposing a tensor, traditional computing systems access memory by jumping to a specific source address to retrieve the desired data segment and then directly writing to the destination memory address. This process is inherently slow due to the non-sequential memory access patterns that occur when transposing tensor axes. In the worst case, this can result in only a single byte of valid data being retrieved in each memory read loop, leading to extremely low efficiency.

[0042] A simple way to address this inefficiency is to increase memory bandwidth, allowing the system to read as much data as possible in each loop. However, this solution doesn't scale well for different tensor transposes (which can involve a large number of axes, each potentially large in size). Because tensor data is pre-stored in memory, transposing based on different axes leads to a variety of unpredictable memory access patterns. This variation makes predicting the amount of vector data required for each transpose operation challenging. Unless the entire tensor can be loaded into the cache simultaneously (which is impractical for large tensors), inefficient reads are inevitable during each loop.

[0043] To illustrate, consider a scenario where a three-dimensional tensor [a, b, c] of size [64, 128, 512] bytes is stored such that each 512 bytes (the innermost dimension) is stored sequentially in memory, with a memory bandwidth of 512 bytes per cycle. If this tensor [a, b, c] is transposed to [b, a, c]—preserving the order of the innermost dimension (called the stationary axis) and transposing the two outer dimensions (a and b, called the transposed axes)—the memory access pattern remains relatively efficient, allowing 512 bytes of data to be moved per cycle. In this case, large, sequentially stored data segments can be read in blocks, maximizing throughput.

[0044] However, if the innermost dimension is not the largest, for example, a tensor [a,b,c] of size [64,128,64] bytes is transposed to [b,a,c] (i.e., swapping dimensions a and b), then the most efficient bandwidth per read loop drops to 64 bytes. This smaller bandwidth significantly reduces throughput compared to the first example (i.e., only 64 / 512 or 1 / 8 of the memory bandwidth is effectively utilized).

[0045] The problem worsens if the transpose involves the two innermost dimensions, such as transposing [a,b,c] of size [64,128,64] to [a,c,b] (i.e., swapping dimensions b and c). In this case, there are no large blocks of sequentially stored data available, and if these inner dimensions are swapped, the throughput could be reduced to just 1 byte per loop, representing a significant drop in efficiency.

[0046] Figure 1B The worst-case scenario described above is illustrated using a 3×3 tensor 110, where the height axis (H), width axis (W), and channel axis (C) are of equal size, and the channel dimension is typically the outermost, with the width dimension being the innermost; however, the height dimension could also be the innermost, with similar results. In practical applications, the channel dimension (C) is usually divided into multiple channel groups. Each channel group comprises multiple consecutive channels and is denoted as the channel group dimension. For example, there might be 128 channels, and every 16 channels are grouped together as a channel group. Typically, the channel group dimension is the innermost dimension, and data within the same channel group is stored sequentially in memory. However, for simplicity, we assume the channel group dimension is 1, thus making the W dimension the innermost dimension. The elements of tensor 110 are labeled to help explain the transpose process and memory storage.

[0047] Typically, in existing technologies, tensors are stored in memory 120, with the innermost dimension stored contiguously first, followed by the next innermost dimension, and so on. For example... Figure 1B As shown, the sequence (1,2,3) from channel 0, height 0 row is stored consecutively in memory 120, followed by (4,5,6), (7,8,9), etc.

[0048] If the transpose of tensor 110 involves swapping the height and width axes while keeping the channel axes unchanged, then the elements in channel 0 will be rearranged along its main diagonal after the transpose, such as... Figure 1B As shown.

[0049] In this transpose process, assuming the system can read 3 bytes from memory in each loop, the system typically reads data sequentially from the initial memory address to fill the 3-byte capacity, and then writes this data to the specified memory location. In this case, in the first read loop, the tensor data (1,2,3) is acquired using 3 bytes / loop bandwidth. However, from this data, only the first byte (1) is actually acquired, because the first row of the transposed target tensor only requires (1), making the remaining bytes (2,3) unnecessary. Similarly, in subsequent loops, only one byte is effectively used per loop, such as (4) in the next set, resulting in memory bandwidth utilization efficiency of only one-third or 33%.

[0050] The inventors of this application recognize that the main reason for the above-mentioned inefficiency is that tensors are read row-by-row from source memory and written row-by-row to target memory when processing both the transposed axis and the stationary axis in tensor transposition. To overcome these technical limitations, this disclosure introduces a tensor transposition processor equipped with an internal transposition engine that performs row-by-row tensor reads and column-by-column tensor writes for the transposed axis. Furthermore, an address scheduler is included to manage the read and write operations for tensor data on the stationary axis.

[0051] Figure 2 An exemplary architecture diagram of a tensor transpose processor (TTP) according to various embodiments is shown. Figure 2 The tensor transpose processor (TTP) shown in the image is merely an example illustrating a basic component for improving memory efficiency for tensor transposes. Depending on the specific implementation, the TTP may include fewer, more, or alternative components.

[0052] In some embodiments, the TTP may include an instruction decoder for decoding tensor transpose instructions. Tensor transpose instructions may be received from a machine learning computation algorithm (e.g., a convolutional layer in a deep neural network). Each tensor transpose instruction causes an input tensor to be transposed into a target tensor. The input tensor may include multiple axes (dimensions) and is stored in input tensor memory 250. The tensor transpose instruction may indicate the transpose axis and the stationary axis among the multiple axes. The transpose axis may include two or more transposed axes, and the stationary axis may include axes other than the transpose axis among the multiple axes. It is important to note that even if the tensor data in the stationary axis does not change logically, the actual memory storage location of this tensor data may change during the transpose process (e.g., when the stationary dimension is sandwiched between transpose axes, the product of the sizes of all internal dimensions within the stationary dimension will change after transpose).

[0053] In some embodiments, the TTP may further include an internal transpose engine 260 that provides a pool of tensor buffer cells 254. These buffer cells are pre-configured to accommodate tensors of various sizes and serve as intermediate buffering areas within the TTP for performing internal tensor transposes before the generation of the final transposed tensors. This configuration contrasts with existing methods, in which input tensors are directly transposed from input tensor memory and written to output tensor memory. Instead, the TTP utilizes these internal buffer cells to perform buffered internal transposes of subtensors extracted from input tensors. Once these buffer cells are full, the intermediate transpose results are transferred to the output tensor memory, thereby significantly improving the utilization efficiency of memory read / write bandwidth.

[0054] In some embodiments, even if the size of the tensor buffer units is predetermined to accommodate most tensor sizes, the tensor buffer units of pool 254 can be dynamically allocated to manage incoming tensor transpose instructions. This dynamic allocation allows for seamless processing of operations: when a tensor buffer unit is occupied by the first tensor transpose instruction, subsequent requests for the same tensor buffer unit size do not need to wait for it to be released. Instead, additional tensor buffer units of the same size can be dynamically allocated using a standard memory allocation interface, thereby ensuring that processing proceeds continuously without delay.

[0055] In some embodiments, the size of a tensor buffer unit may be determined based on the maximum bandwidth of the tensor memory (e.g., input tensor memory 250). For example, assuming a maximum memory bandwidth of 512 bytes per read cycle, the tensor buffer unit may include two or more of the following sizes: [8,8,64], [16,16,32], [32,32,16], [4,4,128], [2,2,256], and [1,1,512]. Here, the first two dimensions represent the two outer dimensions, and the last dimension represents the innermost dimension, much like a typical HWCg (height, width, channel group).

[0056] In some embodiments, the product of the two innermost axes of a tensor buffer cell is equal to the maximum memory bandwidth, such as 512 in this example. This design is intended to utilize the maximum memory bandwidth to read multiple portions of contiguously stored tensor data.

[0057] In some embodiments, each of the plurality of tensor buffer units includes a first axis, a second axis, and a third axis. The first axis is equal to the second axis, and the third axis represents a dimension smaller than the maximum memory bandwidth of the input tensor memory (e.g., a channel group dimension).

[0058] When the internal transpose engine 260 is instructed to transpose the input tensor stored in the input tensor memory 250, it can first select a suitable tensor buffer cell to activate. The selection of the tensor buffer can be based on the size of the innermost dimension of the input tensor.

[0059] Taking a commonly used Gc*H*W*Cg (channel dimension group, height dimension, width dimension, and channel group dimension) tensor as an example, the innermost dimension is the channel group dimension. Each channel group can be considered a sub-part of the channel dimension. Each channel group includes multiple channels, and the tensor data within each channel group is stored contiguously in the input tensor memory. In this case, the selection of a suitable tensor buffer unit to be activated in the internal transpose engine 260 can be based on the size of the channel group of the input tensor. That is, the innermost dimension of the selected and activated tensor buffer unit can be the same size as the channel group of the input tensor.

[0060] After the selected tensor buffer cell is activated, tensor data from the transpose axis of the input tensor is read row-by-row from the input tensor memory 250 and then written column-by-column to the tensor buffer cell. This setup allows the transposition of tensor data from the transpose axis to be performed directly within the tensor buffer cell, with all transposition activities confined to that buffer. When the tensor buffer cell becomes full, the transposed tensor data is directly transferred to the output tensor memory 270 without requiring an additional transposition step.

[0061] The process of filling a tensor buffer cell requires several read loops, the number of which depends on the outermost dimension of the cell. Each read loop involves sequentially reading tensor data from the input tensor memory 250, thus utilizing the full buffer bandwidth (which matches the product of the two innermost dimensions of the tensor buffer cell, as shown in the example). Figure 3A (As shown).

[0062] Similarly, when the transposed tensor data is transferred from the tensor buffer unit to the output tensor memory 270, the same number of write loops are executed. Each write loop involves transferring a plane of tensor data from the buffer unit to the output memory, also utilizing the full memory bandwidth.

[0063] This method optimizes both reading and writing of memory to fully utilize available memory bandwidth, thereby improving memory access efficiency throughout the transpose process.

[0064] In some use cases, the input tensor may have a non-standard channel group size, resulting in no tensor buffer cell being perfectly suited for performing the internal transpose. To address this issue, in some embodiments, the internal transpose engine 260 may further include a tensor buffer cell mask 255. This tensor buffer cell mask 255 can be configured to mask a portion of the tensor buffer cells when the size of the input tensor does not match any of the tensor buffer cells in the pool 254. In other words, the tensor buffer cell mask 255 can be used to dynamically create “suitable” tensor buffer cells. More details about this tensor mask are available in […]. Figure 4 As shown.

[0065] In some embodiments, the internal transpose engine 260 is configured to process the transpose axis of the input tensor, i.e., the tensor data from the axis being transposed. To handle tensor data movement from the stationary axis of the input tensor, the TTP may also include an address scheduler 230, configured to calculate the target memory address of this tensor data from the stationary axis. This scheduler 230 is responsible for calculating the target memory address of the tensor data associated with the stationary axis. Given that the tensor structure before and after transpose is defined when decoding the tensor transpose instruction, this process involves calculating the source memory address and target memory address of the tensor data in the stationary axis, which is primarily achieved by updating the step size.

[0066] In this disclosure, the memory access (e.g., read / write) bandwidth of the input tensor memory 250 and the output tensor memory 270 refers to the ability of a computing system to read data from or write data to its memory within a single operating cycle, typically measured in bytes per cycle. Essentially, this metric represents the amount of data that can be efficiently transferred between a computer's memory and its processor within each clock cycle. The importance of memory bandwidth becomes particularly pronounced in high-performance computing tasks because the speed at which data is read from and written to memory significantly impacts overall system performance.

[0067] Figure 3A Exemplary tensor transpositions using a tensor transpose processor (TTP) are illustrated according to various embodiments. This example is provided to aid understanding. Figure 2 The TTP and transpose process are outlined in the document.

[0068] As shown in the figure, the input tensor 310 includes a height dimension, a width dimension, and a channel group dimension. Since the channel group dimension is the innermost dimension, the tensor data in the channel group dimension is stored contiguously in memory.

[0069] To use the TTP transpose input tensor 310, TTP's internal transpose engine (e.g., ...) Figure 2 (260) can activate a suitable tensor buffer unit 305. The selection of a suitable tensor buffer unit 305 can be based on the size of the channel group dimension of the input tensor 310. In this particular example, the size of the channel group dimension of the input tensor 310 is the same as the size of the innermost dimension of the tensor buffer unit 305.

[0070] Using TTP to transpose the input tensor 310 can involve multiple rounds, each round including an internal transpose of a sub-tensor in the input tensor. The size of the transposed sub-tensor in each round can be the same as the size of the active tensor buffer unit 305.

[0071] Within each transpose round, multiple read loops can be executed to fully utilize the memory read bandwidth and the buffer size of the tensor buffer unit 305. For example, in Figure 3A In this example, the height and width dimensions of the tensor buffer unit 305 are both set to 8. To completely fill it, 8 read loops are required, each loop involving 8 reads of the tensor data stored continuously in the input tensor memory according to the channel group dimension.

[0072] In some embodiments, tensor data is read from the input tensor memory row-wise (e.g., one row at a time), while tensor data is written to the tensor buffer unit 305 column-wise (e.g., one column at a time). For example, the first row of subtensors of the input tensor 310 (i.e., the dark highlighted portion of the first plane in the input tensor 310, assuming the channel group dimension is considered a unit) can be read all at once (because they are stored contiguously in memory). Here, the size of this subtensor is the same as the size of the first plane of the tensor buffer unit 305.

[0073] Each read cycle involves reading contiguous tensor data (equal to the product of the channel group dimension and the width dimension of tensor buffer unit 305). This configuration is set to match the maximum memory bandwidth, ensuring that each read cycle (in Figure 3A The term "full bandwidth read 312" in Chinese can fully utilize the available memory bandwidth.

[0074] After reading a row of tensor data, that row of tensor data is shifted and written column-wise to tensor buffer unit 305. For example... Figure 3A As shown, the first row read from input tensor 310 is written across all columns of tensor buffer unit 305 in a column-wise direction. When tensor buffer unit 305 is fully filled after 8 read cycles, the tensor data within it has been transposed. This transposed data is then transferred to output tensor 320 through 8 write cycles.

[0075] In each write loop, each row of tensor data is read from tensor buffer unit 305 and written to output tensor 320 in a shifted manner along the row direction. It is important to note that all tensor data in the same row has been transposed, allowing them to be written consecutively to the output tensor. However, because the rows in tensor buffer unit 305 have been shifted during loading (e.g., from internal memory), writing a transposed row from tensor buffer unit 305 to output tensor 320 requires restoring the shifted rows. Figure 3B A simplified example is shown.

[0076] Figure 3B Example loading and storing procedures for implementing internal transpose in a tensor transpose processor according to various embodiments are shown.

[0077] Figure 3B The term "bank" used here refers to different portions or sub-partitions of a memory array (in the context of SRAM). SRAM can be divided into multiple banks to improve parallelism and access speed. Each bank can be accessed independently, allowing simultaneous read or write operations in different banks. Figure 3BIn this diagram, for ease of explanation, each column represents a library. Those skilled in the art will understand how to apply the same process by treating each row as a library.

[0078] To achieve high-bandwidth transpose operations, the internal transpose process (i.e., the load and store process) needs to ensure that the input data resides in different databases, and the output data also resides in different databases. This is achieved through a "shift" operation.

[0079] As shown in the figure, in the "Load Data" phase, loop 0 involves reading a row of tensor data (1,2,3,4) and writing it to the internal buffer unit in a column-wise shift manner across all libraries: tensor data (1) is stored in library 0, tensor data (2) is stored in library 1, tensor data (3) is stored in library 2, and tensor data (4) is stored in library 3. During loop 1, the same process is repeated for the next row of tensor data (5,6,7,8), where tensor data (5) is stored in library 1, tensor data (6) is stored in library 2, tensor data (7) is stored in library 3, and tensor data (8) is stored in library 0. This process continues to complete all four rows.

[0080] As shown in the figure, each row of data in the input tensor is distributed across all libraries (to maximize parallelism), and the starting library used to load each row is right-shifted. After the "load data" phase, each row stored in the internal tensor buffer has been transposed. However, during the loading process, different rows in the internal tensor buffer are shifted, so these rows need to be shifted back for alignment, ensuring that the resulting tensor is fully transposed and aligned.

[0081] Specifically, in the "store data" phase, each loop includes writing one row of tensor data from the internal tensor buffer to the output tensor. During the (i+1)th loop, the (i+1)th row of the internal tensor buffer is written to the (i+1)th row of the output tensor, but the starting library in the (i+1)th loop is shifted to the right compared to the starting library in the ith loop. Figure 3B As shown, loop 4 includes writing the first tensor data to library 0, and loop 5 includes writing the first tensor data to library 1. Figure 3B The “result tensor” indicates that after the shift loading and shift storage processes, the input tensor is internally transposed and stored as the output tensor.

[0082] In summary, the multiple read cycles from the input tensor memory and the multiple write cycles to the output tensor memory are designed to fully utilize the maximum memory bandwidth, which is facilitated by tensor buffer cells of a specially configured size.

[0083] Figure 3CExample architecture diagrams of an internal transpose engine 260 according to various embodiments are shown. While the internal transpose engine 260 is also... Figure 2 As shown in the text, but Figure 3C An example internal configuration of the internal transpose engine 260 is shown. In some embodiments, the internal transpose engine 260 can be understood as a standalone tensor transpose processing unit that works in conjunction with existing input memory (which stores the tensors to be transposed) and output memory (which stores the transposed tensors).

[0084] In some embodiments, the internal transpose engine 260 may include an input shift buffer 262, a temporary buffer 263, and an output shift buffer 267. The temporary buffer 263 may include one or more buffers, depending on the implementation. Each buffer in the temporary buffer 263 may include multiple memory banks. In an SRAM context, a "memory bank" represents a specific segment or sub-partition within a buffer or memory array. Dividing SRAM into multiple banks significantly improves parallelism and access speed. Each bank operates independently, facilitating simultaneous read or write operations across different banks. An "entry" in SRAM refers to a single storage location within a bank for holding a given amount of data. Each entry is uniquely addressable, meaning that by specifying the entry's address within the bank, the entry can be accessed, read, or written.

[0085] To achieve high-bandwidth transpose operations, it is necessary to ensure that input data is written to different libraries in temporary buffer 263 and that output data is read from different libraries. Therefore, internal shift operations are performed to distribute data to multiple libraries. This distribution allows simultaneous access to multiple data entries, thereby improving processing efficiency and minimizing latency. The main objectives of the input shift operations performed by input shift buffer 262 and the output shift operations performed by output shift buffer 267 are to leverage the parallelism of data access and ensure a continuous data flow during transpose.

[0086] In some embodiments, the input shift buffer 262 is a multi-functional component configured to read data (e.g., input tensors) from the input memory for transposition, perform multiple input tensor shift loops on the data read from the input memory, and write the results of the multiple input tensor shift loops to the temporary buffer 263.

[0087] In some embodiments, the input shift buffer 262 performs multiple input tensor shifting loops sequentially on the data read from the input memory, i.e., one loop after another. In each loop, the input shift buffer 262 reads a sub-tensor of the input tensor from the input memory and writes the sub-tensor across multiple memory banks of the temporary buffer 263 along a first shift direction. In some embodiments, a sub-tensor herein refers to a row of data in the input tensor, and that row of data is written along a column direction across multiple memory banks of the temporary buffer 267.

[0088] To better understand this input shifting process, pay attention to the direction. Figure 3B .like Figure 3B As shown in the "Load Data" stage, a row of data (1,2,3,4) is read from the input memory. In loop 0, this row of data (1,2,3,4) is written column-wise across multiple memory banks (banks 0, 1, 2, 3), with each bank receiving one element. Then, in loop 1, another row of data (5,6,7,8) is read from the input memory and then written column-wise across multiple memory banks. It's important to note that the writing of shifted data across memory banks is a circular write, meaning the memory banks are traversed sequentially, and the sequence wraps back to the first memory bank after the last memory bank in the sequence has been written.

[0089] During multiple input tensor shifting loops, intermediate data is gradually stored in temporary buffer 263. It is important to note that this intermediate data does not yet represent the transposed version of the input tensor. The transposed version of the input tensor (also known as the output tensor) is generated only after multiple output tensor shifting loops are performed on the intermediate data.

[0090] In some embodiments, the output shift buffer 267 is also a multi-functional component configured to read data from the temporary buffer 263 (e.g., the result of a plurality of input tensor shift loops written to the input shift buffer 262), perform a plurality of output tensor shift loops on the data read from the temporary buffer 263, and write the result of the plurality of output tensor shift loops to the output memory.

[0091] In some embodiments, the output shift buffer 267 performs multiple output tensor shift cycles sequentially on the data read from the temporary buffer 263, i.e., one cycle after another. In each cycle, the output shift buffer 267 reads a sub-tensor from the temporary buffer 263 and writes the sub-tensor to the output memory along a second shift direction. In some embodiments, the sub-tensor refers to a row of data in the temporary buffer, which is shifted along the second shift direction and then written in the row direction (i.e., still row by row).

[0092] Return to reference againFigure 3B Multiple output tensor shift loops are the process of transferring data from the "stored data" stage to the "result data" stage. In the first loop ( Figure 3B In loop 4), a row of data (1,5,9,13) is read from temporary buffer 263 and written to the output buffer by shifting it in the opposite direction to the input shift direction. Figure 3B In the first loop, the input shift direction is to the right, so the output shift direction is to the left. Figure 3B In loop 4), the shift step is 0, so the data (1,5,9,13) is copied to the output buffer. In the second loop ( Figure 3B In loop 5), the second row of data (14,2,6,10) is read from the temporary buffer 263 and shifted one element to the left to get a row of data (2,6,10,14), which is then written to the output buffer.

[0093] In summary, the complete tensor transpose of the input tensor requires: (1) reading the input tensor along the row direction and writing the data along the column direction to the temporary buffer 263; (2) shifting the column data across multiple memory banks along the first direction while writing the data; and (3) reading the tensor from the temporary buffer, shifting it along the second direction opposite to the first direction, and writing the shifted data to the output memory.

[0094] While the above description outlines the overall flow of the internal transpose engine, improved hardware design of the scratch buffer 263 can further enhance the efficiency of the internal transpose engine 260. In some embodiments, the scratch buffer 263 comprises a pair of identical buffers, also known as twin buffers or ping-pong buffers. Figure 3C As shown, the two temporary buffers 265 and 266 are implemented and coordinated by the ping-pong buffer controller 263.

[0095] The purpose of including this pair of buffers is to enable parallel processing and avoid idle cycles in the input shift buffer 262 and the output shift buffer 267. For example... Figure 3B As shown, both the input and output shift processes require multiple loops. If there is only one temporary buffer, the input shift buffer 262 will have to wait for the output shift buffer 267 to finish reading data from the temporary buffer before it can start the next round of input shift operations. Figure 3D An example process is shown that maximizes the efficiency of the internal transpose engine 260 using a pair of temporary buffers.

[0096] Figure 3D The parallel processing pipeline in the internal transpose engine according to various embodiments is illustrated. As shown, a pair of temporary buffers (e.g., ping buffer 280 and pong buffer 282) are configured to facilitate parallel tensor transpose.

[0097] For simplicity, Figure 3D Let's use an example to illustrate how this pair of buffers 280 and 282 prevents idle loops during tensor transposition. In each loop from 5 to 8, the output shift buffer reads tensor data from ping buffer 280 (i.e., the intermediate tensor data written by the input shift buffer during loops 1-4) to perform an output shift. During this period, ping buffer 280 cannot receive new data from the input shift buffer before the output shift buffer finishes. However, with the help of ping buffer 282, the input shift buffer can begin processing the next sub-tensor of the input tensor, performing an input shift and writing to ping buffer 282. Then, during loops 9-12, the output shift buffer can consume the data from ping buffer 282, and the input shift buffer can use ping buffer 280 to begin processing the next sub-tensor. In this way, both the input and output shift buffers can remain busy. This pattern continues, avoiding idle loops and ensuring efficient processing.

[0098] Figure 4 Another exemplary tensor transposition using a tensor transpose processor (TTP) is illustrated according to various embodiments. In some embodiments, pre-configured tensor buffer units in the TTP are adapted to standard tensor sizes (more specifically, standard channel group sizes). However, when the input tensor 410 has irregular channel group sizes, no existing tensor buffer unit can match the input tensor 410.

[0099] In some embodiments, if the innermost (i.e., channel group) dimension of none of the existing tensor buffer units matches the channel group size of the input tensor 410, a nearby tensor buffer unit can be selected within the TTP to perform the internal transpose. The selected nearby tensor buffer unit has a minimum innermost dimension exceeding the channel group size of the input tensor 410.

[0100] After selecting a close tensor buffer cell, a tensor buffer mask can be applied to that close tensor buffer cell to obtain a masked tensor buffer cell 420, such that the innermost dimension of the masked tensor buffer cell 420 is the same size as the channel group dimension of the input tensor 410 (in Figure 4 The Chinese text is written as “concealment to match” (444).

[0101] Using the masked tensor buffer unit 420, multiple cyclic reads from the input tensor and multiple cyclic writes to the output tensor can be performed. Figure 3AThis is performed in the manner described. Even if some buffer space of the adjacent output buffer is masked and therefore underutilized, the product of the masked tensor buffer units 420 (channel group dimension × width dimension) can still be configured to the maximum memory bandwidth. In this way, each read cycle and write cycle can still make full use of the memory bandwidth.

[0102] Figure 5A Examples of high-dimensional tensor representations are shown, providing Figures 5B-5D This forms the basis of the description. While low-dimensional tensors can be represented as vectors or matrices, high-dimensional tensor representations are often less intuitive. To simplify the explanation, Figure 5A (and subsequent) Figure 5B-5D A direct approach is used to represent 5-dimensional tensors, where each small square represents nested data in the next dimension. This representation effectively utilizes a tree structure to depict high-dimensional tensors.

[0103] Figure 5B Exemplary tensor transpositions are shown using a tensor transposition processor when consecutive cells of a tensor are less than the memory bandwidth, according to various embodiments. Because consecutive cells of a tensor are less than the memory bandwidth, multiple consecutive cells can be read in the same iteration to fully utilize the memory bandwidth.

[0104] exist Figure 5B In this context, "continuous unit" refers to data in a tensor that is stored contiguously in memory. Generally, the innermost dimension of a tensor is stored contiguously (e.g., ...). Figure 5B (D4 dimension in the tensor), but continuous units can include more than one round of the innermost dimension of the tensor.

[0105] Specifically Figure 5B The source tensor (D0D1D2D3D4) on the left side is being transposed to Figure 5B The target tensor on the right side is (D1D0D3D2D4). The transpose axes are D2 and D3, and the other axes (D0 and D1) are stationary axes. D4 is a continuous cell. The continuous cell D4 is smaller than the memory bandwidth.

[0106] In this case, transposition involves reading multiple “inner columns” from D3 of the source tensor using selected tensor buffer cells. The number of consecutive “inner columns” read is determined by the “read step size” in the D2 dimension (i.e., the dimension preceding D3 of the input tensor). The product of the “read step size” and the “inner columns” equals the memory bandwidth.

[0107] During a write cycle, multiple “internal columns” are read from the tensor buffer cells and written to the output buffer. The number of “internal columns” read from the tensor buffer is determined by the “write stride” in the new D3 dimension (i.e., the dimension before the innermost dimension D2 of the transposed tensor). The product of the “write stride” and the “write iteration” (e.g., write iteration 1 and write iteration 0 in the new D3 dimension) equals the memory bandwidth.

[0108] During this process, the D0 and D1 dimensions in the input tensor are not internally transposed by the internal transpose engine. Instead, the permutation of these two dimensions is accomplished by the address scheduler manipulating memory addresses.

[0109] Figure 5C This illustrates another exemplary tensor transposition performed using a tensor transposition processor when the consecutive cells of a tensor are less than the memory bandwidth, according to various embodiments. Figure 5C The example involves transposing (swapping) the discontinuous dimensions D1 and D3 in the input tensor. Again, the continuous unit is dimension D4. In the input tensor, the read step size in dimension D1 multiplied by the "internal column" equals the memory bandwidth. Similarly, in the output (transposed) tensor, the write step size in dimension D3 multiplied by the "write iteration" (e.g., write iteration 0 and write iteration 1 in the new D3 dimension) equals the memory bandwidth.

[0110] During this process, the D0 and D2 dimensions in the input tensor are not internally transposed by the internal transpose engine. Instead, the permutation of these two dimensions is accomplished by the address scheduler manipulating memory addresses.

[0111] Figure 5D Exemplary tensor transpositions are shown using a tensor transposition processor when consecutive units of a tensor are equal to or greater than the memory bandwidth, according to various embodiments. When consecutive units are equal to or greater than the memory bandwidth, the memory bandwidth is fully utilized in each read and write cycle.

[0112] During this process, the D0, D1, and D2 dimensions in the input tensor are not internally transposed by the internal transpose engine. Instead, the permutation of these three dimensions is accomplished by the address scheduler manipulating memory addresses.

[0113] Figure 6A Exemplary method 600 for tensor transposition using a tensor transpose processor is illustrated according to various embodiments. In some implementations, Figure 6A One or more processing blocks can be executed by the tensor transpose processor.

[0114] like Figure 6AAs shown, process 600 may include decoding a tensor transpose instruction for transposing an input tensor stored in an input tensor memory, wherein the input tensor may include multiple axes, and the tensor transpose instruction indicates a transposed axis and a stationary axis among the multiple axes, wherein the transposed axis may include two or more transposed axes, and the stationary axis may include one or more untransposed axes (block 610). For example, a tensor transpose processor may decode a tensor transpose instruction for transposing an input tensor stored in an input tensor memory, wherein the input tensor may include multiple axes, and the tensor transpose instruction indicates a transposed axis and a stationary axis among the multiple axes, wherein the transposed axis may include two or more transposed axes, and the stationary axis may include one or more untransposed axes, as described above.

[0115] like Figure 6A As shown, process 600 may further include activating a tensor buffer unit for transposing the input tensor, wherein the tensor buffer unit is selected from a plurality of tensor buffer units of different sizes (block 620). For example, a tensor transpose processor may activate a tensor buffer unit for transposing the input tensor, wherein the tensor buffer unit is selected from a plurality of tensor buffer units of different sizes, as described above.

[0116] Further as Figure 6A As shown, process 600 may include reading tensor data from the transpose axis of the input tensor row by row from the input tensor memory (block 630). For example, the tensor transpose processor may read tensor data from the transpose axis of the input tensor row by row from the input tensor memory, as described above.

[0117] In addition, such as Figure 6A As shown, process 600 may include writing tensor data column-wise to a tensor buffer unit (block 640). For example, a tensor transpose processor may write tensor data column-wise to the tensor buffer unit as described above. Further, as shown in FIG6, process 600 may include copying tensor data from the tensor buffer unit to an output tensor memory in response to the tensor buffer unit being full (block 650). For example, a tensor transpose processor may copy tensor data from the tensor buffer unit to an output tensor memory in response to the tensor buffer unit being full, as described above.

[0118] although Figure 6A The example block of process 600 is shown, but in some implementations, process 600 may include additional blocks, fewer blocks, different blocks, or blocks similar to... Figure 6A The blocks are arranged differently as shown. Alternatively, two or more blocks of process 600 can be executed in parallel.

[0119] Figure 6BAnother exemplary method 680 for tensor transposition using a tensor transpose processor according to various embodiments is shown. Figure 6B The blocks (steps) in the code are for illustrative purposes. Depending on the specific implementation, process 680 may include additional blocks, fewer blocks, different blocks, or blocks related to... Figure 6B The blocks are arranged differently as shown. Alternatively, two or more blocks of process 680 can be executed in parallel.

[0120] Block 660 includes performing multiple input tensor shift loops on input tensors read from input memory using an input tensor shift buffer.

[0121] Block 662 includes writing the result of the input tensor shifting cycle to a temporary buffer during each of the plurality of input tensor shifting cycles, using an input tensor shifting buffer. In some embodiments, the temporary buffer includes a plurality of memory banks.

[0122] Block 664 includes performing multiple output tensor shift loops on the results read from the temporary buffer using an output tensor shift buffer.

[0123] Block 666 includes writing the result of the output tensor shift cycle to the output memory using an output tensor shift buffer during each of the multiple output tensor shift cycles.

[0124] In some embodiments, performing multiple input tensor shifting cycles on an input tensor read from an input memory includes, during each of the multiple input tensor shifting cycles: reading a sub-tensor of the input tensor from the input memory, and, while writing to a temporary buffer, shifting the sub-tensor of the input tensor across multiple memory banks along a first shift direction.

[0125] In some embodiments, the sub-tensor read from the input memory includes a row of data from the input tensor, and writing the result of the input tensor shift loop to the temporary buffer includes: writing a row of data read from the input tensor along the column direction across a plurality of memory banks spanning the temporary buffer.

[0126] In some embodiments, performing multiple output tensor shift cycles on the result read from the temporary buffer includes, during each of the multiple output tensor shift cycles: reading a sub-tensor from the temporary buffer and shifting the sub-tensor along a second shift direction when writing to the output memory.

[0127] In some embodiments, the sub-tensor read from the temporary buffer includes a row of data read from the temporary buffer, and writing the result of the output tensor shift cycle to the output memory includes writing the row of data by shifting a row of data from the temporary buffer along a second shift direction.

[0128] In some embodiments, the direction of input tensor shift is opposite to the direction of output tensor shift.

[0129] In some embodiments, the temporary buffer includes a pair of buffers that are the same size.

[0130] In some embodiments, the pair of buffers includes a first buffer and a second buffer that support parallel processing, and the output tensor shift buffer is reading from the second buffer while the input tensor shift buffer is writing to the first buffer.

[0131] In some embodiments, the pair of buffers includes a first buffer and a second buffer, and after the input tensor shift buffer writes the first result obtained by performing multiple input tensor shift cycles on the first input tensor into the first buffer, the output tensor shift buffer begins to read the first result from the first buffer to execute multiple output tensor shift cycles; and when the output tensor shift buffer reads from the first buffer, the input tensor shift buffer begins to write the second result obtained by performing multiple input tensor shift cycles on the second input tensor into the second buffer, so that the input tensor shift buffer and the output tensor shift buffer avoid idle time.

[0132] Figure 7 An example computing device is shown that can implement any of the embodiments described herein. This computing device can be used to implement one or more components of the systems and methods shown in Figures 1-6. The computing device 700 may include a bus 702 or other communication mechanism for transmitting information and one or more hardware processors 704 coupled to the bus 702 for processing information. The hardware processors 704 may, for example, be one or more general-purpose microprocessors.

[0133] The computing device 700 may also include a main memory 707 coupled to a bus 702, such as random access memory (RAM), cache, and / or other dynamic storage devices, for storing instructions and information to be executed by one or more processors 704. The main memory 707 may also be used to store temporary variables or other intermediate information during instruction execution by one or more processors 704. Such instructions stored in storage media accessible to one or more processors 704 enable the computing device 700 to become a dedicated machine tailored to perform the operations specified in the instructions. The main memory 707 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common forms of media may include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, EPROM, FLASH-EEPROM, NVRAM, any other memory chip or cartridge, or networked versions of these items.

[0134] The computing device 700 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic (which, in combination with the computing device, enable the computing device 700 to be a dedicated machine or programmed to be a special-purpose machine). According to one embodiment, the techniques described herein are executed by the computing device 700 in response to one or more sequences of instructions contained in main memory 707 executed by processor(s) ...

[0135] The computing device 700 also includes a communication interface 710 coupled to a bus 702. The communication interface 710 provides bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, the communication interface 710 may be a local area network (LAN) card for providing data communication connectivity to a compatible LAN (or a WAN component communicating with a WAN). Wireless links may also be implemented.

[0136] The execution of certain operations may be distributed across processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., a home environment, an office environment, or a server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0137] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors, including computer hardware, and can be automated, wholly or partially, through these code modules. These processes and algorithms can also be implemented, wholly or partially, in dedicated circuitry.

[0138] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects that contribute to the prior art, may be embodied in the form of a software product. This software product may be stored in a storage medium and includes several instructions for causing a computing device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods of the embodiments of this application. The storage medium may include a flash drive, a portable hard disk drive, ROM, RAM, a magnetic disk, an optical disk, other additional media operable for storing program code, or any combination of the foregoing.

[0139] Specific embodiments also provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to the steps in the methods of any of the above embodiments. Specific embodiments also provide a non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause one or more processors to perform operations corresponding to the steps in the methods of any of the above embodiments.

[0140] The embodiments disclosed herein can be implemented through a cloud platform, server, or group of servers (collectively referred to as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device capable of installing platform applications.

[0141] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, in some implementations, certain method or process blocks may be omitted. The methods and processes described herein are not limited to any particular order, and the associated blocks or states may be executed in other suitable orders. For example, the described blocks or states may be executed in a different order than specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed example embodiments.

[0142] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. This algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). Such algorithms may include machine learning algorithms. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform a function, but may learn from training samples to build a predictive model that performs that function.

[0143] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which can be temporarily or permanently configured (software) to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute an engine for a processor implementation that operates to perform one or more of the operations or functions described herein.

[0144] Similarly, the methods described herein can be implemented at least partially by a processor, with one or more specific processors serving as examples of hardware. For example, at least some operations of a method can be performed by one or more processors or an engine implemented by a processor. Furthermore, the one or more processors can also be used to support the execution of related operations in a "cloud computing" environment or as "Software as a Service" (SaaS). For instance, at least some operations can be performed by a set of computers (as an example of a machine containing processors), and these operations can be accessed via a network (e.g., the Internet) and one or more suitable interfaces (e.g., application programming interfaces (APIs)).

[0145] The execution of certain operations can be distributed across multiple processors, residing not only on a single machine but also deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., a home environment, an office environment, or a server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0146] In this specification, multiple instances may implement components, operations, or structures described as single instances. Although individual operations of one or more methods are described and depicted as separate operations, one or more individual operations may be performed simultaneously, and these operations are not required to be performed in the order described. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other changes, modifications, additions, and improvements fall within the scope of this disclosure.

[0147] As used herein, “or” is inclusive rather than exclusive unless explicitly stated otherwise or the context indicates otherwise. Therefore, in this document, “A, B, or C” means “A, B, A and B, A and C, B and C, or A, B, and C” unless explicitly stated otherwise or the context indicates otherwise. Furthermore, “and” is both united and separate unless explicitly stated otherwise or the context indicates otherwise. Therefore, in this document, “A and B” means “A and B, united or separate” unless explicitly stated otherwise or the context indicates otherwise. Additionally, multiple instances may be provided for a resource, operation, or structure described herein as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are described within specific illustrative configurations. Other functional assignments are foreseeable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as separate resources in example configurations may be implemented as composite structures or resources. Similarly, structures and functions presented as separate resources may be implemented as separate resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0148] The terms "comprising" or "including" are used to indicate the presence of subsequently declared features, but do not preclude the addition of other features. Conditional language, such as "can," "may," "may," or "will," is generally intended to convey, unless specifically stated or understood in the context of use, that certain embodiments include certain features, elements, and / or steps while other embodiments do not. Therefore, such conditional language is not generally intended to imply that features, elements, and / or steps are necessary for one or more embodiments in any way, or that one or more embodiments necessarily include logic for determining (with or without user input or prompting) whether such features, elements, and / or steps are included in any particular embodiment or whether they should be performed.

[0149] Although an overview of the subject matter has been described with reference to specific example embodiments, various modifications and variations may be made to these embodiments without departing from the broader scope of embodiments of this disclosure. Embodiments of such subject matter may be referred to individually or collectively as “inventions” for convenience only and are not intended to intentionally limit the scope of this application to any single disclosure or concept (if more than one is actually disclosed).

[0150] The embodiments described herein are illustrated in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. Therefore, the specific implementation should not be considered limiting, and the scope of the various embodiments is defined only by the full scope of the appended claims and their equivalents.

Claims

1. A processor for accelerating tensor transpose operations in machine learning, comprising: Input tensor shift buffer; Temporary buffer; as well as Output tensor shift buffer; in: The input tensor shift buffer is configured as follows: Read the input tensor from the input memory; Perform multiple input tensor shift loops on the input tensor read from the input memory; and Write the results of the multiple input tensor shift loops into the temporary buffer; and The output tensor shift buffer is configured as follows: Read the results of the plurality of input tensor shift loops from the temporary buffer; Perform multiple output tensor shift loops on the results read from the temporary buffer; and The results of the multiple output tensor shift cycles are written into the output memory.

2. The processor according to claim 1, wherein, The temporary buffer includes multiple memory libraries.

3. The processor according to claim 2, wherein, The input tensor shift buffer is also configured to, during each of the plurality of input tensor shift cycles: Read a sub-tensor of the input tensor from the input memory; and The plurality of memory banks spanning the temporary buffer are used to write sub-tensors of the input tensor along a first shift direction.

4. The processor according to claim 3, wherein, In order to read sub-tensors of the input tensor, the input tensor shift buffer is further configured to: Read one line of data from the input tensor from the input memory, and In order to write sub-tensors of the input tensor along the first shift direction across the plurality of memory banks of the temporary buffer, the input tensor shift buffer is further configured to: The plurality of memory banks spanning the temporary buffer are used to write the row of data read from the input tensor along the column direction.

5. The processor according to claim 2, wherein, The output tensor shift buffer is also configured to, during each of the plurality of output tensor shift cycles: Read the sub-tensor from the temporary buffer, and The sub-tensor is written to the output memory along the second shift direction.

6. The processor according to claim 5, wherein, In order to read the sub-tensor from the temporary buffer, the output tensor shift buffer is further configured to: Read a line of data from the temporary buffer; and The row of data is written from the temporary buffer by shifting the row of data along the second shift direction.

7. The processor according to claim 1, wherein, The direction of the input tensor shift is opposite to the direction of the output tensor shift.

8. The processor according to claim 1, wherein, The temporary buffer includes a pair of buffers that are the same size.

9. The processor according to claim 8, wherein, The pair of buffers includes a first buffer and a second buffer that support parallel processing, and When the input tensor shift buffer is writing to the first buffer, the output tensor shift buffer is reading from the second buffer.

10. The processor according to claim 8, wherein, The pair of buffers includes a first buffer and a second buffer, and After the input tensor shift buffer writes the first result obtained by performing the plurality of input tensor shift loops on the first input tensor into the first buffer, the output tensor shift buffer begins to read the first result from the first buffer to execute the plurality of output tensor shift loops; and While the output tensor shift buffer is reading from the first buffer, the input tensor shift buffer begins writing the second result obtained by performing the plurality of input tensor shift cycles on the second input tensor into the second buffer, thereby avoiding idle time for both the input tensor shift buffer and the output tensor shift buffer.

11. A method for accelerating tensor transpose operations in machine learning, comprising: Perform multiple input tensor shift loops on the input tensor read from the input memory using an input tensor shift buffer; During each of the plurality of input tensor shift loops, the result of the input tensor shift loop is written to a temporary buffer using the input tensor shift buffer; Multiple output tensor shift loops are performed on the results read from the temporary buffer using an output tensor shift buffer; as well as During each of the plurality of output tensor shift loops, the result of the output tensor shift loop is written to the output memory using the output tensor shift buffer.

12. The method according to claim 11, wherein, The temporary buffer includes multiple memory libraries.

13. The method according to claim 12, wherein, Performing the plurality of input tensor shift loops on the input tensor read from the input memory includes: During each of the plurality of input tensor shifting loops: Read the sub-tensor of the input tensor from the input memory, and When writing to the temporary buffer, the sub-tensor of the input tensor is shifted across the plurality of memory banks along a first shift direction.

14. The method according to claim 13, wherein, The sub-tensor read from the input memory includes one line of data from the input tensor, and Writing the result of the input tensor shift loop to the temporary buffer includes: A row of data read from the input tensor is written along the column direction across the plurality of memory banks spanning the temporary buffer.

15. The method according to claim 12, wherein, Performing the plurality of output tensor shift loops on the results read from the temporary buffer includes: During each of the plurality of output tensor shift cycles: Read the sub-tensor from the temporary buffer, and When writing to the output memory, the sub-tensor is shifted along the second shift direction.

16. The method according to claim 15, wherein, The sub-tensor read from the temporary buffer includes a row of data read from the temporary buffer, and Writing the result of the output tensor shift loop to the output memory includes: The row of data is written from the temporary buffer by shifting the row of data along the second shift direction.

17. The method according to claim 11, wherein, The direction of the input tensor shift is opposite to the direction of the output tensor shift.

18. The method according to claim 11, wherein, The temporary buffer includes a pair of buffers that are the same size.

19. The method according to claim 18, wherein, The pair of buffers includes a first buffer and a second buffer that support parallel processing, and When the input tensor shift buffer is writing to the first buffer, the output tensor shift buffer is reading from the second buffer.

20. The method according to claim 18, wherein, The pair of buffers includes a first buffer and a second buffer, and After the input tensor shift buffer writes the first result obtained by performing the plurality of input tensor shift loops on the first input tensor into the first buffer, the output tensor shift buffer begins to read the first result from the first buffer to execute the plurality of output tensor shift loops; and While the output tensor shift buffer is reading from the first buffer, the input tensor shift buffer begins writing the second result obtained by performing the plurality of input tensor shift cycles on the second input tensor into the second buffer, thereby avoiding idle time for both the input tensor shift buffer and the output tensor shift buffer.

Images