Tensor layout optimization method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a tensor layout optimization method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target operation in a calculation graph; on the basis of the calculation graph, obtaining all associated operations having a data dependency relationship with the target operation, and determining a data stream comprising the target operation and each associated operation as a to-be-optimized data stream under the condition that each associated operation meets a second preset condition; based on the memory access channel characteristics of the computing hardware, a target layout code is created, and the target layout code is used for dividing one thread bundle into at least two thread groups, so that different thread groups perform time division multiplexing on the memory access channel of the computing hardware; and based on the target layout code, updating the memory layout of the tensor associated with each operation in the to-be-optimized data stream. According to hardware characteristics of a specific framework, the memory bandwidth utilization rate and the memory access performance are remarkably improved by optimizing the memory layout of the tensor and performing time division multiplexing on the memory access channel.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a tensor layout optimization method, electronic device, and storage medium. Background Technology

[0002] In the training and inference processes of deep learning models, large-scale tensor computation is a core component. To improve computational efficiency and fully utilize the powerful parallel processing capabilities of computing hardware, the industry has developed various tensor computation optimization techniques, such as static partitioning, compiler-automated optimization, and hardware-specific instructions. Among these, static partitioning optimizes data reuse by pre-setting fixed data block sizes in the underlying library; compiler-automated optimization optimizes tensor computation by automatically searching for the optimal computation code for a specific model and hardware; and hardware-specific instructions directly utilize dedicated hardware units on the computing hardware to accelerate specific computations.

[0003] However, the aforementioned optimization techniques are typically based on general hardware architecture designs and fail to adequately optimize for the hardware characteristics of certain specialized architectures, particularly in memory access mechanisms. For example, some specific computing hardware architectures have fewer physical memory access channels (e.g., 16) than the number of threads within a standard warp (e.g., 32). When all threads within a warp simultaneously initiate memory access, the hardware cannot process these accesses in parallel due to insufficient memory access channels, and execution must be sequential. This results in low memory bandwidth utilization and creates a performance bottleneck. Summary of the Invention

[0004] This invention provides a tensor layout optimization method, electronic device, and storage medium to address the shortcomings of related technologies that do not adequately consider specific hardware architectures, resulting in low memory access efficiency when performing tensor computation tasks on certain computing hardware.

[0005] This invention provides a tensor layout optimization method, comprising: Obtain the target operation in the computation graph, wherein the target operation is a memory access operation that satisfies a first preset condition; Based on the computation graph, all associated operations that have data dependencies on the target operation are obtained, and if each associated operation satisfies the second preset condition, the data flow including the target operation and each associated operation is determined as the data flow to be optimized. Based on the memory access channel characteristics of computing hardware, a target layout code is created. The target layout code is used to divide multiple threads within a thread bundle into at least two thread groups so that different thread groups can time-division multiplex the memory access channels of the computing hardware. Based on the target layout encoding, update the memory layout of the tensors associated with each operation in the data stream to be optimized.

[0006] According to a tensor layout optimization method provided by the present invention, the first preset condition includes that the target operation is at least one of memory loading, memory storage, and atomic read-write operation, and the data volume of the tensor associated with the target operation is not less than a preset data volume threshold.

[0007] According to a tensor layout optimization method provided by the present invention, each associated operation includes at least one of the direct predecessor operation, direct successor operation, indirect predecessor operation, and indirect successor operation of the target operation, and the method further includes: If any associated operation does not meet the second preset condition, layout optimization of the tensor associated with the target operation is abandoned. The second preset condition is that the data volume of the tensor associated with the associated operation is not less than a preset data volume threshold.

[0008] According to a tensor layout optimization method provided by the present invention, the step of creating a target layout code based on the memory access channel characteristics of computing hardware includes: Based on the memory access channel characteristics of the computing hardware, the target layout code is created, and the attribute in the target layout code used to define the thread distribution within the thread bundle is set to a dimension corresponding to the at least two thread groups.

[0009] According to a tensor layout optimization method provided by the present invention, after creating the target layout encoding, the method further includes: Determine whether the tensor associated with each operation in the data stream to be optimized is a one-dimensional tensor; If so, then based on the dimension of the thread distribution in the target layout encoding, the one-dimensional tensor is reconstructed into a multi-dimensional tensor, and the logical dimension of the multi-dimensional tensor matches the dimension of the thread distribution.

[0010] According to a tensor layout optimization method provided by the present invention, the step of reconstructing the one-dimensional tensor into a multi-dimensional tensor includes: When the dimension of the thread distribution is two-dimensional, the one-dimensional tensor is reconstructed into a two-dimensional tensor. The dimensions of the two-dimensional tensor include a first dimension and a second dimension. The size of the first dimension is determined based on the shape of the one-dimensional tensor and a preset data volume threshold. The size of the second dimension is determined based on the preset data volume threshold. The preset data volume threshold is determined based on the amount of data processed in parallel by threads within a thread bundle in a memory access transaction.

[0011] According to a tensor layout optimization method provided by the present invention, the step of updating the memory layout of the tensors associated with each operation in the data stream to be optimized based on the target layout encoding includes: Iterate through each operation in the data stream to be optimized; For the current operation that is traversed, the operands and return value type information of the current operation are updated based on the target layout encoding.

[0012] According to a tensor layout optimization method provided by the present invention, the attributes of the target layout encoding include at least one of the following: the number of data elements processed by a single thread, the thread distribution within a thread bundle, and the dimension processing order.

[0013] The present invention also provides a tensor layout optimization device, comprising: The acquisition unit is used to acquire the target operation in the computation graph, wherein the target operation is a memory access operation that satisfies a first preset condition; The determining unit is configured to, based on the computation graph, acquire all associated operations that have data dependencies on the target operation, and, if all associated operations satisfy a second preset condition, determine the data flow including the target operation and all associated operations as the data flow to be optimized; A creation unit is used to create a target layout code based on the memory access channel characteristics of the computing hardware. The target layout code is used to divide multiple threads within a thread bundle into at least two thread groups so that different thread groups can time-division multiplex the memory access channels of the computing hardware. An update unit is used to update the memory layout of the tensors associated with each operation in the data stream to be optimized based on the target layout encoding.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the tensor layout optimization method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the tensor layout optimization method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the tensor layout optimization method as described above.

[0017] The tensor layout optimization method, electronic device, and storage medium provided by this invention obtain memory access operations in the computation graph as target operations and analyze all related operations that have data dependencies on the target operations. This determines the data flow to be optimized, ensuring that subsequent optimization operations are performed on memory-intensive and most likely bottleneck-prone critical paths. Then, based on the specific memory access channel characteristics of the computing hardware, a novel target layout code is created to divide a thread bundle into at least two thread groups. This thread grouping is not arbitrary but precisely adapted to the time-sharing multiplexing mechanism of the specific hardware memory access channel. This solves the memory access serialization problem caused by the mismatch between the hardware memory access channel and the number of threads, effectively triggering the hardware's time-sharing efficient memory access mode, thereby significantly improving memory bandwidth utilization and achieving improved memory access performance. Furthermore, this invention not only creates optimized layout codes but also updates the tensor memory layout associated with all operations in the entire data flow to be optimized based on these codes. This ensures the consistency and correctness of the optimized layout across the entire data flow, avoiding data errors or type mismatches caused by local layout changes. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the tensor layout optimization method provided by the present invention; Figure 2 This is a flowchart illustrating the tensor layout optimization method for a specific GPU architecture provided by the present invention. Figure 3 This is a schematic diagram of the tensor layout optimization device provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] In the training and inference processes of deep learning models, large-scale tensor computation is a core component. To improve computational efficiency and fully utilize the powerful parallel processing capabilities of computing hardware, the industry has developed various tensor computation optimization techniques. Here, computing hardware can refer to hardware accelerators with parallel computing capabilities, such as GPUs (Graphics Processing Units), GPGPUs (General-purpose computing on Graphics Processing Units), and TPUs (Tensor Processing Units). For ease of understanding, the following discussion will primarily focus on GPUs as the computing hardware.

[0022] Current mainstream tensor computation optimization techniques include static partitioning, compiler-automated optimization, and hardware-specific instructions. Among these, static partitioning is a classic optimization technique commonly found in low-level computing libraries. It decomposes large tensor operations (such as matrix multiplication) into a series of subtasks performed on small data blocks of a preset size. These partitioning strategies are usually fixed at compile time or during library development, aiming to improve data reuse and memory access efficiency by maximizing the utilization of the GPU's memory hierarchy (such as shared memory).

[0023] Compiler-based automatic optimization techniques, exemplified by modern compiler frameworks such as TVM (Tensor Virtual Machine) and MLIR (Multi-Level Intermediate Representation), transform the optimization process from fixed, universal strategies to automated searches tailored to specific models, input sizes, and hardware platforms. The compiler constructs a vast search space for optimization strategies and explores and generates optimal computational kernel code through cost model predictions or actual measurements, achieving end-to-end automated performance tuning.

[0024] Hardware-specific instruction techniques are optimizations that directly utilize dedicated hardware units on the GPU designed for specific computations. For example, Tensor Cores can efficiently perform mixed-precision MMA (Matrix-Multiply-Accumulate) operations, replacing a large number of general-purpose computational instructions with a single hardware instruction, thereby greatly improving computational throughput.

[0025] However, the aforementioned technologies have certain limitations in practical applications. These technologies are typically designed based on general hardware architecture assumptions and fail to be fully optimized for the hardware characteristics of certain special GPU architectures, especially in memory access mechanisms. For example, in a general GPU architecture, when 32 threads within a warp access memory, there are usually 32 physical memory access channels corresponding to them, enabling fully parallel processing of the memory access requests from the 32 threads.

[0026] However, in certain GPU architectures, the hardware design may differ; for example, the number of physical memory access channels may be less than the number of threads within a single thread bundle. Taking a GPU architecture with 16 physical memory access channels as an example, when a thread bundle containing 32 threads simultaneously initiates memory access requests, the insufficient number of memory access channels prevents these requests from being processed in parallel. The hardware must process these requests in batches and serially through these 16 channels, leading to increased memory access latency, severely insufficient memory bandwidth utilization, and ultimately, lower-than-expected overall computational performance. The root cause is that the memory access patterns generated by current tensor optimization techniques cannot effectively trigger the efficient memory access mechanisms (such as time-sharing multiplexing of channels) of this specific hardware architecture, thus creating a performance bottleneck.

[0027] To address this issue, this invention provides a tensor layout optimization method, aiming to solve the problem that the aforementioned optimization techniques do not fully consider specific hardware architectures, resulting in low memory access efficiency on certain GPUs (especially GPUs with fewer physical memory access channels than the number of threads within a thread bundle). This invention optimizes the memory layout of tensors at the software level, forcing the generation of a memory access mode that maximizes the utilization of hardware parallel memory access capabilities, thereby significantly improving memory bandwidth utilization and achieving improved memory access performance. This invention has strong specificity and practical value, significantly improving the execution efficiency of computational tasks such as deep learning on these specific hardware platforms.

[0028] It's important to note that tensor layout optimization refers to the process of altering the logical organization of tensor data in memory and its mapping to GPU execution threads during compilation. It doesn't change the actual physical storage location of the tensor data, but rather modifies the metadata associated with the tensor (i.e., layout encoding), thereby guiding the compiler to generate different memory access instructions. The goal of optimization is to maximize the utilization of the GPU's parallel processing capabilities, particularly memory bandwidth, through the memory access patterns produced by the generated instructions.

[0029] It is understood that the tensor layout optimization method provided by this invention can be applied to GPU-accelerated scenarios in deep learning training or inference frameworks. In deep learning model training or inference tasks, the computational task is typically represented as a computation graph, which consists of a series of operations and tensors representing data. To execute efficiently on parallel computing hardware such as GPUs and GPGPUs, the computation graph needs to be compiled and optimized. The method of this invention adjusts the memory layout of tensors during this compilation and optimization stage, taking into account the characteristics of specific hardware, to generate more efficient execution code.

[0030] The execution entity of this method can be the host running the compiler, such as the CPU (Central Processing Unit) of a server. When the compiler receives a computation graph to be optimized, it executes the method provided by this invention, which analyzes the operations in the computation graph, logically reconstructs the shape and layout of the tensor, and finally generates machine instructions adapted to the specific hardware. These instructions are loaded onto the specific hardware (such as a GPU with fewer physical memory access channels than the number of threads in a thread bundle) for execution. Since the memory layout of the tensor has been optimized according to the hardware characteristics during the compilation stage, this guides the compiler to generate a specific sequence of memory access instructions. This sequence enables threads within a thread bundle to access different memory regions in groups. This structured access pattern can be recognized by the hardware and trigger the time-sharing mechanism of its memory access channels, thereby achieving efficient memory access and bandwidth utilization.

[0031] Figure 1 This is a flowchart illustrating the tensor layout optimization method provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step S10: Obtain the target operation in the computation graph, wherein the target operation is a memory access operation that satisfies a first preset condition.

[0032] Specifically, a computation graph is an intermediate representation of a deep learning model, describing how data (tensors) flows through a series of computational or memory access operations. In the fields of deep learning and high-performance computing, a complex computational task is often represented as a directed acyclic graph (DAG), where the nodes represent the various operations in the computation graph, such as matrix multiplication, convolution, addition, and other mathematical operations or memory access operations; and the edges represent the data flowing between operations, i.e., tensors. Therefore, a computation graph completely describes the data flow and dependencies of the entire computational task.

[0033] To obtain the target operations in the computation graph, the compiler traverses all operation nodes and examines each operation to identify memory access operations that meet specific conditions (i.e., a first preset condition). This allows the compiler to determine one or more target operations that require tensor layout optimization. Here, memory access operations refer to those that transfer data between different levels of storage in the GPU, such as reading and writing data between the GPU's global memory and registers or shared memory within the EU (Execution Unit). The execution efficiency of these operations is directly constrained by memory bandwidth and represents a potential performance bottleneck.

[0034] The first precondition refers to a set of rules used to precisely filter out memory access operations from a large pool of operations that require tensor layout optimization. Not all memory access operations are suitable for or require tensor layout optimization; this condition serves as an initial filter to ensure that optimization brings benefits without negative impacts. For example, this condition could stipulate that optimization is only meaningful when the amount of data involved in the memory access operation reaches a certain scale. If the data volume is too small, the overhead of complex layout transformations may outweigh its performance gains.

[0035] Step S20: Based on the computation graph, obtain all associated operations that have data dependencies on the target operation, and if each associated operation satisfies the second preset condition, determine the data flow including the target operation and each associated operation as the data flow to be optimized.

[0036] It's important to note that after identifying a potential target operation, you cannot simply change the layout of the tensor associated with that operation in isolation. Because a tensor may be used by multiple operations, changing its layout will affect all operations that use it. Therefore, tracing and propagation analysis along the data flow is necessary.

[0037] Specifically, for each target operation obtained in step S10, all associated operations that have data dependencies on the target operation can first be obtained based on the computation graph. Here, data dependency means that in the computation graph, the output tensor of one operation is the input tensor of another operation. Specifically, to obtain all associated operations, one can start from the target operation and traverse the entire computation graph forward and backward to find all associated operations on the same data flow as the target operation. It should be understood that all associated operations that have data dependencies on the target operation refer to all other operations in the computation graph that are directly or indirectly connected to the target operation through tensors (data edges).

[0038] Subsequently, each associated operation on this data stream is checked to determine if it meets the second pre-defined condition. Here, the second pre-defined condition is a verification constraint imposed on all these associated operations. The core purpose of this condition is to ensure that all operations on the entire data stream are compatible with the upcoming layout transformation, thus guaranteeing the feasibility of the transformation. For example, this condition could be similar to the first pre-defined condition, requiring that the tensors processed by the associated operations also meet a certain data size requirement. If any operation in the data stream does not meet this condition, it means that the overall optimization of the data stream is infeasible. In this case, the compiler will abandon the optimization of the initial target operation to ensure the correctness of the computation.

[0039] It can be understood that the data flow, including the target operation and all related operations, can be viewed as a subgraph of the computation graph. It consists of a connected path formed by a target operation that has passed the initial screening and all its related operations with which it has data dependencies. Only when all operations in the entire data flow pass the check (i.e., the target operation satisfies the first preset condition, and all its related operations satisfy the second preset condition) is this data flow identified as the data flow to be optimized. Here, the data flow to be optimized is the execution object of subsequent optimization steps; it represents a verified set of operations suitable for overall layout transformation.

[0040] Step S30: Based on the memory access channel characteristics of the computing hardware, a target layout code is created. The target layout code is used to divide multiple threads within a thread bundle into at least two thread groups so that different thread groups can time-division multiplex the memory access channels of the computing hardware.

[0041] Specifically, the aforementioned computing hardware refers to the specific GPU device targeted by this compilation task. This embodiment of the invention primarily targets GPU architectures with specific hardware characteristics, where the memory access channel characteristics of the computing hardware represent such specific hardware limitations. These limitations manifest as physical memory access channel count restrictions, standard memory access mode bottlenecks, and special high-efficiency memory access modes.

[0042] The physical memory access channel limit refers to the fact that the number of physical memory access channels of the computing hardware's memory controller is less than the number of threads in a thread bundle. For example, a GPU may only have 16 physical parallel data channels, while a thread bundle includes 32 threads. The standard memory access mode bottleneck occurs when 32 threads in a thread bundle initiate 32 concurrent memory access requests to contiguous addresses; these 16 channels cannot process these requests in parallel, resulting in half the actual bandwidth. A special high-efficiency memory access mode refers to the hardware's memory access channels supporting time-sharing multiplexing. To trigger this mechanism, the software needs to generate a specific memory access mode, organizing the requests from the 32 threads in a thread bundle into two independent 16-thread requests, each targeting a different cache line (e.g., two adjacent 64-byte blocks). When the hardware detects this mode, it can execute efficiently in a pipelined manner, achieving an equivalent memory access bandwidth of 128 bytes. It should be understood that, assuming that each memory access channel on the GPU processes data with a bit width of 32 bits (i.e., 4 bytes), then the data block size processed in parallel by 16 channels is 64 bytes. If 32 threads in a Warp time-share memory access channels, the data block size that can be accessed simultaneously is 128 bytes.

[0043] Based on the memory access channel characteristics of the aforementioned computing hardware, target layout codes can be created. The purpose of these codes is to generate memory access patterns that can trigger time-sharing mechanisms. Here, the process of creating target layout codes involves translating these characteristics into instructions that the compiler can understand and execute. Target layout codes are a set of metadata or attributes attached to the tensor type definition. They do not change the actual data content of the tensor, but rather describe to the encoder how threads should be organized to process this data.

[0044] To achieve the above objectives, a key function of this layout coding is to divide multiple threads within a thread bundle into at least two thread groups. For example, a thread bundle containing 32 threads can be logically divided into two thread groups, each containing 16 threads. Thus, when the compiler generates memory access instructions, the 16 threads in the first thread group will access the first 64-byte cache line, while the second thread group will access another adjacent 64-byte cache line. This carefully constructed memory access pattern in software precisely triggers the time-sharing multiplexing mechanism of the memory access channels on the computing hardware, thereby doubling the memory access bandwidth.

[0045] Step S40: Based on the target layout encoding, update the memory layout of the tensors associated with each operation in the data stream to be optimized.

[0046] Specifically, after creating the new target layout code, it can be applied to the data stream to be optimized as determined in step S20. In other words, based on the created target layout code, the memory layout of the tensors associated with each operation in the data stream to be optimized is updated. Here, the tensors associated with each operation in the data stream to be optimized refer to all tensors that serve as inputs or outputs of operations in the data stream to be optimized. Because these operations are a link in a data dependency chain, when the layout of one tensor is changed, in order to maintain the consistency of the data format, the memory layout of all related tensors in the entire data stream must be changed simultaneously.

[0047] Tensor memory layout is a logical concept that describes how the multidimensional data elements of a tensor are mapped to the one-dimensional linear address space of memory, and how GPU execution threads are grouped and mapped onto these data elements. This invention alters this logical layout, rather than physically rearranging the data in memory.

[0048] Specifically, to update the memory layout of tensors based on the target layout encoding, the compiler iterates through each operation in the data stream to be optimized. For each operation, the compiler obtains the tensors corresponding to its operands (i.e., inputs) and return value (i.e., outputs). Each tensor has type information. This type information contains attributes describing the tensor's shape, data type (e.g., float32), and memory layout. By updating the type information of the tensors associated with these operations to a new type containing the newly created target layout encoding, the subsequent compiler backend, when generating the final machine instructions, will calculate memory addresses and generate memory access instructions based on this new layout information. The generated instructions naturally achieve the optimization goal of grouping threads and accessing memory in a time-sharing multiplexing mode.

[0049] Understandably, in traditional tensor computation optimization techniques, because no optimization is made for the hardware characteristics of GPUs with special architectures, the hardware is forced to split a memory access instruction into multiple serial memory access transactions during execution. However, the embodiments of the present invention optimize the memory layout of tensors so that the memory access request of a thread bundle can be recognized by the hardware as one or a few merged memory access transactions, thereby completing them efficiently.

[0050] For example, in a traditional tensor computation optimization scenario, 32 threads (e.g., T0~T31) within a thread bundle might access 32 logically consecutive but physically scattered data points. When the GPU's memory controller detects that these 32 access requests cannot be merged, it has no choice but to force serialization due to having only 16 physical memory access channels. That is, the scheduler selects the requests from the first 16 threads (e.g., T0-T15) to acquire data through the 16 physical channels; after the first memory access transaction is completed, the scheduler selects the requests from the next 16 threads (e.g., T16-T31) to acquire data again through the same 16 physical channels. Clearly, this is a passive serialization caused by hardware bottlenecks. The entire thread bundle must wait two or more clock cycles to complete the reading of all data, doubling the memory access latency and resulting in low bandwidth utilization.

[0051] The embodiments of this invention modify the layout of tensors in memory, moving beyond a simple row-major or column-major order to a rearranged layout. The goal of this layout is to ensure that when the 32 threads within a thread bundle access data in indexed order, the memory addresses they access are efficiently processed by the hardware. In other words, through this optimized layout, the thread bundle's access pattern becomes highly hardware-friendly.

[0052] In practical implementation, the 32 threads can be logically divided into two groups (e.g., T0-T15 and T16-T31). Through careful memory layout, the 16 data blocks accessed by the first group of threads (T0-T15) are physically contiguous, filling a 64-byte memory segment (or cache line). Similarly, the second group of threads (T16-T31) also accesses another physically contiguous cache line. When the 32 threads initiate memory access simultaneously, the hardware memory controller recognizes these as two merged access requests that can be processed in a highly parallel or pipelined manner. Since each request is merged, the GPU can complete the access of a group of 16 threads in a single memory transaction using an extremely wide load instruction. Although they appear to be two groups, they can be efficiently scheduled by the hardware as a whole, almost like a single, ultra-wide access. It should be understood that the optimization strategy in this embodiment is a proactive optimization designed to cater to hardware characteristics. Logical grouping is to construct a hardware-supported merged access mode, ultimately achieving physical merged memory access. This transforms an operation that would otherwise be split into two serial accesses by the hardware into one or two highly optimized parallel accesses, greatly improving memory bandwidth utilization.

[0053] The method provided in this invention identifies memory access operations in the computation graph as target operations and analyzes all related operations that have data dependencies on the target operations. This determines the data flow to be optimized, ensuring that subsequent optimization operations target memory-intensive critical paths most likely to cause bottlenecks. Then, based on the specific memory access channel characteristics of the computing hardware, a novel target layout code is created to divide a thread bundle into at least two thread groups. This thread grouping is not arbitrary but precisely adapted to the time-sharing multiplexing mechanism of the specific hardware memory access channel. This solves the memory access serialization problem caused by the mismatch between the hardware memory access channel and the number of threads, effectively triggering the hardware's time-sharing efficient memory access mode, thereby significantly improving memory bandwidth utilization and enhancing memory access performance. Furthermore, this invention not only creates an optimized layout code but also updates the tensor memory layout associated with all operations in the entire data flow to be optimized based on this code. This ensures the consistency and correctness of the optimized layout across the entire data flow, avoiding data errors or type mismatches caused by local layout changes.

[0054] Based on the above embodiments, the first preset condition includes that the target operation is at least one of memory loading, memory storage, and atomic read-write operations, and the data volume of the tensor associated with the target operation is not less than a preset data volume threshold.

[0055] Specifically, the first precondition includes two aspects: operation type condition and data volume condition, and both aspects must be satisfied simultaneously. The operation type condition refers to the requirement that the operation must be a memory access operation, such as a load operation (e.g., LoadOp, ULoadOp) that loads data from global memory into a register or shared memory, or a store operation (e.g., StoreOp, UStoreOp) that writes the calculation result back to global memory from a register, as well as operations that require atomic read-modify-write operations (AtomicRMWOp). It should be understood that an atomic read-modify-write operation is a composite memory operation that guarantees that the three steps of reading a value, modifying that value, and writing the new value back are executed as an indivisible whole.

[0056] The data volume condition refers to the fact that the data volume of the tensor associated with the operation is not less than a preset data volume threshold. Here, the data volume of the tensor associated with the operation refers to the total amount of data that the operation needs to process at one time, usually in bytes. This data volume is determined by the shape of the tensor and the data type of its elements. For example, a one-dimensional tensor with shape

[256] , if its data type is a 32-bit floating-point number (float32, i.e., 4 bytes), then its data volume is 256 × 4 = 1024 bytes.

[0057] The preset data volume threshold is a pre-defined critical value (e.g., 128 bytes) used to determine whether it is worthwhile to activate the optimization strategy of this invention. The reason for setting this threshold is that the optimization mechanism (time-division multiplexing) of this invention is designed to solve the bandwidth bottleneck when transmitting large amounts of data. If the amount of data involved in the operation is very small (e.g., only 64 bytes), the existing hardware channel can already meet the requirements in one go, and there is no need for time-division multiplexing optimization. Forcibly changing the layout may introduce unnecessary overhead.

[0058] It is understandable that the preset data volume threshold is not arbitrarily set, but is closely related to the memory access channel characteristics of the computing hardware. According to the specific GPU architecture targeted in this embodiment of the invention, if a thread bundle (containing 32 threads) wants to fully utilize its 16 physical memory access channels through time-sharing multiplexing, it needs to process enough data to fill two independent memory access transactions at once. Assuming that 32-bit (4-byte) data is being processed, the hardware can process 16 × 4 = 64 bytes in parallel through 16 channels at a time. To trigger time-sharing multiplexing and achieve the theoretical peak bandwidth, the data volume of a single memory access operation should constitute at least two such transactions, i.e., 2 × 64 = 128 bytes. Therefore, in this embodiment of the invention, the preset data volume threshold can be specifically set to 128 bytes.

[0059] For each memory access operation in the computation graph, if the data size of its associated tensor is not less than (i.e., greater than or equal to) a preset data size threshold, it means that the data scale processed by the memory access operation has reached the minimum scale that can trigger a hardware memory access bottleneck and benefit from the time-sharing optimization strategy of this invention. In this case, the memory access operation can be identified as the target operation to be optimized. If the data size is less than the preset data size threshold, it means that the hardware's native physical memory access channel can complete the task efficiently in one go, without the need for complex thread grouping and layout transformation. In this case, optimization of the memory access operation should be abandoned.

[0060] Based on any of the above embodiments, each associated operation includes at least one of the direct predecessor operation, direct successor operation, indirect predecessor operation, and indirect successor operation of the target operation.

[0061] Specifically, in determining the data flow to be optimized, this embodiment of the invention comprehensively examines all associated operations that have data dependencies on the target operation and applies a strict check condition (i.e., a second preset condition). In a computation graph, the data dependencies between operations constitute a data flow. For a target operation, its associated operations may include direct predecessor operations, direct successor operations, indirect predecessor operations, indirect successor operations, etc.

[0062] Here, a direct predecessor operation refers to an operation whose output tensor is directly used as input by the target operation. A direct successor operation refers to an operation that directly uses the output tensor of the target operation as input. An indirect predecessor operation is an operation upstream in the data stream, whose output data reaches the target operation after passing through one or more intermediate operations; it is the predecessor of the target operation's predecessor. An indirect successor operation is an operation downstream in the data stream, whose data comes from the successor of the target operation; it is the successor of the target operation's successor.

[0063] Specifically, for each target operation, the method to obtain all its associated operations is to start from the target operation node and perform a graph traversal algorithm (such as breadth-first search or depth-first search) on the computation graph. Traversing along the input edge direction (against the data flow direction) can find all direct and indirect predecessor operations; traversing along the output edge direction (up with the data flow direction) can find all direct and indirect successor operations. All these traversed operations together constitute the set of associated operations.

[0064] Accordingly, the method further includes: If any associated operation does not meet the second preset condition, layout optimization of the tensor associated with the target operation is abandoned. The second preset condition is that the data volume of the tensor associated with the associated operation is not less than a preset data volume threshold.

[0065] It should be noted that the embodiments of the present invention introduce a key protection mechanism, namely, for each target operation, when any of its associated operations does not meet the second preset condition, the layout optimization of the tensor associated with the target operation is abandoned, thereby avoiding the overall performance degradation or logical errors caused by local optimization.

[0066] Specifically, the second preset condition is similar to the first preset condition, namely, the amount of data in the tensor processed by the associative operation is not less than a preset data size threshold (e.g., 128 bytes). When all associative operations satisfy the second preset condition, it means that the data scale is large enough to support and benefit from the optimization strategy of this invention throughout the entire chain from data generation (source operation) to final use (end operation). If an operation occurs midway and the amount of data it processes suddenly drops below the threshold, then that operation becomes the breakpoint for optimization. Forcibly performing a uniform layout transformation on the entire data stream will cause a mismatch at this breakpoint, which may require the insertion of additional, costly layout transformation operations, or even lead to compilation errors. Therefore, if any associative operation does not satisfy the second preset condition, the optimization of the current target operation is abandoned to ensure global correctness and performance.

[0067] Based on any of the above embodiments, the attributes of the target layout encoding include at least one of the following: the number of data elements processed by a single thread, the thread distribution within a thread bundle, and the dimension processing order.

[0068] Specifically, target layout encoding is a set of properties or metadata associated with tensor types at the compiler's intermediate representation level. It provides the compiler with detailed guidance on how to organize threads to process data and how the data is logically arranged. Common properties include the number of data elements processed by a single thread (corresponding to the property field sizePerThread), the thread distribution within a thread (corresponding to the property field threadsPerWarp), and the dimension processing order (also known as the dimension processing priority, corresponding to the property field order).

[0069] Here, the thread distribution within a thread (threadsPerWarp) defines how all threads within a thread bundle are logically organized. To trigger the 16-channel time-division multiplexing mechanism of specific hardware, embodiments of this invention can modify this attribute from the default one-dimensional distribution (e.g.,

[32] ) to a two-dimensional distribution, such as [2,16] or [16,2]. This attribute setting logically divides the 32 threads within a warp into two thread groups, each containing 16 threads.

[0070] The `sizePerThread` property defines the number and arrangement of data elements processed by a single thread in a single operation. It specifies the finest-grained unit of work; for example, `sizePerThread=[1,2]` means each thread is responsible for processing a 1×2 data block (i.e., 2 elements). This property, together with `threadsPerWarp`, determines the total size of the data block processed by a warp in a single operation.

[0071] The dimension processing order (order) is used to specify the priority of these dimensions when they are mapped to one-dimensional linear memory, i.e., row-major or column-major, when threads and data are organized into a multi-dimensional logical structure. For example, when threadsPerWarp=[2,16], there are two logical dimensions (dimension 0 of size 2 and dimension 1 of size 16), and order can be set to [1,0]. Because order=[1,0] indicates that dimension 1 (i.e., the dimension of size 16) is the priority dimension, this ensures that the 16 threads in the same thread group are logically arranged along the most contiguous direction of memory. This is crucial for generating merged memory accesses. Only when 16 threads in a group access a contiguous 64-byte block of memory can the hardware process it as a single, efficient memory access transaction. Similarly, when threadsPerWarp=[16,2], the corresponding order attribute should be [0,1].

[0072] In this embodiment of the invention, by coordinating the configuration of the above attributes, a highly customized target layout code can be created. This code not only meets the requirements of hardware time-sharing multiplexing, but also ensures the merging and efficiency of memory access.

[0073] Based on any of the above embodiments, step S30 specifically includes: Based on the memory access channel characteristics of the computing hardware, the target layout code is created, and the attribute in the target layout code used to define the thread distribution within the thread bundle is set to a dimension corresponding to the at least two thread groups.

[0074] Specifically, when creating the target layout code based on the memory access channel characteristics of the computing hardware, the most crucial aspect is setting the thread distribution attribute within the thread bundle. The attribute used to define the thread distribution within the thread bundle refers to the `threadsPerWarp` attribute. The default value of this attribute is typically a one-dimensional array. For example, for a thread bundle (Warp) containing 32 threads, its default distribution might be

[32] , indicating that all threads are linearly arranged. This embodiment of the invention no longer uses this default distribution but instead resets it based on the memory access channel characteristics of the computing hardware.

[0075] As described in the foregoing embodiments, the specific GPU hardware targeted by this invention has 16 physical memory access channels, but it employs a time-sharing multiplexing mechanism. This mechanism enables the processing of two independent memory access requests from 16 threads targeting different cache lines, achieving an equivalent memory access bandwidth of 128 bytes. To trigger this mechanism, the software layer needs to precisely divide the 32 threads of a warp into two thread groups, each with 16 threads. This invention achieves this by setting the `threadsPerWarp` attribute to a two-dimensional dimension corresponding to these two thread groups. Specifically, it can be set to a two-dimensional array [2, 16] (or [16, 2] under different dimensional priorities).

[0076] Here, the two-dimensional array [2, 16] refers to logically organizing the 32 threads within a warp into a 2x16 logical grid, rather than treating them as a linear sequence. Dimension 2 corresponds to the two thread groups to be divided in this invention. Dimension 16 corresponds to the number of threads in each thread group. When the compiler reads this new layout encoding of threadsPerWarp=[2, 16] during the subsequent code generation stage, it will generate a new address calculation logic and memory access instructions. These instructions will cause the 16 threads in the first row (group 0) of the logical grid to collaboratively access the first 64-byte memory block (e.g., a cache line), and the other 16 threads in the second row (group 1) of the logical grid to collaboratively access the next adjacent 64-byte memory block (another cache line).

[0077] The method provided in this invention deeply binds the logical distribution of thread bundles with the physical characteristics of hardware by modifying the dimension of the key attribute threadsPerWarp, thereby enabling the software to proactively and precisely guide the hardware into the most efficient working mode.

[0078] Based on any of the above embodiments, after step S30, the method further includes: Determine whether the tensor associated with each operation in the data stream to be optimized is a one-dimensional tensor; If so, then based on the dimension of the thread distribution in the target layout encoding, the one-dimensional tensor is reconstructed into a multi-dimensional tensor, and the logical dimension of the multi-dimensional tensor matches the dimension of the thread distribution.

[0079] It should be noted that, in order to adapt to the memory access characteristics of specific hardware, this invention modifies the thread logic distribution of a warp from one-dimensional (e.g.,

[32] ) to two-dimensional (e.g., [2,16]). However, in actual computation graphs, many tensors, especially those flattened arrays representing weight biases or intermediate activation values, may originally have a one-dimensional shape (e.g., shape [N]). This creates a potential conflict: using a two-dimensional thread grid to process a one-dimensional data array. This dimensionality mismatch can make it very complex for the compiler to generate address calculation instructions, or even prevent it from generating efficient code.

[0080] To address this issue, this embodiment of the invention adds further judgment and processing steps after creating the target layout code, specifically as follows: First, it determines whether the tensors associated with each operation in the data stream to be optimized are one-dimensional tensors. This can be achieved by checking the shape defined in the type information of each tensor in the data stream to be optimized. If the shape of a tensor is an array containing only a single element, such as

[1024] , then it is identified as a one-dimensional tensor.

[0081] If it is a one-dimensional tensor, its logical shape must be reconstructed. This embodiment of the invention reconstructs a one-dimensional tensor into a multi-dimensional tensor based on the dimension of thread distribution in the target layout encoding. The core principle is that the logical dimension of the multi-dimensional tensor matches the dimension of the thread distribution.

[0082] This means that if threadsPerWarp is set to two dimensions [2, 16], then the corresponding one-dimensional tensor must also be logically reconstructed into a two-dimensional tensor. The purpose of this is to ensure that the logical layout of the data is consistent with the physical layout of the threads, allowing the compiler to generate the simplest and most efficient address calculation instructions. A thread can directly and easily calculate the data address Tensor[row][col] in the two-dimensional logical tensor it needs to access using its coordinates (row, col) in the two-dimensional logical grid, without complex conversions.

[0083] Furthermore, the reconstructing of the one-dimensional tensor into a multi-dimensional tensor includes: When the dimension of the thread distribution is two-dimensional, the one-dimensional tensor is reconstructed into a two-dimensional tensor. The dimensions of the two-dimensional tensor include a first dimension and a second dimension. The size of the first dimension is determined based on the shape of the one-dimensional tensor and a preset data volume threshold. The size of the second dimension is determined based on the preset data volume threshold. The preset data volume threshold is determined based on the amount of data processed in parallel by threads within a thread bundle in a memory access transaction.

[0084] Specifically, in the specific hardware scenario addressed by this invention, the thread distribution is two-dimensional (i.e., threadsPerWarp=[2,16]). Therefore, the one-dimensional tensor will be reconstructed into a two-dimensional tensor. The rules for determining the dimensions (including the first and second dimensions) of this new two-dimensional tensor are as follows: The size of the second dimension is determined based on a preset data volume threshold. As described in the previous embodiment, this threshold itself is determined based on the amount of data that threads within a thread bundle can process in parallel within a memory access transaction (using a time-sharing mechanism). For example, the size of the second dimension can be 128. The value of this dimension is fixed, and it represents the size of the data block that can be processed in one optimized memory access operation.

[0085] The size of the first dimension is determined based on the original shape of the one-dimensional tensor and the aforementioned preset data size threshold. Its calculation formula is ceil(N / 128), where N is the total number of elements in the original one-dimensional tensor, and ceil is the floor function. This dimension represents how many data blocks of size 128 are needed to completely cover the original N elements.

[0086] For example, suppose the data stream to be optimized contains a one-dimensional tensor of shape

[1000] with a 32-bit floating-point data type. To determine if this tensor is one-dimensional, it needs to be reconstructed into a two-dimensional tensor. The second dimension of this two-dimensional tensor is fixed at 128, and the first dimension is ceil(1000 / 128) = 8. Therefore, this one-dimensional tensor is logically reconstructed into a two-dimensional tensor of shape [8, 128]. It should be understood that this is a logical reconstruction; the new two-dimensional tensor contains a total of 8 × 128 = 1024 elements, more than the original 1000 elements. For accesses outside the original data range (i.e., elements with indices 1000 to 1023), the compiler handles this using a mask mechanism when generating code. This means that threads assigned to handle these out-of-bounds elements will be set to an inactive state; they will not perform actual memory read / write operations, thus ensuring the correctness of the calculation.

[0087] Based on any of the above embodiments, step S40 specifically includes: Step S41: Traverse each operation in the data stream to be optimized; Step S42: For the current operation that has been traversed, update the operand and return value type information of the current operation based on the target layout encoding.

[0088] Specifically, updating the tensor's memory layout according to the target layout encoding is an operation performed at the compiler's intermediate representation level. In detail, the compiler first traverses each operation in the data stream to be optimized. This traversal means that the compiler visits each operation node in the data stream sequentially in a certain order (e.g., topological order or code order).

[0089] For each operation encountered, the compiler needs to modify the description information of the input and output tensors. Specifically, for each operation, the compiler obtains the tensors corresponding to its operands and return values. Each tensor has type information, which encapsulates the tensor's shape, data type, and key memory layout. The compiler replaces the old type information of these tensors with new type information containing the encoding of the newly created target layout. For example, the layout attributes of a tensor are updated from the default one-dimensional memory access mode to a new layout containing attributes such as threadsPerWarp=[2,16]. It should be understood that the operands mentioned above refer to the input data required for the execution of an operation, and the return value refers to the output data produced after the execution of an operation.

[0090] Understandably, the above update is a purely logical one. The bit sequence of the tensor data in memory does not change. However, this update is crucial for the compiler's subsequent code generation phase. When the code generator processes this updated operation, it reads the new layout encoding and generates entirely different, optimized machine instructions based on it. These new instructions guide the GPU threads to execute efficient memory access operations that trigger hardware time-sharing, grouped in a [2,16] manner.

[0091] Based on any of the above embodiments Figure 2 This is a flowchart illustrating the tensor layout optimization method for a specific GPU architecture provided by the present invention, as shown below. Figure 2 As shown, the method includes the following steps: Step S1: Identify operations that can be optimized.

[0092] Specifically, the compiler first obtains the entire computation graph to be optimized and initializes a traversal process to check each operation in the computation graph one by one. In the main loop, it first determines whether the currently traversed operation is an optimizable operation. This is an initial screening process. This check verifies whether the current operation is a memory access operation (such as LoadOp, StoreOp, AtomicRMWOp, ULoadOp, UStoreOp, etc.) and whether the number of data elements of its associated tensor is greater than or equal to a preset threshold (such as 128 bytes).

[0093] If the result is negative, it means the current operation does not meet the optimization conditions. In this case, the optimization of the current operation will be abandoned, and the process will continue to traverse the next operation in the computation graph. If the result is positive, the current operation is considered a potential optimization target, and the process proceeds to the next step.

[0094] Step S2: Collect the upstream and downstream operations of the operation to be optimized and verify the consistency of the data flow.

[0095] Specifically, for target operations that pass the initial screening, the next step is to perform upstream and downstream verification. This involves traversing all direct predecessors and successors of the current target operation, as well as all indirect predecessors and successors, and determining whether any of these upstream and downstream operations (also known as associated operations) are unoptimizable. The system will trace back and forward along the data dependencies, and determine whether each associated operation meets the second preset condition (i.e., whether the number of data elements in its associated tensor is greater than or equal to 128 bytes).

[0096] If there are non-optimizable operations—that is, if any breakpoint in the entire data stream fails to meet the conditions—it means that a complete layout transformation is not feasible. To ensure computational correctness and global performance, the current operation will be abandoned, and the main loop will be returned to process the next operation. If there are no non-optimizable operations, it indicates that the entire data stream originating from the target operation has optimization potential, and this data stream can be identified as the data stream to be optimized. Finally, one or more data streams that can be optimized will be obtained.

[0097] Step S3: Create the optimized layout and update the tensor shape.

[0098] For a given data flow to be optimized, a series of transformation operations are performed: First, a new, optimized layout code is created based on the memory access channel characteristics of the specific hardware. The core of this step is to set the threadsPerWarp attribute to a two-dimensional distribution (e.g., [2,16]), which means that the warp is divided into two groups, each with 16 threads. Each group accesses different cache lines, and the order, sizePerThread, and other attributes are configured in a coordinated manner to construct a memory access mode that can trigger time-sharing multiplexing.

[0099] Next, determine whether the tensors in the data stream are one-dimensional tensors. If not, the original dimension can be preserved; if so, they need to be reconstructed into two-dimensional tensors. For example, the original one-dimensional tensor with shape [N] can be logically reconstructed into [ceil(N / 128),128] to match the logical dimension of the data with the two-dimensional distribution of the threads.

[0100] After completing the layout creation and shape reconstruction, the type information of all tensors in the data stream where the optimizable operations are located can be updated to a new type containing the new layout code and the new shape.

[0101] After transforming a data stream to be optimized, determine whether the traversal has ended. If not, return to the main loop to continue obtaining and processing the next operation in the computation graph; if yes, it means that all operations in the computation graph have been checked and processed, and the entire optimization process ends.

[0102] In this embodiment of the invention, through the above-described systematic process, the part suitable for optimization can be located automatically, safely and efficiently in a complex computation graph, and its layout and shape transformation can be performed precisely and hardware-awarely, thereby making full use of the memory bandwidth of the specific hardware while ensuring correctness.

[0103] The tensor layout optimization apparatus provided by the present invention is described below. The tensor layout optimization apparatus described below can be referred to in correspondence with the tensor layout optimization method described above.

[0104] Based on any of the above embodiments Figure 3 This is a schematic diagram of the tensor layout optimization device provided by the present invention, as shown below. Figure 3 As shown, the device includes: The acquisition unit 310 is used to acquire the target operation in the computation graph, wherein the target operation is a memory access operation that satisfies a first preset condition; The determining unit 320 is used to obtain all associated operations that have data dependencies on the target operation based on the computation graph, and determine the data flow including the target operation and the associated operations as the data flow to be optimized when all associated operations meet the second preset condition; Creation unit 330 is used to create target layout code based on the memory access channel characteristics of computing hardware. The target layout code is used to divide multiple threads in a thread bundle into at least two thread groups so that different thread groups can time-division multiplex the memory access channels of the computing hardware. The update unit 340 is used to update the memory layout of the tensors associated with each operation in the data stream to be optimized based on the target layout encoding.

[0105] The apparatus provided in this invention identifies memory access operations in the computation graph as target operations and analyzes all associated operations that have data dependencies on the target operations. This determines the data flow to be optimized, ensuring that subsequent optimization operations target memory-intensive and most likely bottleneck-prone critical paths. Then, based on the specific memory access channel characteristics of the computing hardware, a novel target layout code is created to divide a thread bundle into at least two thread groups. This thread grouping is not arbitrary but precisely adapted to the time-sharing multiplexing mechanism of the specific hardware memory access channel. This solves the memory access serialization problem caused by the mismatch between the hardware memory access channel and the number of threads, effectively triggering the hardware's time-sharing efficient memory access mode, thereby significantly improving memory bandwidth utilization and enhancing memory access performance. Furthermore, this invention not only creates optimized layout codes but also updates the tensor memory layout associated with all operations in the entire data flow to be optimized based on these codes. This ensures the consistency and correctness of the optimized layout across the entire data flow, avoiding data errors or type mismatches caused by local layout changes.

[0106] Based on any of the above embodiments, the first preset condition includes that the target operation is at least one of memory loading, memory storage, and atomic read-write operation, and the data volume of the tensor associated with the target operation is not less than a preset data volume threshold.

[0107] Based on any of the above embodiments, each associated operation includes at least one of the direct predecessor operation, direct successor operation, indirect predecessor operation, and indirect successor operation of the target operation. Accordingly, the determining unit 320 is further configured to: If any associated operation does not meet the second preset condition, layout optimization of the tensor associated with the target operation is abandoned. The second preset condition is that the data volume of the tensor associated with the associated operation is not less than a preset data volume threshold.

[0108] Based on any of the above embodiments, the creation unit 330 is specifically used for: Based on the memory access channel characteristics of the computing hardware, the target layout code is created, and the attribute in the target layout code used to define the thread distribution within the thread bundle is set to a dimension corresponding to the at least two thread groups.

[0109] Based on any of the above embodiments, the device further includes a reconfiguration unit, the reconfiguration unit being used for: Determine whether the tensor associated with each operation in the data stream to be optimized is a one-dimensional tensor; If so, then based on the dimension of the thread distribution in the target layout encoding, the one-dimensional tensor is reconstructed into a multi-dimensional tensor, and the logical dimension of the multi-dimensional tensor matches the dimension of the thread distribution.

[0110] Based on any of the above embodiments, the reconstruction unit is specifically used for: When the dimension of the thread distribution is two-dimensional, the one-dimensional tensor is reconstructed into a two-dimensional tensor. The dimensions of the two-dimensional tensor include a first dimension and a second dimension. The size of the first dimension is determined based on the shape of the one-dimensional tensor and a preset data volume threshold. The size of the second dimension is determined based on the preset data volume threshold. The preset data volume threshold is determined based on the amount of data processed in parallel by threads within a thread bundle in a memory access transaction.

[0111] Based on any of the above embodiments, the updating unit 340 is specifically used for: Iterate through each operation in the data stream to be optimized; For the current operation that is traversed, the operands and return value type information of the current operation are updated based on the target layout encoding.

[0112] Based on any of the above embodiments, the attributes of the target layout encoding include at least one of the following: the number of data elements processed by a single thread, the thread distribution within a thread bundle, and the dimension processing order.

[0113] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a tensor layout optimization method, which includes: obtaining a target operation in the computation graph, wherein the target operation is a memory access operation that satisfies a first preset condition; based on the computation graph, obtaining all associated operations that have data dependencies on the target operation, and determining the data flow including the target operation and the associated operations as the data flow to be optimized if all associated operations satisfy a second preset condition; creating a target layout code based on the memory access channel characteristics of the computing hardware, wherein the target layout code is used to divide multiple threads within a thread bundle into at least two thread groups so that different thread groups can time-division multiplex the memory access channels of the computing hardware; and updating the memory layout of the tensors associated with each operation in the data flow to be optimized based on the target layout code.

[0114] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the tensor layout optimization method provided by the above methods. The method includes: obtaining a target operation in a computation graph, wherein the target operation is a memory access operation that satisfies a first preset condition; based on the computation graph, obtaining all associated operations that have data dependencies on the target operation, and determining a data flow including the target operation and the associated operations as a data flow to be optimized when all associated operations satisfy a second preset condition; creating a target layout code based on the memory access channel characteristics of the computing hardware, wherein the target layout code is used to divide multiple threads within a thread bundle into at least two thread groups so that different thread groups can time-division multiplex the memory access channels of the computing hardware; and updating the memory layout of the tensors associated with each operation in the data flow to be optimized based on the target layout code.

[0116] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the tensor layout optimization method provided by the above methods. The method includes: obtaining a target operation in a computation graph, the target operation being a memory access operation that satisfies a first preset condition; based on the computation graph, obtaining all associated operations that have data dependencies on the target operation, and, if all associated operations satisfy a second preset condition, determining a data flow including the target operation and all associated operations as a data flow to be optimized; based on the memory access channel characteristics of the computing hardware, creating a target layout code, the target layout code being used to divide multiple threads within a thread bundle into at least two thread groups, so that different thread groups can time-division multiplex the memory access channels of the computing hardware; and based on the target layout code, updating the memory layout of the tensors associated with each operation in the data flow to be optimized.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A tensor layout optimization method, characterized in that, include: Obtain the target operation in the computation graph, wherein the target operation is a memory access operation that satisfies a first preset condition; Based on the computation graph, all associated operations that have data dependencies on the target operation are obtained, and if each associated operation satisfies the second preset condition, the data flow including the target operation and each associated operation is determined as the data flow to be optimized. Based on the memory access channel characteristics of computing hardware, a target layout code is created. The target layout code is used to divide multiple threads within a thread bundle into at least two thread groups so that different thread groups can time-division multiplex the memory access channels of the computing hardware. Based on the target layout encoding, update the memory layout of the tensors associated with each operation in the data stream to be optimized.

2. The tensor layout optimization method according to claim 1, characterized in that, The first preset condition includes that the target operation is at least one of memory loading, memory storage, and atomic read-write operations, and the data volume of the tensor associated with the target operation is not less than a preset data volume threshold.

3. The tensor layout optimization method according to claim 1, characterized in that, Each associated operation includes at least one of the direct predecessor operation, direct successor operation, indirect predecessor operation, and indirect successor operation of the target operation, and the method further includes: If any associated operation does not meet the second preset condition, layout optimization of the tensor associated with the target operation is abandoned. The second preset condition is that the data volume of the tensor associated with the associated operation is not less than a preset data volume threshold.

4. The tensor layout optimization method according to claim 1, characterized in that, The creation of target layout encoding based on the memory access channel characteristics of computing hardware includes: Based on the memory access channel characteristics of the computing hardware, the target layout code is created, and the attribute in the target layout code used to define the thread distribution within the thread bundle is set to a dimension corresponding to the at least two thread groups.

5. The tensor layout optimization method according to claim 4, characterized in that, After creating the target layout code, the method further includes: Determine whether the tensor associated with each operation in the data stream to be optimized is a one-dimensional tensor; If so, then based on the dimension of the thread distribution in the target layout encoding, the one-dimensional tensor is reconstructed into a multi-dimensional tensor, and the logical dimension of the multi-dimensional tensor matches the dimension of the thread distribution.

6. The tensor layout optimization method according to claim 5, characterized in that, The process of reconstructing the one-dimensional tensor into a multi-dimensional tensor includes: When the dimension of the thread distribution is two-dimensional, the one-dimensional tensor is reconstructed into a two-dimensional tensor. The dimensions of the two-dimensional tensor include a first dimension and a second dimension. The size of the first dimension is determined based on the shape of the one-dimensional tensor and a preset data volume threshold. The size of the second dimension is determined based on the preset data volume threshold. The preset data volume threshold is determined based on the amount of data processed in parallel by threads within a thread bundle in a memory access transaction.

7. The tensor layout optimization method according to any one of claims 1 to 6, characterized in that, The step of updating the memory layout of the tensors associated with each operation in the data stream to be optimized based on the target layout encoding includes: Iterate through each operation in the data stream to be optimized; For the current operation that is traversed, the operands and return value type information of the current operation are updated based on the target layout encoding.

8. The tensor layout optimization method according to any one of claims 1 to 6, characterized in that, The attributes of the target layout encoding include at least one of the following: the number of data elements processed by a single thread, the thread distribution within a thread bundle, and the dimension processing order.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the tensor layout optimization method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the tensor layout optimization method as described in any one of claims 1 to 8.

Images