Tensor processing method, tensor processing apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure provide a tensor processing method, a tensor processing apparatus, an electronic device and a storage medium. The method comprises: in response to a first condition being met by a block quantity of a first tensor determined based on a preset block strategy, performing a dimension reconstruction and layout conversion operation based on the first tensor to obtain a second tensor that is arranged continuously in a non-row-major order and meets a target hardware constraint; and calling a target hardware operator interface to perform calculation on the second tensor. The method not only avoids the risk of running failure caused by the dimension parameter exceeding the limit or the memory layout being incompatible with the underlying hardware, but also fully utilizes the hardware acceleration capability, realizes the overall consideration of the running ability, compatibility and calculation performance on the premise of keeping the original dimension calculation semantics unchanged.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence, specifically to a tensor processing method, tensor processing apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of Artificial Intelligence (AI) technology, AI processors have become crucial hardware devices supporting the training and inference of deep learning models. Operators are the basic units for building deep learning models, and their execution efficiency determines the overall performance of the model. Current underlying hardware architectures exhibit significant diversity and heterogeneity. How to ensure computational efficiency while meeting the constraints of the target hardware has become one of the key technical challenges that needs to be addressed. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a tensor processing method, the method comprising: responding to a first tensor whose number of blocks determined based on a preset block strategy satisfies a first condition, performing dimension reconstruction and layout transformation operations based on the first tensor to obtain a second tensor arranged continuously in non-row-major order and satisfying target hardware constraints; and calling a target hardware operator interface to perform calculations on the second tensor.

[0004] In at least one embodiment of the tensor processing method provided in this disclosure, the first condition includes: the number of blocks is greater than a first hardware support threshold.

[0005] In a tensor processing method provided in at least one embodiment of this disclosure, the step of performing dimension reconstruction and layout transformation operations based on the first tensor includes: performing a first type of dimension reconstruction operation and layout transformation operation based on the first tensor to obtain an intermediate tensor; and performing a second type of dimension reconstruction operation on the intermediate tensor to obtain a second tensor.

[0006] The tensor processing method provided in at least one embodiment of this disclosure further includes: determining a target reconstruction dimension based on the first dimension of the first tensor; and determining whether to perform a padding operation based on the first dimension of the first tensor and the target reconstruction dimension.

[0007] In a tensor processing method provided in at least one embodiment of this disclosure, determining the target reconstruction dimension size based on the first dimension size of the first tensor includes: determining a first reconstruction dimension size from the preset reconstruction dimension size set based on the amount of data to be filled corresponding to multiple reconstruction dimension sizes in the preset reconstruction dimension size set; determining a second reconstruction dimension size based on the first dimension size of the first tensor and the first reconstruction dimension size; and determining the target reconstruction dimension size based on the first reconstruction dimension size and the second reconstruction dimension size.

[0008] In at least one embodiment of the tensor processing method provided in this disclosure, the step of performing a first type of dimension reconstruction operation and a layout transformation operation based on the first tensor to obtain an intermediate tensor includes: performing a first type of dimension reconstruction operation on the first tensor; and performing a layout transformation operation on the dimension-reconstructed first tensor to obtain the intermediate tensor.

[0009] In a tensor processing method provided in at least one embodiment of this disclosure, performing a first type of dimension reconstruction operation on a first tensor includes: splitting the first dimension of the first tensor into a first reconstruction dimension and a second reconstruction dimension based on a first reconstruction dimension size and a second reconstruction dimension size.

[0010] In the tensor processing method provided in at least one embodiment of this disclosure, the step of performing a layout transformation operation on the first tensor after dimension reconstruction includes: writing the first tensor after dimension reconstruction into a storage area based on a target physical layout, wherein the target physical layout is a non-row-major continuous arrangement.

[0011] In at least one embodiment of the tensor processing method provided in this disclosure, the step of writing the first tensor after dimension reconstruction based on the target physical layout into the storage area includes: allocating the storage area corresponding to the target physical layout; and copying the first tensor after dimension reconstruction to the storage area through a memory copy operation, so as to convert the first tensor after dimension reconstruction into the target physical layout during the copying process.

[0012] In a tensor processing method provided in at least one embodiment of this disclosure, the step of performing a first type of dimension reconstruction operation and a layout transformation operation based on the first tensor to obtain an intermediate tensor includes: in response to determining to perform a padding operation, performing the padding operation on the first tensor to obtain a third tensor; performing the first type of dimension reconstruction operation on the third tensor; and performing the layout transformation operation on the dimension-reconstructed third tensor to obtain the intermediate tensor.

[0013] In at least one embodiment of the tensor processing method provided in this disclosure, the step of performing the filling operation on the first tensor includes: allocating a contiguous storage region based on the target reconstruction dimension size, copying the first tensor to the contiguous storage region, and filling the tail of the first tensor with a preset filling value in the contiguous storage region.

[0014] In at least one embodiment of the tensor processing method provided in this disclosure, the step of performing a second type of dimension reconstruction operation on the intermediate tensor to obtain the second tensor includes: merging the first reconstruction dimension and the second reconstruction dimension of the intermediate tensor, and performing truncation processing based on the first dimension size of the first tensor to obtain the second tensor, wherein the first dimension size of the second tensor is equal to the first dimension size of the first tensor.

[0015] In at least one embodiment of the tensor processing method provided in this disclosure, the preset block strategy includes: in response to the first dimension size of the first tensor satisfying a second condition, determining a target block size from the preset block size set based on the amount of data to be filled corresponding to multiple block sizes in the preset block size set; determining the number of blocks of the first tensor according to the first dimension size of the first tensor and the target block size, wherein the second condition includes: the first dimension size is greater than a second hardware support threshold.

[0016] The tensor processing method provided in at least one embodiment of this disclosure further includes: in response to the fact that the number of blocks of the first tensor determined based on the preset block strategy does not meet the first condition, the underlying computing framework performs a second layout transformation operation on the first tensor and calls the target hardware operator interface.

[0017] In at least one embodiment of the tensor processing method provided in this disclosure, the target hardware operator interface includes a target hardware matrix multiplication interface, and the step of calling the target hardware operator interface to perform calculation on the second tensor includes: calling the target hardware matrix multiplication interface to perform matrix multiplication calculation on the second tensor.

[0018] At least one embodiment of this disclosure provides a tensor processing apparatus, which includes: a conversion module configured to, in response to a first tensor whose number of blocks, determined based on a preset block strategy, satisfies a first condition, perform dimension reconstruction and layout conversion operations based on the first tensor to obtain a second tensor arranged continuously in non-row-major order and satisfying target hardware constraints; and a calling module configured to call a target hardware operator interface to perform calculations on the second tensor.

[0019] At least one embodiment of this disclosure provides an electronic device including at least one processor and at least one memory, wherein the at least one memory stores program code that, when executed by the at least one processor, causes the at least one processor to perform a tensor processing method according to at least one embodiment of this disclosure.

[0020] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to perform a tensor processing method according to at least one embodiment of this disclosure.

[0021] In the tensor processing method, tensor processing device, electronic device, and storage medium provided in at least one embodiment of this disclosure, before calling the target hardware operator interface, it is proactively determined whether the number of blocks of the first tensor determined based on the preset block strategy exceeds the hardware rearrangement limit. If the limit is exceeded, the dimension reconstruction and layout transformation operations are proactively executed in advance. This proactively avoids the risk of operational failure due to excessive dimension parameters or incompatible memory layouts in the underlying hardware. Furthermore, since the physical layout of the tensor in memory matches the computational unit organization of the underlying hardware, the underlying operators can directly and efficiently read and write according to the hardware's native data access mode, thus fully utilizing the hardware acceleration capabilities. While maintaining the original semantics of external dimension computation, this achieves a comprehensive balance of operability, compatibility, and computational performance. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0023] Figure 1 A flowchart illustrating a tensor processing method provided in at least one embodiment of this disclosure.

[0024] Figure 2A and Figure 2B This is a schematic diagram of a block-based column main sequence arrangement provided for at least one embodiment of the present disclosure.

[0025] Figure 3 A flowchart of another tensor processing method provided for at least one embodiment of this disclosure.

[0026] Figure 4 A flowchart illustrating yet another tensor processing method provided in at least one embodiment of this disclosure.

[0027] Figure 5 This is a schematic block diagram of a tensor processing apparatus provided for at least one embodiment of the present disclosure.

[0028] Figure 6 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0029] Figure 7 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0030] Figure 8 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0032] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0033] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0034] A neural network model can be viewed as a directed acyclic graph (DAG) composed of multiple nodes, where each node corresponds to an operator. Operators in a neural network typically refer to the basic mathematical operations or processes used in the network layers. These operators are used to construct the various layers and components of the network, enabling data transfer, transformation, and computation. Operators are the fundamental building blocks of a network model, defining its structure and computational flow, including inputs, outputs, and intermediate computations. In the model, the connections between operators form a directed graph, reflecting the order of computation for different operations. By combining these operators, complex and powerful neural network models can be constructed to handle various complex tasks and data. Some examples of operators include convolution, pooling, loops, activation functions, normalization, matrix multiplication, and element-wise operations.

[0035] Taking sparse convolution as an example, the activation matrix typically participates in matrix multiplication with a two-dimensional logical shape of M×K, where M represents the height dimension of the activation matrix and K represents the width dimension. The corresponding weight matrix has a shape of K×Cout, where Cout is the number of output channels. The output matrix obtained from the matrix multiplication has a shape of M×Cout.

[0036] Because the underlying hardware imposes dimensional constraints on the shape of the processable two-dimensional matrices, for example, requiring a two-dimensional matrix of shape {H, W} to satisfy H≤Th and W≤Tw, where H represents the height of the matrix, W represents the width of the matrix, and Th and Tw are the upper limit constraints imposed by the underlying hardware on the height H and width W, respectively. Therefore, when the height dimension M of the active matrix is ​​greater than the height constraint value Th, it is necessary to perform block processing on the M dimension. In this embodiment, an N dimension is introduced as the layout dimension generated by performing a block operation on the M dimension, and the value of the N dimension represents the number of sub-blocks obtained after dividing the M dimension.

[0037] When performing such operations, the underlying computing framework (such as the underlying operator library) typically defaults to performing a layout transformation adapted to the underlying hardware acceleration features based on the logical shape of the input, and then directly calls the computing interface provided by the target hardware to complete the computation task. However, this default logic has some drawbacks: when the M dimension is large (e.g., M>Th), the underlying computing framework needs to perform block processing. The resulting N dimension (i.e., the number of sub-blocks obtained by dividing the M dimension) may exceed the maximum threshold Tn allowed by the hardware data rearrangement unit, on-chip buffer, or matrix multiplication backend. This can cause the layout transformation adapted to the underlying hardware acceleration features or the subsequent matrix multiplication operation to fail to meet the hardware backend's limitations. Due to the lack of a pre-verification mechanism for the N dimension, such anomalies are usually triggered only during the underlying execution phase, specifically manifesting as errors during the layout transformation phase, causing the operator to fail to execute normally.

[0038] To circumvent the aforementioned issues, when the N-dimensional limit is exceeded, the approach can degenerate into directly performing computation using a row-major continuous arrangement (also known as the Buffer format). Specifically, this method calls the corresponding computation interface and passes the memory address under the Buffer format to the interface. The underlying computation unit directly reads the data from the buffer memory and completes the computation, ultimately outputting the result. While this method bypasses hardware layout limitations to allow operators to execute, the target hardware lacks specific data read / write optimizations for the Buffer format, resulting in low bandwidth utilization. Therefore, direct computation in the Buffer format cannot fully utilize the optimization capabilities of the underlying computation unit (e.g., the target matrix multiplication hardware) designed for the specific layout, leading to poor computational performance and failing to meet the performance requirements of deep learning training or inference computations in large-scale shape scenarios. For ease of description, the above scheme will be referred to as the Buffer direct computation path below.

[0039] Furthermore, using multiple piecewise matrix multiplications to adapt to hardware constraints introduces additional data padding, scheduling overhead, and result merging costs, further reducing computational efficiency. For example, assuming a height constraint value Th = 8192, in the multiple piecewise matrix multiplication scheme, a tensor with an original logical shape of 8193×64 needs to be split into two sub-tensors: 8192×64 and 1×64. First, matrix multiplication is performed on the 8192×64 sub-tensor in four blocks, then matrix multiplication is performed on the 1×64 sub-tensor separately, and finally the results of each block are merged. This approach not only increases logical complexity but also degrades computational performance due to multiple segmented scheduling.

[0040] This disclosure provides at least one embodiment of a tensor processing method, a tensor processing apparatus, an electronic device, and a storage medium.

[0041] The tensor processing method provided in at least one embodiment of this disclosure includes: responding to a first condition that the number of blocks of a first tensor determined based on a preset block strategy satisfies a first condition, performing dimension reconstruction and layout transformation operations based on the first tensor to obtain a second tensor that is continuously arranged in non-row-major order and satisfies the target hardware constraints; and calling the target hardware operator interface to perform calculations on the second tensor.

[0042] In the tensor processing method provided in at least one embodiment of this disclosure, before calling the target hardware operator interface, it is proactively determined whether the number of blocks of the first tensor determined based on the preset block strategy exceeds the hardware rearrangement limit. When the limit is exceeded, the method avoids using the inefficient buffer direct calculation path and instead proactively performs dimension reconstruction and layout transformation operations. This proactively avoids the risk of runtime failure due to excessive dimension parameters or incompatible memory layouts in the underlying hardware. Furthermore, since the physical layout of the tensor in memory matches the computational unit organization of the underlying hardware, the underlying operator can directly and efficiently read and write according to the hardware's native data access mode, thus fully utilizing hardware acceleration capabilities. This achieves a comprehensive balance of operability, compatibility, and computational performance while maintaining the original semantics of external dimension calculations.

[0043] Figure 1 A flowchart illustrating a tensor processing method provided in at least one embodiment of this disclosure.

[0044] For example, such as Figure 1 As shown, the tensor processing method provided in at least one embodiment of this disclosure may include steps S101 to S102.

[0045] Step S101: In response to the first condition being met by the number of blocks of the first tensor determined based on the preset block strategy, perform dimension reconstruction and layout transformation operations based on the first tensor to obtain a second tensor that is continuously arranged in non-row-major order and meets the target hardware constraints.

[0046] Step S102: Call the target hardware operator interface to perform calculations on the second tensor.

[0047] For example, the number of blocks for the first tensor determined based on the preset block partitioning strategy represents the number of blocks obtained by the default block partitioning strategy executed by the underlying computing framework after the first tensor is input. In other words, the number of blocks determined based on the preset block partitioning strategy is consistent with the number of blocks generated when the underlying computing framework performs block partitioning on the first tensor under the default configuration.

[0048] For example, the first condition can be used to characterize the number of blocks that the current underlying computing framework cannot support or process. For instance, the number of blocks exceeds the processing capacity or support range of the underlying hardware device. In some examples, the first condition may include: the number of blocks is greater than a first hardware support threshold. The first hardware support threshold can represent the maximum upper limit of the number of blocks that the current underlying hardware device can process. An example of the first hardware support threshold is the N-dimensional threshold Tn mentioned above.

[0049] It should be noted that the terms "first tensor," "second tensor," etc., used in the embodiments of this disclosure are merely name identifiers for different tensor objects and are not intended to limit the order, quantity, or hierarchical relationship of tensors. In the embodiments of this disclosure, the first tensor refers to the tensor to be processed input to the tensor processing method. For example, the first tensor may include at least one of voice data, image data, text data, video data, autonomous driving sensor data, biomedical data, and scientific experimental data. Correspondingly, the tensor processing method provided in at least one embodiment of this disclosure can be applied to fields such as voice processing, image processing, text processing, video processing, autonomous driving, biomedical analysis, and scientific research.

[0050] Furthermore, tensors are defined in memory by both logical shape and physical layout: logical shape reflects the logical semantic structure of tensors; physical layout indicates the actual mapping and arrangement rules of tensor elements in physical storage.

[0051] For example, common tensor physical layout types include, but are not limited to, the following: Row-major contiguous arrangement: Consecutive rows of data are stored sequentially in contiguous storage locations (e.g., memory). Taking a two-dimensional array as an example, all data within a row is stored contiguously, while data between different rows may have significant address jumps.

[0052] Column-major contiguous arrangement: Consecutive columns of data are stored sequentially in contiguous memory locations. For example, in a two-dimensional array, all data within a single column is stored contiguously, while data between different columns may have significant memory address jumps.

[0053] Stride layout: By specifying the stride for each dimension, the jump distance of the tensor elements in the storage space is defined. In other words, the stride for each dimension defines how many bytes to jump to the next element, which can flexibly support custom layouts of any dimension.

[0054] Dimensional order arrangement: By specifying the order of the dimensions of a tensor, its layout in storage space is determined. For example, taking a four-dimensional tensor as an example, it includes the batch (B) dimension, channel (C) dimension, height (H) dimension, and width (W) dimension. Two common dimensional order layouts are as follows: NCHW layout: Batch, Channel, Height, Width are arranged in dimensional order, which is suitable for mathematical operations on the CPU and facilitates channel-level operations.

[0055] NHWC layout: arranged in dimensional order of Batch, Height, Width, and Channel, suitable for GPU memory access patterns.

[0056] The physical layout types described above are merely illustrative examples. The physical layout types involved in this disclosure are not limited to the foregoing examples. Various physical layouts can be flexibly defined and extended according to different application scenarios and hardware characteristics. This disclosure does not impose any restrictions on this.

[0057] For example, in step S101, the dimension reconstruction operation (also known as shape folding) represents the logical reconstruction of the original dimensions of the tensor. For instance, the original M dimensions are split into N'×F dimensions (F being the folding factor), thereby logically reconstructing the tensor with the original shape M×K into a shape of N'×F×K. It should be noted that this dimension reconstruction operation modifies metadata parameters such as the shape in the memory descriptor; the actual storage location of the underlying data in physical memory is not moved. Its essence is merely changing the software's logical interpretation perspective of that block of physical memory.

[0058] Further, in step S101, the layout transformation operation represents the process of mapping logical tensor indices to physical memory address spaces. This operation involves actual data memory copying, such as writing data from a buffer into the target physical memory according to the target physical layout and hardware alignment requirements. In the aforementioned example, the layout transformation operation can move and rearrange tensor data that has been logically reconstructed into an N'×F×K shape into memory blocks that conform to the target physical layout.

[0059] For example, row-major contiguous arrangement means that tensor elements are stored sequentially along the row dimension in physical memory at consecutive memory addresses. That is, tensor elements within the same row are arranged adjacently in physical memory, and elements in adjacent rows maintain contiguous memory addresses. The method of storing tensors in a buffer described earlier can be understood as a specific implementation of row-major contiguous arrangement.

[0060] Correspondingly, non-row-major continuous layout can include other physical layout methods besides row-major continuous layout, such as column-major layout, block layout, or any layout method supported by the target hardware.

[0061] For example, satisfying the target hardware constraints can mean that the shape of the tensor satisfies the dimensional constraints of the target hardware. For example, for a two-dimensional matrix, the target hardware requires the tensor to have a shape that satisfies {H, W}, where H ≤ Th and W ≤ Tw. For a three-dimensional matrix, the target hardware requires the tensor to have a shape that satisfies {N, H, W}, where N <= Tn, H ≤ Th, and W ≤ Tw. In some examples, Tn = 1024, Th = 8192, and Tw = 8192. It should be noted that the specific values ​​of the above thresholds are merely examples, and this disclosure does not limit the scope of the embodiments.

[0062] Figure 2A and Figure 2B This is a schematic diagram of a block-based column main sequence arrangement provided for at least one embodiment of the present disclosure.

[0063] like Figure 2A As shown, the complete tensor data is divided into multiple data blocks. The scan path traverses multiple data blocks in a "Z" shape in the two-dimensional plane, first columns and then rows. The arrows in the tensor indicate the access order of the data blocks until all data blocks have been traversed.

[0064] Figure 2B for Figure 2A The diagram illustrates the data element arrangement within each data block, specifically showcasing a 4-row, 32-column block-based column-major order storage structure. For example... Figure 2B As shown, each data block contains 4 rows (arranged along the height H direction) and 32 columns (arranged along the width W direction) of data elements. A data element is represented as E(h+i, w+j), where i∈{0,1,2,3} represents the row offset, and j∈{0,1,…,31} represents the column offset. In this arrangement, data elements within the same row are stored consecutively in ascending column address order; that is, row h is arranged sequentially from E(h,w) to E(h,w+31). The data elements in the next row (row h+1) are stored immediately after the end of the previous row, and so on, until the last element E(h+3,w+31) in row h+3. This storage method constitutes a column-major block internal structure, similar to... Figure 2A The inter-block scanning paths work together to achieve the block-based column-major order arrangement of the overall data.

[0065] For example, in step S102, after the layout conversion is completed, the hardware computing unit can perform corresponding computing operations based on the new physical layout that meets the target hardware constraints, thereby ensuring the efficient execution of the underlying hardware.

[0066] In step S102, the target hardware operator interface can refer to a program interface provided by the target hardware platform that can directly call the underlying hardware acceleration resources to execute a specific computing task. By calling the target hardware operator interface, computing requests from the upper software layer can be directly sent to the lower hardware acceleration unit for execution, thereby avoiding the additional overhead caused by the general software simulation layer and fully utilizing the computing performance of the target hardware.

[0067] In some examples, the target hardware operator interface may include a target hardware matrix multiplication interface. Correspondingly, an example of step S102 may include: invoking the target hardware matrix multiplication interface to perform a matrix multiplication calculation on the second tensor.

[0068] It should be understood that the above matrix multiplication calculation is merely illustrative. The technical solutions of this disclosure are universal and not limited to matrix multiplication operators. They can also be extended to other types of operators, such as convolution operators, reduction operators, activation function operators, element-wise operation operators, etc. Correspondingly, the target hardware operator interface may also include at least one of the following: target hardware convolution interface, target hardware reduction interface, target hardware activation function interface, and target hardware element-wise operation interface. Step S102 may accordingly include calling at least one of the above-mentioned interfaces to perform the corresponding calculation operation on the second tensor.

[0069] In this disclosure, the target hardware refers to a hardware device capable of providing hardware-level accelerated computing capabilities. The hardware device can be an artificial intelligence processor, and may include a Graphics Processing Unit (GPU), a General-purpose Graphics Processing Unit (GPGPU), a Tensor Processing Unit (TPU), a Deep Learning Processing Unit (DPU), an Accelerated Processing Unit (APU), a Neural Network Processing Unit (NPU), an Application-Specific Integrated Circuit (ASIC), or a Field-Programmable Gate Array (FPGA), etc. This disclosure does not impose any limitations on this.

[0070] For example, steps S101-S102 can be executed by the application in user space, constituting the upper-layer application logic; while the underlying operator framework is a lower-layer system module, and the two are at different logical levels in the system architecture. Compared with the scheme where the underlying operator framework directly performs dimension reconstruction and layout transformation, the method of this disclosure embodiment has significant advantages: on the one hand, due to the inherent defects of the underlying dimension reconstruction logic, the number of blocks may exceed the first hardware support threshold in certain scenarios, thus causing computation failure. This disclosure embodiment effectively avoids this defect by moving dimension reconstruction to user space for execution, improving the success rate of computation and the stability of the system; on the other hand, if the physical memory layout changes and is not effectively synchronized with the upper layer, the upper-layer module will perform subsequent calculations based on an incorrect physical memory layout, thereby causing data parsing anomalies or calculation errors. This disclosure embodiment ensures the transparency and consistency of the physical memory layout change to the upper-layer logic by completing dimension reconstruction and layout transformation at the upper layer, ensuring the correctness of the subsequent calculation process.

[0071] In the tensor processing method provided in at least one embodiment of this disclosure, before calling the target hardware operator interface, it is proactively determined whether the number of blocks of the first tensor determined based on the preset block strategy exceeds the hardware rearrangement limit. When the limit is exceeded, the method avoids using the inefficient buffer direct calculation path and instead proactively performs dimension reconstruction and layout transformation operations. This proactively avoids the risk of runtime failure due to excessive dimension parameters or incompatible memory layouts in the underlying hardware. Furthermore, since the physical layout of the tensor in memory matches the computational unit organization of the underlying hardware, the underlying operator can directly and efficiently read and write according to the hardware's native data access mode, thus fully utilizing hardware acceleration capabilities. This achieves a comprehensive balance of operability, compatibility, and computational performance while maintaining the original semantics of external dimension calculations.

[0072] In some examples, the preset block partitioning strategy described above may include the following steps S201 to S202. This preset block partitioning strategy is, for example, the default execution logic of the underlying computing framework, and in the tensor processing method provided in this embodiment, it can be invoked for computation in the upper-level user space. For example, steps S201 to S202 can be executed before step S101, and it can be determined whether the number of blocks obtained in step S202 satisfies the aforementioned first condition.

[0073] Step S201: In response to the first dimension size of the first tensor satisfying the second condition, the target block size is determined from the preset block size set based on the amount of data to be filled corresponding to multiple block sizes in the preset block size set.

[0074] Step S202: Determine the number of blocks for the first tensor based on the first dimension size and the target block size.

[0075] For example, a first tensor may include multiple dimensions, where the size value corresponding to the first dimension is denoted as the first dimension size. The first dimension size can characterize the number of elements in the first tensor along the first dimension.

[0076] For example, if the first tensor is a two-dimensional matrix, then the first dimension can be either the height or the width of the two-dimensional matrix, and correspondingly, the size of the first dimension is the number of elements contained in that height or width dimension. If the first tensor is a high-dimensional tensor (e.g., a three-dimensional, four-dimensional, or higher-dimensional tensor), then the first dimension can be a pre-specified dimension of that high-dimensional tensor, and the size of the first dimension is the number of elements in that pre-specified dimension. For example, when the first tensor is a two-dimensional matrix of shape M×K, if the height dimension is determined as the first dimension, then the size of the first dimension is M; if the width dimension is determined as the first dimension, then the size of the first dimension is K.

[0077] For example, the second condition could include: the first dimension of the first tensor is greater than a second hardware support threshold. The second hardware support threshold could represent the maximum upper limit of the first dimension that the current underlying hardware device can process. An example of the second hardware support threshold when the first dimension is a height dimension is the height constraint value Th described above.

[0078] For example, if the size of the first dimension of the first tensor is less than or equal to the second hardware support threshold, the underlying hardware can process it directly without performing a block operation on the first dimension of the first tensor; if the size of the first dimension of the first tensor is greater than the second hardware support threshold, then a block operation on the first dimension of the first tensor is required.

[0079] For example, the amount of data to be filled refers to the amount of extra data that the last sub-block needs to fill in order to reach the size required by the block size when a certain block size is selected to divide the first dimension of the first tensor.

[0080] For example, the preset block size set may include multiple block sizes no larger than a second hardware support threshold. For instance, the value of each block size may all satisfy the form of a power of 2, where n is a positive integer, to match the processing granularity of the underlying hardware. An example of the preset block size set could be {8192, 4096, 2048, 1024, 512, 256}. This disclosure does not limit the specific number or content of elements in the preset block size set.

[0081] In some examples, for step S201, during the process of determining the target block size from the preset block size set, the preset block size set can be traversed in descending order of numerical values, and the modulo operation result (i.e., M%a) of the first dimension size M of the first tensor and the currently traversed block size a can be calculated sequentially. When the modulo operation result is detected to be zero for the first time during the traversal (i.e., M is divisible by a), the block size a is taken as the target block size and returned. Since the elements in the preset block size set are arranged in descending order, the above traversal mechanism can preferentially return the largest block size that satisfies the divisibility condition. This largest block size is the largest granularity that does not require padding. Using this granularity for subsequent processing can effectively reduce the number of batches generated after the dimension reconstruction operation, thereby optimizing the overall processing performance.

[0082] In other examples, when there is no block size in the preset block size set that can be precisely divided by the first dimension size of the first tensor, a target block size with lower filling cost can be further determined from the preset block size set.

[0083] In determining the target block size, the largest block size in the preset block size set (e.g., 8192) can be used as the initial reference granularity. Then, each block size in the preset block size set, excluding the largest block size, is traversed sequentially in descending numerical order. For the currently traversed block size i, the number of zero elements required to fill it (denoted as the first filling amount) and the number of zero elements required to fill it using the block size from the previous traversal (denoted as the prior block size) can be calculated (denoted as the second filling amount). The first filling amount, need_zero, can be obtained by calculating the difference between the current block size i and the first dimension M of the first tensor modulo the current block size i, i.e., need_zero = i - M % i; the second filling amount, prev_need, can be obtained by calculating the difference between the prior block size prev_i and the first dimension M of the first tensor modulo the prior block size, i.e., prev_need = prev_i - M %prev_i.

[0084] During the traversal, if the first padding amount is detected to be greater than or equal to the second padding amount (i.e., need_zero ≥ prev_need), it indicates that using the current smaller block size has not effectively reduced the padding cost. In this case, the previous block size can be determined as the target block size and returned. If the first padding amount is less than the second padding amount, it indicates that the current smaller block size can effectively reduce the padding cost. The current block size i can be updated to the new previous block size, and the traversal of the next block size in the preset block size set continues until the traversal is complete.

[0085] Through the above mechanism, when the reduction in filling cost brought about by using smaller granularity for block division is not significant, a larger block size is preferred as the target block size. This strategy can prioritize the use of larger block sizes when the filling cost is similar, thereby effectively avoiding an excessive number of batches generated after dimension reconstruction operations.

[0086] The code example for the above traversal process is as follows:

[0087] In step S202, the number of blocks for the first tensor can be determined as follows: Divide the first dimension of the first tensor by the target block size to obtain the quotient, and then round up the quotient to obtain the number of blocks for the first tensor. For example, the number of blocks N for the first tensor can be determined using the following formula:

[0088] Where M is the first dimension of the first tensor, and sub_h is the target block size. This indicates the rounding up operation.

[0089] By using the above method, the number of blocks of the first tensor determined based on the preset block division strategy can be obtained, and then it can be determined whether the number of blocks satisfies the first condition.

[0090] The tensor processing method provided in at least one embodiment of this disclosure may further include step S103.

[0091] Step S103: In response to the fact that the number of blocks of the first tensor determined based on the preset block strategy does not meet the first condition, the underlying computing framework performs a second layout transformation operation on the first tensor and calls the target hardware operator interface.

[0092] For example, in step S103, if the number of blocks does not exceed the processing capacity or support range of the underlying hardware device, the underlying computing framework can, by default, directly perform the second layout transformation operation on the first tensor, and then call the target hardware operator interface. For example, the second layout transformation operation can be a layout transformation adapted to the acceleration characteristics of the underlying hardware. The type of the second layout transformation operation can be the same as or different from the type of layout transformation operation performed in step S101 above, and this embodiment of the disclosure does not impose any restrictions on this. Under this branch, the underlying computing framework can normally handle the layout transformation requirements, so there is no need to perform the dimension reconstruction and layout transformation operations in the upper-level user space in step S101.

[0093] In some examples, step S101, “perform dimension reconstruction and layout transformation operations based on the first tensor”, may include steps S1011 to S1012.

[0094] Step S1011: Perform a first-type dimension reconstruction operation and a layout transformation operation based on the first tensor to obtain an intermediate tensor.

[0095] Step S1012: Perform a second-type dimension reconstruction operation on the intermediate tensor to obtain the second tensor.

[0096] For example, a first type of dimension reconstruction operation may include a dimension expansion operation, used to increase the number of logical dimensions of a tensor by splitting the original dimensions, such as reconstructing a two-dimensional tensor into a three-dimensional tensor; a second type of dimension reconstruction operation may include a dimension merging operation, used to reduce the number of logical dimensions of a tensor by merging multiple dimensions, such as reconstructing a three-dimensional tensor into a two-dimensional tensor. It should be understood that the above examples are for illustrative purposes only, and the embodiments of this disclosure do not impose specific limitations on the number of dimensions involved in the dimension reconstruction operation.

[0097] By employing the above method, tensors can be rearranged in memory to meet the target physical layout while maintaining the original semantics and contextual logic of two-dimensional matrix calculations. This adapts to underlying hardware constraints and improves computational performance. Simultaneously, the original two-dimensional input interface and corresponding valid output rows are retained externally, allowing downstream target hardware operator interface call logic to be modified for adaptability, thus effectively ensuring the compatibility of the computation process and system stability. Compared to schemes that directly use three-dimensional shapes for data transmission, this embodiment avoids the additional overhead caused by downstream computation modules being forced to modify their computation logic due to receiving three-dimensional data, or by forcibly converting the data back to two-dimensional data after computation.

[0098] The tensor transformation method provided in at least one embodiment of this disclosure may further include steps S301 to S302.

[0099] Step S301: Determine the target reconstruction dimension size based on the first dimension size of the first tensor.

[0100] Step S302: Determine whether to perform a padding operation based on the first dimension size of the first tensor and the target reconstruction dimension size.

[0101] For example, in step S301, the target reconstructed dimension size can represent the dimension size obtained by reconstructing the first dimension of the first tensor after performing a dimension reconstruction operation on the first tensor.

[0102] For example, one example of step S301 may include steps S3011 to S3013.

[0103] Step S3011: Based on the amount of data to be filled corresponding to multiple reconstruction dimension dimensions in the preset reconstruction dimension size set, determine the first reconstruction dimension size from the preset reconstruction dimension size set.

[0104] Step S3012: Determine the second reconstruction dimension based on the first dimension size and the first reconstruction dimension size of the first tensor.

[0105] Step S3013: Determine the target reconstruction dimension based on the first reconstruction dimension dimension and the second reconstruction dimension dimension.

[0106] For example, in step S3011, the preset set of reconstructed dimension dimensions may include multiple reconstructed dimension dimensions (also called folding factors). For example, the value of each reconstructed dimension dimension may all satisfy the form of a power of 2, where n is a positive integer. For example, the value of each reconstructed dimension dimension may all satisfy hardware alignment requirements. An example of the preset set of reconstructed dimension dimensions may be {256, 512, 1024, 2048, 4096}. This embodiment of the disclosure does not limit the specific number and content of elements in the preset set of reconstructed dimension dimensions.

[0107] For example, in step S3011, the preset set of reconstruction dimension sizes can be traversed in ascending numerical order to calculate the amount of data to be filled for each reconstruction dimension size. The reconstruction dimension size with the smallest amount of data to be filled is selected as the first reconstruction dimension size, and the second reconstruction dimension size calculated subsequently is not greater than the first hardware support threshold. For example, the amount of data to be filled refers to the amount of extra data that needs to be filled in the last sub-block to reach the size required by the block size when a certain reconstruction dimension size is selected to divide the first dimension of the first tensor.

[0108] For example, in step S3012, the second reconstructed dimension size can be determined as follows: The first dimension size of the first tensor is divided by the first reconstructed dimension size to obtain a quotient, and this quotient is rounded up to obtain the second reconstructed dimension size. For example, the second reconstructed dimension size N' can be determined using the following formula:

[0109] Where M is the first dimension of the first tensor, and F is the first reconstruction dimension. This indicates the rounding up operation.

[0110] For example, in step S3013, the target reconstruction dimension size can be determined based on the product of the first reconstruction dimension size and the second reconstruction dimension size. For instance, the target reconstruction dimension size Padded_M can be determined using the following formula: Padded_M=N'×F For example, in step S302, if the first dimension of the first tensor is smaller than the target reconstruction dimension, it can be determined that a padding operation will be performed; if the first dimension of the first tensor is equal to the target reconstruction dimension, it can be determined that a padding operation will not be performed.

[0111] In some examples, step S1011 may include steps S401 to S402 as follows.

[0112] Step S401: Perform a first-type dimension reconstruction operation on the first tensor.

[0113] Step S402: Perform a layout transformation operation on the first tensor after dimension reconstruction to obtain an intermediate tensor.

[0114] In some examples, if it is determined in step S302 that no padding operation is required, steps S401 to S402 can be executed to perform dimension reconstruction and layout transformation operations.

[0115] In some examples, one instance of step S401 may include: splitting the first dimension of the first tensor into the first reconstruction dimension and the second reconstruction dimension based on the first reconstruction dimension size and the second reconstruction dimension size.

[0116] For example, for a first tensor with a logical shape of M×K, after performing a first-type dimension reconstruction operation, its logical shape can be transformed into N'×F×K. Here, M is the first dimension size of the first tensor, F is the first reconstruction dimension size, and N' is the second reconstruction dimension size.

[0117] In some examples, one instance of step S402 may include: writing the first tensor after dimension reconstruction to a storage area based on the target physical layout, wherein the target physical layout is a non-row-major contiguous arrangement.

[0118] For example, a storage area corresponding to the target physical layout can be allocated; and the first tensor after dimension reconstruction can be copied to the storage area through a memory copy operation, so as to convert the first tensor after dimension reconstruction into the target physical layout during the copying process.

[0119] Among them, the storage area refers to the address space in memory used to store tensor data; allocating a storage area means requesting a memory space of a specified size and layout; and the memory copy operation means copying data from the source address to the target address element by element according to the target physical layout.

[0120] For example, for a first tensor with a logical shape of M×K and a physical layout of row-major consecutive arrangement, after executing step S401, its logical shape is transformed into N'×F×K, while the physical layout remains row-major consecutive arrangement. During step S402, a storage area is first allocated, and through a memory copy operation, the row-major consecutively arranged data is copied element-by-element from the source address to the target address in the storage area according to the target physical layout. This copying process completes the transformation from row-major consecutive arrangement to the target physical layout (e.g., column-major arrangement). Therefore, after step S402 is completed, an intermediate tensor with a logical shape of N'×F×K and a physical layout of column-major arrangement can be obtained.

[0121] In some examples, another example of step S1011 may include steps S403 to S405 as follows.

[0122] Step S403: In response to determining to perform a fill operation, perform a fill operation on the first tensor to obtain the third tensor.

[0123] Step S404: Perform a first-type dimension reconstruction operation on the third tensor.

[0124] Step S405: Perform a layout transformation operation on the third tensor after dimension reconstruction to obtain an intermediate tensor.

[0125] In some examples, if it is determined in step S302 that a padding operation needs to be performed, steps S403 to S405 can be executed to perform dimensional reconstruction and layout transformation operations.

[0126] In some examples, one instance of step S403 may include: allocating a contiguous storage region based on the target reconstruction dimension size, copying the first tensor to the contiguous storage region, and padding the tail of the first tensor in the contiguous storage region with a preset padding value.

[0127] For example, for a first tensor with a logical shape of M×K and a physical layout of row-major contiguous arrangement, in step S403, a contiguous storage region of size padded_M×K is first allocated. The data of the first tensor is copied element by element from the source address to the target address in the contiguous storage region without changing the physical layout, maintaining the row-major contiguous arrangement. Then, (padded_M-M)×K preset padding values ​​are added to the contiguous storage region. In other examples, when allocating a contiguous storage region of size padded_M×K, the entire contiguous storage region can be filled with preset padding values, and then the data of the first tensor can be copied element by element from the source address to the target address in the contiguous storage region starting from the beginning of the contiguous storage region. Therefore, after step S403 is completed, a third tensor with a logical shape of padded_M×K and a physical layout of row-major contiguous arrangement can be obtained.

[0128] For example, the preset fill value can be set according to actual needs, such as 0 or 1, and this embodiment does not limit this.

[0129] The descriptions of steps S404 and S405 can be found in the descriptions of steps S401 and S402 above, except that the objects being processed are different, so they will not be repeated here.

[0130] Through the above operations, regardless of whether a filling operation is required, an intermediate tensor with a logical shape of N'×F×K and a physical layout of column-major order can be obtained.

[0131] In some examples, one instance of step S1012 may include: merging the first and second reconstruction dimensions of the intermediate tensor and performing truncation based on the first dimension size of the first tensor to obtain a second tensor, wherein the first dimension size of the second tensor is equal to the first dimension size of the first tensor.

[0132] For example, the product of the first and second reconstructed dimensions of the intermediate tensor can be calculated to obtain the merged dimension. This merged dimension is then compared to the first dimension M of the first tensor to determine the first dimension of the second tensor. Specifically, when the merged dimension equals the first dimension, it is used as the first dimension of the second tensor; when the merged dimension is greater than the first dimension of the first tensor, it is truncated to the first dimension of the first tensor, and the truncated dimension is used as the first dimension of the second tensor.

[0133] For example, for an intermediate tensor with a logical shape of N'×F×K and a column-major physical layout, the first reconstruction dimension (F dimension) and the second reconstruction dimension (N' dimension) can be merged to obtain a merged intermediate tensor with a logical shape of padded_M×K and a column-major physical layout, where padded_M is the merged dimension size. Furthermore, when the merged dimension size padded_M is greater than the first dimension size M of the first tensor, the merged dimension size of the intermediate tensor can be truncated to the first dimension size of the first tensor (which can be understood as truncating the first M rows of valid data), and the truncated size M is used as the first dimension size of the second tensor. Through these operations, a second tensor with a logical shape of M×K and a column-major physical layout can be obtained. In subsequent calculations, the target hardware operator interface can perform read operations on the corresponding storage area in memory for the second tensor according to the transformed logical shape M×K.

[0134] Figure 3 A flowchart of another tensor processing method provided for at least one embodiment of this disclosure.

[0135] For example, such as Figure 3 As shown, the tensor processing method provided in at least one embodiment of this disclosure may include steps S31 to S35.

[0136] Step S31: Determine the number of blocks for the first tensor based on the preset block division strategy.

[0137] Step S32: Determine whether the number of blocks meets the first condition. If yes, proceed to step S34; otherwise, proceed to step S33.

[0138] Step S33: The underlying computing framework performs a second layout transformation operation on the first tensor.

[0139] Step S34: Perform dimension reconstruction and layout transformation operations based on the first tensor to obtain a second tensor that is continuously arranged in non-row-major order and satisfies the target hardware constraints.

[0140] Step S35: Call the target hardware operator interface to perform the calculation.

[0141] In the tensor processing method provided in at least one embodiment of this disclosure, when the number of blocks determined based on a preset block partitioning strategy exceeds a first hardware support threshold, the method avoids using a directly computed path with insufficient performance using a buffer. Instead, it proactively performs dimension reconstruction and layout transformation operations. This avoids the risk of runtime failure due to excessive dimension parameters or incompatible memory layouts in the underlying hardware. Furthermore, since the physical layout of the tensor in memory matches the computational unit organization of the underlying hardware, the underlying operators can directly and efficiently read and write data according to the hardware's native data access mode. This fully utilizes the hardware acceleration capabilities, achieving a comprehensive balance of operability, compatibility, and computational performance while maintaining the original semantics of external dimension computation.

[0142] Figure 4 A flowchart illustrating yet another tensor processing method provided in at least one embodiment of this disclosure.

[0143] For example, such as Figure 4 As shown, the tensor processing method provided in at least one embodiment of this disclosure may include steps S41 to S56.

[0144] Step S41: Obtain the first tensor. For example, in step S41, the shape of the first tensor, the target hardware constraints, the target physical layout, and the preset set of reconstructed dimension sizes can be obtained.

[0145] Step S42: Determine the number of blocks for the first tensor based on the preset block division strategy.

[0146] Step S43: Determine whether the number of blocks meets the first condition. If yes, proceed to step S45; otherwise, proceed to step S44.

[0147] Step S44: The underlying computing framework performs a second layout transformation operation on the first tensor and calls the target hardware operator interface, followed by step S56.

[0148] Step S45: Based on the amount of data to be filled corresponding to multiple reconstruction dimension dimensions in the preset reconstruction dimension size set, determine the first reconstruction dimension size from the preset reconstruction dimension size set.

[0149] Step S46: Determine the second reconstruction dimension based on the first dimension size and the first reconstruction dimension size of the first tensor.

[0150] Step S47: Determine whether the second reconstruction dimension size meets the first condition. If yes, proceed to step S48; if no, proceed to step S49. For example, it can be determined whether the second reconstruction dimension size exceeds the first hardware support threshold. If yes, it means that there is no second reconstruction dimension size that meets the requirements.

[0151] Step S48: Execute the segmentation scheme or fall back to the general implementation, then execute step S56. For example, the segmentation scheme can be the segmented matrix multiplication scheme described above; the general implementation can be the Buffer direct computation path described above.

[0152] Step S49: Determine the target reconstruction dimension based on the first reconstruction dimension dimension and the second reconstruction dimension dimension.

[0153] Step S50: Determine whether the first dimension of the first tensor is smaller than the target reconstruction dimension. If yes, it means that a padding operation needs to be performed, and proceed to steps S51-S52; if no, it means that a padding operation does not need to be performed, and proceed to step S53.

[0154] Step S51: Perform a filling operation on the first tensor to obtain the third tensor.

[0155] Step S52: Perform a first-type dimension reconstruction operation on the third tensor. Write the dimension-reconstructed third tensor into the storage area based on the target physical layout to obtain an intermediate tensor.

[0156] Step S53: Perform a first-type dimension reconstruction operation on the first tensor. Write the dimension-reconstructed first tensor into the storage area based on the target physical layout to obtain an intermediate tensor.

[0157] Step S54: Merge the first and second reconstruction dimensions of the intermediate tensor, and perform truncation based on the first dimension of the first tensor to obtain the second tensor.

[0158] Step S55: Call the target hardware operator interface to perform calculations on the second tensor.

[0159] Step S56: Output the calculation results.

[0160] In some examples, executing step S41 yields the following information: the logical shape of the first tensor is M×K=1049089×64, the current physical layout is a row-major continuous arrangement, the target hardware constraints are {Tn=1024, Th=8192, Tw=8192}, the target physical layout is a column-major layout, and the preset set of reconstruction dimension sizes is {256, 512, 1024, 2048, 4096}. Executing step S42 determines the number of blocks to be N=1025. Since the first hardware support threshold Tn=1024, N>Tn satisfies the first condition, and the process jumps to step S45, selecting the first reconstruction dimension size F=2048 from the preset set of reconstruction dimension sizes {256, 512, 1024, 2048, 4096}. Execute step S46 to determine the second reconstruction dimension size based on the first dimension size M=1049089 and the first reconstruction dimension size F=2048 of the first tensor. Step S47 is executed. Since N' < Tn does not satisfy the first condition, the process jumps to step S49, where the target reconstruction dimension size Padded_M = N' × F = 513 × 2048 = 1050624 is determined. Because Padded_M > M, a padding operation is required. Step S51 is executed, and a padding operation is performed on the first tensor to obtain a third tensor with a logical shape of Padded_M × K and a physical layout of row-major continuous arrangement. Step S52 is executed, and a first-type dimension reconstruction operation is performed on the third tensor. The reconstructed logical shape is N' × F × K = 513 × 2048 × 64. Based on the target physical layout, the reconstructed third tensor is written into the storage area, resulting in an intermediate tensor with a logical shape of N' × F × K = 513 × 2048 × 64 and a physical layout of column-major arrangement. Step S54: Merge the first reconstruction dimension (F dimension) and the second reconstruction dimension (N' dimension) of the intermediate tensor, and truncate it based on the first dimension size M of the first tensor to obtain a second tensor with a logical shape of M×K=1049089×64 and a physical layout of column-major order. Step S55: Call the target hardware operator interface to perform calculations on the second tensor. Step S56: Output the calculation results corresponding to the M rows of data in the original first tensor.

[0161] It should also be noted that the execution order of the steps in the tensor processing method is not limited in the various embodiments of this disclosure. Although the execution process of each step has been described in a specific order above, this does not constitute a limitation on the embodiments of this disclosure. The steps in the tensor processing method can be executed sequentially or in parallel, depending on the actual needs.

[0162] For example, compared to the above description, the tensor processing method provided in at least one embodiment of this disclosure may include more or fewer steps, and the embodiments of this disclosure do not limit this.

[0163] Figure 5 This is a schematic block diagram of a tensor processing apparatus provided for at least one embodiment of the present disclosure.

[0164] For example, the tensor processing device can be implemented as a processor or disposed in a processor. The processor may include a central processing unit, a graphics processing unit, a general-purpose graphics processing unit, a tensor processor, a deep learning processor, an accelerator processor, a neural network processor, an application-specific integrated circuit, or a field-programmable gate array, etc. Of course, the embodiments of this disclosure are not limited thereto, and the processor may also be any other type of processor.

[0165] For example, such as Figure 5 As shown, the tensor processing apparatus 500 provided in at least one embodiment of this disclosure may include a conversion module 501 and a calling module 502.

[0166] For example, the conversion module 501 is configured to perform dimension reconstruction and layout conversion operations based on the first tensor in response to the first condition being met by the number of blocks of the first tensor determined based on a preset block strategy, thereby obtaining a second tensor that is continuously arranged in non-row-major order and satisfies the target hardware constraints.

[0167] For example, module 502 is configured to invoke the target hardware operator interface to perform computation on the second tensor.

[0168] In some examples, the first condition includes: the number of blocks is greater than a first hardware support threshold.

[0169] In some examples, the transformation module includes a first transformation unit and a second transformation unit. The first transformation unit is configured to perform a first type of dimension reconstruction operation and a layout transformation operation on the first tensor to obtain an intermediate tensor; the second transformation unit is configured to perform a second type of dimension reconstruction operation on the intermediate tensor to obtain a second tensor.

[0170] In some examples, the tensor processing apparatus may also include a determination module configured to determine a target reconstruction dimension based on a first dimension of the first tensor; and to determine whether to perform a padding operation based on the first dimension of the first tensor and the target reconstruction dimension.

[0171] In some examples, the determination module is further configured to determine a first reconstruction dimension size from the preset reconstruction dimension size set based on the amount of data to be filled corresponding to multiple reconstruction dimension sizes in the preset reconstruction dimension size set; determine a second reconstruction dimension size based on the first dimension size of the first tensor and the first reconstruction dimension size; and determine a target reconstruction dimension size based on the first reconstruction dimension size and the second reconstruction dimension size.

[0172] In some examples, the first transformation unit is further configured to perform a first type of dimension reconstruction operation on the first tensor; and to perform a layout transformation operation on the dimensionally reconstructed first tensor to obtain an intermediate tensor.

[0173] In some examples, the first transformation unit is further configured to split the first dimension of the first tensor into the first reconstruction dimension and the second reconstruction dimension based on the first reconstruction dimension size and the second reconstruction dimension size.

[0174] In some examples, the first transformation unit is further configured to write the first tensor after dimension reconstruction to the storage area based on the target physical layout, wherein the target physical layout is a non-row-major contiguous arrangement.

[0175] In some examples, the first transformation unit is further configured to: allocate a storage area corresponding to the target physical layout; and copy the first tensor after dimension reconstruction to the storage area through a memory copy operation, so as to convert the first tensor after dimension reconstruction into the target physical layout during the copying process.

[0176] In some examples, the first transformation unit is further configured to: in response to determining to perform a padding operation, perform a padding operation on the first tensor to obtain a third tensor; perform a first-type dimension reconstruction operation on the third tensor; and perform a layout transformation operation on the dimension-reconstructed third tensor to obtain an intermediate tensor.

[0177] In some examples, the first transformation unit is further configured to: allocate a contiguous storage region based on the target reconstruction dimension size, copy the first tensor to the contiguous storage region, and fill the tail of the first tensor in the contiguous storage region with a preset padding value.

[0178] In some examples, the second transformation unit is further configured to merge the first and second reconstruction dimensions of the intermediate tensor and perform truncation based on the first dimension size of the first tensor to obtain a second tensor, wherein the first dimension size of the second tensor is equal to the first dimension size of the first tensor.

[0179] In some examples, the preset chunking strategy includes: in response to the first dimension size of the first tensor satisfying a second condition, determining a target chunk size from the preset chunk size set based on the amount of data to be filled corresponding to multiple chunk sizes in the preset chunk size set; determining the number of chunks of the first tensor according to the first dimension size of the first tensor and the target chunk size, wherein the second condition includes: the first dimension size is greater than a second hardware support threshold.

[0180] In some examples, the tensor processing apparatus also includes an underlying computation framework configured to: perform a second layout transformation operation on the first tensor and invoke the target hardware operator interface in response to the first tensor not meeting a first condition, as determined by a preset partitioning strategy.

[0181] In some examples, the target hardware operator interface includes a target hardware matrix multiplication interface, and the calling module is further configured to call the target hardware matrix multiplication interface to perform a matrix multiplication calculation on the second tensor.

[0182] It should be noted that the above-mentioned modules can be implemented by software, hardware, firmware or any combination thereof. For example, the conversion module can be implemented as a conversion circuit, and the calling module can be implemented as a calling circuit. The embodiments of this disclosure do not limit their specific implementation methods.

[0183] It should be understood that the tensor processing apparatus provided in at least one embodiment of this disclosure can be used to implement the aforementioned tensor processing method and can also achieve similar technical effects as the aforementioned tensor processing method, which will not be elaborated here.

[0184] It should be noted that, in the embodiments of this disclosure, the tensor processing device may include more or fewer modules or units, and the connection relationship between the various modules or units is not limited and can be determined according to actual needs. The specific configuration of each module or unit is not limited; it can be constructed from analog devices according to circuit principles, or from digital chips, or in other suitable ways.

[0185] Figure 6 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0186] For example, such as Figure 6As shown, the electronic device 600 includes at least one processor 601 and at least one memory 602. The at least one memory 602 includes one or more computer program modules. These computer program modules are stored in the memory 602 and configured to be executed by the at least one processor 601. The one or more computer program modules include instructions for performing the tensor processing method described above. When executed by the at least one processor 601, they can perform one or more steps of the tensor processing method provided in at least one embodiment of this disclosure. The memory 602 and the processor 601 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0187] For example, processor 601 can be a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), artificial intelligence (AI) accelerator, or other form of processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) can be an x86, ARM, or RISC-V architecture. Processor 601 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 600 to perform desired functions.

[0188] For example, memory 602 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0189] Figure 7 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0190] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0191] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 701 as described below, and the memory may include at least one of ROM 702, RAM 703, and storage device 708 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0192] like Figure 7 As shown, the electronic device 700 may include a processing unit 701 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 702 or a program loaded from storage device 708 into RAM 703. RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interfaces are also connected to bus 704.

[0193] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, displays, speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0194] In particular, according to at least one embodiment of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, at least one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by a processing device 701, it performs the functions defined in the methods of at least one embodiment of this disclosure.

[0195] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In at least one embodiment of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In at least one embodiment of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0196] The aforementioned computer-readable medium may be included in the aforementioned electronic device 700; or it may exist independently and not assembled into the electronic device 700.

[0197] Figure 8 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0198] For example, such as Figure 8 As shown, a non-transitory computer-readable storage medium 800 stores computer-readable instructions 801, which, when executed by at least one processor, perform one or more steps of the tensor processing method described above.

[0199] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard drive for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be... Figure 6 The memory 602 in the memory is described in the foregoing content and will not be repeated here.

[0200] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0201] The following points should be noted regarding this disclosure: (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0202] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0203] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0204] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A tensor processing method, characterized in that, The method includes: In response to the first condition being met by the number of blocks of the first tensor determined based on the preset block strategy, dimension reconstruction and layout transformation operations are performed based on the first tensor to obtain a second tensor that is continuously arranged in non-row-major order and meets the target hardware constraints. Call the target hardware operator interface to perform computation on the second tensor.

2. The method according to claim 1, characterized in that, The first condition includes: the number of blocks is greater than a first hardware support threshold.

3. The method according to claim 1, characterized in that, The dimensional reconstruction and layout transformation operations based on the first tensor include: Based on the first tensor, perform a first type of dimension reconstruction operation and a layout transformation operation to obtain an intermediate tensor; Perform a second type of dimension reconstruction operation on the intermediate tensor to obtain the second tensor.

4. The method according to claim 1, characterized in that, The method further includes: The target reconstruction dimension size is determined based on the first dimension size of the first tensor; Whether to perform a filling operation is determined based on the first dimension size of the first tensor and the target reconstruction dimension size.

5. The method according to claim 4, characterized in that, Determining the target reconstruction dimension based on the first dimension of the first tensor includes: Based on the amount of data to be filled corresponding to multiple reconstruction dimension dimensions in the preset reconstruction dimension size set, the first reconstruction dimension size is determined from the preset reconstruction dimension size set; The second reconstruction dimension is determined based on the first dimension size of the first tensor and the first reconstruction dimension size; The target reconstruction dimension is determined based on the first reconstruction dimension and the second reconstruction dimension.

6. The method according to claim 3, characterized in that, The step of performing a first-type dimension reconstruction operation and a layout transformation operation based on the first tensor to obtain an intermediate tensor includes: Perform a first-type dimension reconstruction operation on the first tensor; Perform a layout transformation operation on the first tensor after dimensional reconstruction to obtain the intermediate tensor.

7. The method according to claim 6, characterized in that, The first type of dimension reconstruction operation on the first tensor includes: Based on the first reconstruction dimension size and the second reconstruction dimension size, the first dimension of the first tensor is split into the first reconstruction dimension and the second reconstruction dimension.

8. The method according to claim 6, characterized in that, The step of performing a layout transformation operation on the first tensor after dimensional reconstruction includes: The first tensor after the dimension reconstruction is written into the storage area based on the target physical layout, wherein the target physical layout is a non-row-major sequential arrangement.

9. The method according to claim 8, characterized in that, The step of writing the first tensor after dimension reconstruction based on the target physical layout into the storage area includes: Allocate the storage area corresponding to the target physical layout; The first tensor after dimension reconstruction is copied to the storage area through a memory copy operation, so as to convert the first tensor after dimension reconstruction into the target physical layout during the copying process.

10. The method according to claim 3, characterized in that, The step of performing a first-type dimension reconstruction operation and a layout transformation operation based on the first tensor to obtain an intermediate tensor includes: In response to determining that a padding operation should be performed, the padding operation is performed on the first tensor to obtain a third tensor; Perform a first-type dimension reconstruction operation on the third tensor; Perform the layout transformation operation on the third tensor after dimensional reconstruction to obtain the intermediate tensor.

11. The method according to claim 10, characterized in that, Performing the padding operation on the first tensor includes: Based on the target reconstruction dimension size, allocate a contiguous storage region and copy the first tensor to the contiguous storage region; A preset padding value is filled at the end of the first tensor in the contiguous storage area.

12. The method according to claim 3, characterized in that, The step of performing a second-type dimension reconstruction operation on the intermediate tensor to obtain the second tensor includes: The first and second reconstruction dimensions of the intermediate tensor are merged, and truncation is performed based on the first dimension of the first tensor to obtain the second tensor. The first dimension of the second tensor is equal to the first dimension of the first tensor.

13. The method according to claim 1, characterized in that, The preset block segmentation strategy includes: In response to the first dimension of the first tensor satisfying the second condition, the target block size is determined from the preset block size set based on the amount of data to be filled corresponding to multiple block sizes in the preset block size set. The number of blocks for the first tensor is determined based on the first dimension size of the first tensor and the target block size. The second condition includes: the first dimension size is greater than the second hardware support threshold.

14. The method according to claim 1, characterized in that, The method further includes: In response to the fact that the number of blocks of the first tensor determined based on the preset block division strategy does not meet the first condition, the underlying computing framework performs a second layout transformation operation on the first tensor and calls the target hardware operator interface.

15. The method according to claim 1, characterized in that, The target hardware operator interface includes a target hardware matrix multiplication interface. The step of calling the target hardware operator interface to perform calculations on the second tensor includes: calling the target hardware matrix multiplication interface to perform matrix multiplication calculations on the second tensor.

16. A tensor processing apparatus, characterized in that, The tensor processing device includes: The conversion module is configured to perform dimension reconstruction and layout conversion operations based on the first tensor in response to the first condition that the number of blocks of the first tensor determined based on the preset block strategy meets the first condition, so as to obtain a second tensor that is continuously arranged in non-row-major order and meets the target hardware constraints. The calling module is configured to invoke the target hardware operator interface to perform computation on the second tensor.

17. An electronic device, characterized in that, The electronic device includes: At least one processor; At least one memory, including one or more computer program modules; The one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the method according to any one of claims 1-15.

18. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions, when executed by at least one processor, perform the method according to any one of claims 1-15.

Images