Data loading method and device, processor, electronic equipment and storage medium

Abstract

The present disclosure provides a data loading method, device, processor, electronic equipment and storage medium. The method comprises: obtaining a loading instruction for loading a to-be-processed tensor from a first memory to a second memory; determining a shape size of the to-be-processed tensor and an initial starting coordinate in a coordinate system determined by the to-be-processed tensor; starting from the initial starting coordinate, executing the loading instruction by sequentially executing Z sub-instructions, wherein for the i-th sub-instruction in the Z sub-instructions, the following operations are performed: returning data of the i-th data block starting from the data block starting coordinate, so that the data in the i-th data block is stored continuously in the second memory, and determining the data block starting coordinate of the i+1-th sub-instruction based on the data block starting coordinate of the i-th sub-instruction, the size of the data block in each block dimension, and the remaining size of the to-be-processed tensor in the corresponding each tensor dimension.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and specifically to a data loading method, apparatus, processor, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence, machine learning, and deep learning, the processing of massive amounts of data has become a core task for computing devices. In these computing scenarios, input data, weight parameters, and intermediate feature maps typically exist in the form of tensors. A tensor can be understood as a multidimensional array, which can encompass feature data with multiple dimensions such as batch size, depth, height, width, and number of channels. Parallel processors (e.g., multi-core processors, general-purpose graphics processing units (GPGPUs), tensor processors, etc.) are widely used to perform computationally intensive operations involving these multidimensional tensors, such as matrix multiplication and convolution operations. Summary of the Invention

[0003] This disclosure provides a data loading method, apparatus, processor, electronic device, and storage medium for efficiently loading linearly laid-out tensor data into a cache at the granularity of data blocks, thereby improving the efficiency of processor data loading and enhancing hardware computing utilization.

[0004] According to one aspect of this disclosure, a data loading method is provided for converting a tensor to be processed from linear layout storage to block layout storage based on data blocks of a preset shape and size, wherein the data in each block dimension of the data block corresponds to the continuous data in each tensor dimension of the tensor to be processed, which corresponds to each block dimension respectively. The method includes: obtaining a loading instruction to load the tensor to be processed from a first memory to a second memory; determining the shape and size of the tensor to be processed and the initial starting coordinates in the coordinate system determined by the tensor to be processed; starting from the initial starting coordinates, executing the loading instruction by sequentially executing Z sub-instructions. Let Z be the total number of sub-instructions and be an integer greater than 1. For the i-th sub-instruction among the Z sub-instructions, perform the following operations: return the data of the i-th data block starting from the starting coordinates of the data block, so that the data in the i-th data block is stored contiguously in the second memory. Determine the starting coordinates of the data block of the (i+1)-th sub-instruction based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in each block dimension, and the remaining size of the tensor to be processed in each corresponding tensor dimension, where i is an index number less than Z, and the total shape size of the Z data blocks corresponding to the Z sub-instructions is not less than the shape size of the tensor to be processed.

[0005] In some embodiments, the size of the data block in the block dimension is less than or equal to the size of the tensor dimension corresponding to the block dimension in the tensor to be processed.

[0006] In some embodiments, the method further includes using the initial starting coordinates as the starting coordinates of the data block of the first sub-instruction in the Z sub-instructions.

[0007] In some embodiments, determining the starting coordinates of the data block of the (i+1)th sub-instruction based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in the first block dimension, and the remaining size of the tensor to be processed in the first tensor dimension includes: in response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is greater than or equal to the size of the data block in the first block dimension, determining the coordinates of the first block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction as the sum of the coordinates of the first block dimension in the starting coordinates of the data block of the i-th sub-instruction and the size of the data block in the first dimension; in response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is less than the size of the data block in the first dimension, determining the coordinates of the first block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction as the coordinates of the first block dimension in the initial starting coordinates.

[0008] In some embodiments, the block dimension further includes a second block dimension higher than the first block dimension. Determining the starting coordinates of the data block of the (i+1)th sub-instruction based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in the first block dimension, and the remaining size of the tensor to be processed in the first tensor dimension further includes: in response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is greater than or equal to the size of the data block in the first dimension, determining the coordinates of the second block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction as the coordinates of the second block dimension in the starting coordinates of the data block of the i-th sub-instruction; in response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension... If the size of the tensor to be processed is less than the size of the data block in the first dimension, perform the following operations: In response to determining that the remaining size of the tensor to be processed in the second tensor dimension corresponding to the second dimension is greater than or equal to the size of the data block in the second dimension, determine the coordinates of the second dimension in the starting coordinates of the data block of the (i+1)th sub-instruction as the sum of the coordinates of the second dimension in the starting coordinates of the data block of the ith sub-instruction and the size of the data block in the second dimension; In response to determining that the remaining size of the tensor to be processed in the second tensor dimension corresponding to the second dimension is less than the size of the data block in the second dimension, determine the coordinates of the second dimension in the starting coordinates of the data block of the (i+1)th sub-instruction as the coordinates of the second dimension in the initial starting coordinates.

[0009] In some embodiments, returning the data of the i-th data block starting from the starting coordinates of the data block, such that the data in the i-th data block is stored continuously in the second memory, includes: sequentially sending multiple requests for loading the i-th data block, and sequentially writing the sub-data returned by each request into the second memory to load the data block into the second memory; wherein, in response to the size relationship between the i-th data block and the tensor to be processed in the first tensor dimension such that when loading the data block, continuous loading is not possible in the first tensor dimension but continuous loading is possible in all dimensions below the first tensor dimension, the requests are partitioned on the tensor dimension to be partitioned, and the sub-data loaded by each request is located in the first memory or is not located in the first memory at all; wherein, the first tensor dimension is the same as the tensor dimension to be partitioned, or the first tensor dimension is adjacent to the tensor dimension to be partitioned.

[0010] In some embodiments, in response to the fact that all the requested sub-data is located in the first memory, the requested sub-data comes from the tensor to be processed and is stored contiguously in the first memory, and in response to the fact that none of the requested sub-data is located in the first memory, a predetermined value is written to the second memory.

[0011] In some embodiments, the first request among the multiple requests and the initial state of the first request entering the state machine are determined based on the starting coordinates of the data block of the i-th data block and the shape and size of the data block. In combination with the initial state, each of the multiple requests other than the first request is determined using the state machine based on the shape and size of the tensor to be processed.

[0012] In some embodiments, the tensor to be processed is stored in the NDHWC data storage format, where N represents the batch dimension, D represents the depth dimension, W represents the width dimension, H represents the height dimension, and C represents the number of channels dimension.

[0013] In some embodiments, the tensor to be processed is stored in a data storage format of N(C / x)DHW(xC), where N represents the batch dimension, D represents the depth dimension, W represents the width dimension, H represents the height dimension, C represents the channel number dimension, and x is a positive integer greater than 1.

[0014] In some embodiments, the preset shape and size of the data block are determined based on the matrix size supported by the processor's arithmetic logic unit in a single calculation cycle.

[0015] According to another aspect of this disclosure, a data loading apparatus is provided for converting a tensor to be processed from linear layout storage to block layout storage based on data blocks of a preset shape and size, wherein the data in each block dimension of the data block corresponds to the continuous data in each tensor dimension of the tensor to be processed, which corresponds to the block dimension respectively. The apparatus includes: an instruction acquisition module for acquiring a loading instruction to load the tensor to be processed from a first memory to a second memory; an instruction parsing module for determining the shape and size of the tensor to be processed and the initial starting coordinates in the coordinate system determined by the tensor to be processed; and a sub-instruction execution module for executing sub-instructions starting from the initial starting coordinates and executing sub-instructions according to the block dimension. The load instruction is executed by executing Z sub-instructions, where Z is the total number of sub-instructions and is an integer greater than 1. For the i-th sub-instruction among the Z sub-instructions, the following operations are performed: the data of the i-th data block is returned from the starting coordinates of the data block, so that the data in the i-th data block is stored contiguously in the second memory. The starting coordinates of the data block of the (i+1)-th sub-instruction are determined based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in each block dimension, and the remaining size of the tensor to be processed in each corresponding tensor dimension, where i is an index number less than Z. The total shape size of the Z data blocks corresponding to the Z sub-instructions is not less than the shape size of the tensor to be processed.

[0016] According to another aspect of this disclosure, a processor is provided, comprising: a first memory for storing tensors to be processed in a linear layout; a second memory for storing tensors to be processed in a block layout of data blocks based on a preset shape and size; an instruction parsing unit for acquiring a load instruction for loading the tensors to be processed from the first memory to the second memory; and a data loading unit coupled to the first memory, the second memory, and the instruction parsing unit, the data loading unit being configured to execute the data loading method according to the above description in response to the load instruction.

[0017] According to another aspect of this disclosure, an electronic device is provided, including a memory that non-transitory stores computer-executable instructions; and a processor configured to execute the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement the data loading method described above.

[0018] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the data loading method described above. Attached Figure Description

[0019] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0020] Figure 1 A schematic diagram of the structure of a general-purpose graphics processor according to an embodiment of the present disclosure is shown.

[0021] Figure 2 An exemplary flowchart of a data loading method according to an embodiment of the present disclosure is shown.

[0022] Figure 3 An exemplary flowchart of a method 300 for loading data by the starting coordinates of a data block according to an embodiment of the present disclosure is shown.

[0023] Figure 4 An exemplary flowchart of a method 400 for determining the starting coordinates of a data block in a first-dimensional block dimension according to an embodiment of the present disclosure is shown.

[0024] Figure 5 An exemplary flowchart of a method 500 for determining the starting coordinates of a data block in a block dimension higher than a first dimension, according to an embodiment of the present disclosure, is shown.

[0025] Figure 6A An exemplary schematic diagram of sub-instruction start coordinate update according to at least one embodiment of the present disclosure is shown.

[0026] Figure 6B An exemplary schematic diagram of sub-instruction start coordinate update according to at least one embodiment of the present disclosure is shown.

[0027] Figure 6C An exemplary schematic diagram of sub-instruction start coordinate update according to at least one embodiment of the present disclosure is shown.

[0028] Figure 7 A schematic diagram illustrating data loading at the data block granularity according to an embodiment of the present disclosure is shown.

[0029] Figure 8 An exemplary block diagram of a data loading apparatus according to an embodiment of the present disclosure is shown.

[0030] Figure 9 An exemplary block diagram of a processor according to an embodiment of the present disclosure is shown.

[0031] Figure 10 An exemplary block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0032] Figure 11A block diagram of an example computer device according to an embodiment of the present disclosure is shown.

[0033] Figure 12 An exemplary block diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0034] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0036] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0037] In typical parallel processor computing architectures, computing units such as the Arithmetic Logic Unit (ALU) and Tensor Core typically need to load tensor data from large but relatively slow external memory (e.g., global memory) into a high-speed on-chip buffer located close to the computing unit before executing specific computational tasks. During this data loading process, the layout and storage method of the data in memory not only affects memory access bandwidth but also directly determines the hardware computation utilization of subsequent computing units and the efficiency of cache space release.

[0038] Typically, for user-friendly or upper-level software programming, tensor data is stored in memory in a linear layout (e.g., contiguous NDHWC format). However, at the hardware execution level, different computational units have specific requirements for data loading formats. For example, for hardware logic performing matrix multiplication and addition (MMA) operations, a block-based storage layout that receives and processes data in "blocks" of specific sizes is often more flexible and efficient. A suitable block-based storage layout can significantly improve the processor's data loading efficiency and enhance hardware computational utilization.

[0039] In technologies related to data storage method conversion, data rearrangement typically relies on the software or compiler level. That is, the software explicitly inserts data rearrangement instructions to split and reassemble linear tensors into a block format before computation. However, this software-dependent data rearrangement approach has significant drawbacks: on the one hand, it greatly increases the complexity of software programming and compiler optimization; on the other hand, software-level rearrangement operations introduce significant additional execution time, leading to inefficient data loading and even making data loading a bottleneck limiting computational power.

[0040] To address the aforementioned issues, embodiments of this disclosure provide a fast and efficient data loading method that can automatically and efficiently load tensor data stored in a linear layout into the cache at the data block level without requiring software intervention for rearrangement, thereby balancing the intuitiveness of programming with the efficiency of hardware computation.

[0041] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0042] Figure 1 A schematic diagram of the structure of a general-purpose graphics processor according to an embodiment of the present disclosure is shown.

[0043] like Figure 1 As shown, a general-purpose graphics processing unit (GPU) is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), which includes, for example,... Figure 1 The diagram shows streaming processor clusters 1 to M, where M is a positive integer greater than 1. In a general-purpose graphics processor, a single streaming processor cluster can handle one or more computational tasks, or multiple streaming processor clusters can handle a single computational task. Multiple streaming processor clusters can share data through a global cache or High Bandwidth Memory (HBM).

[0044] Taking streaming processor cluster 1 as an example, a streaming processor cluster can include multiple computing units (CUs), for example... Figure 1 The system consists of computational units 1, 2, and N, where N is a positive integer greater than 1. Each computational unit performs arithmetic and logical operations, such as matrix multiplication and addition (MMA), accumulation, reduction, and regular addition, subtraction, multiplication, and division. Each computational unit includes multiple cores (also called computational kernels), and each core typically contains arithmetic logic units, floating-point calculation units, etc. The cores are used to execute specific computational task instructions.

[0045] To support these computational tasks, general-purpose graphics processors employ a hierarchical data storage architecture. For example... Figure 1 As shown, the compute unit contains a register file and shared memory for high-speed sharing and temporary storage of source and destination data related to the compute task between cores. Outside the compute unit, within the streaming processor cluster, a cache is provided, which can serve as a local data cache shared by multiple compute units. Outside the streaming processor cluster, a global cache and a large-capacity, high-bandwidth memory are also located.

[0046] In parallel computing, computational tasks are typically executed through multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block dispatch module. Figure 1 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0047] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or high-bandwidth memory for read and write operations, etc.

[0048] In applications such as deep learning and artificial intelligence, computational units need to process massive amounts of tensor data when performing Generalized Matrix Multiplication (GEMM) or tensor operations. This multidimensional tensor data (e.g., input feature maps, weight parameters, image data) is typically stored in external high-bandwidth memory in a user-friendly linear layout, such as the multidimensional continuous NDHWC format. When the computational unit performs computational tasks, it needs to extract this tensor data from the high-bandwidth memory using data loading instructions and load it into a cache area close to the computational core.

[0049] Figure 2 An exemplary flowchart of a data loading method according to an embodiment of the present disclosure is shown.

[0050] In embodiments of this disclosure, a method with Figure 1 The device or electronic equipment shown is a general-purpose graphics processor hardware architecture for performing [the task]. Figure 2 The data loading method 200 is shown in the figure.

[0051] The data loading method 200 is used to convert the tensor to be processed from linear layout storage to block layout storage based on data blocks of a preset shape and size, wherein the data of the data block in each block dimension corresponds to the continuous data in each tensor dimension of the tensor to be processed, which corresponds to each block dimension respectively.

[0052] Tensors to be processed are typically used to perform general matrix multiplication, convolution operations, or other tensor operations in the fields of artificial intelligence or deep learning. Combined with... Figure 1 The architecture of the general-purpose graphics processor shown allows these tensors to be processed (e.g., input feature map data, weight parameters, etc.) to be initially stored in a large-capacity memory outside the chip or on-chip (e.g., Figure 1 In order to enable high-concurrency processing by the computing cores, tensors to be processed need to be fetched from mass storage and loaded into high-speed temporary storage closer to the computing units (e.g., high-bandwidth memory). Figure 1 It is located in the cache or shared memory within the streaming processor cluster for use by the arithmetic logic units within the computing core.

[0053] In some embodiments, the tensor to be processed can be stored in the NDHWC data storage format. Here, N represents the batch dimension, D represents the depth dimension, W represents the width dimension, H represents the height dimension, and C represents the channel dimension. In the NDHWC format, the physical address arrangement of data in memory is with dimension C as the fastest changing inner dimension, followed by W, H, D, and N dimensions in sequence. That is, it follows a purely linear pattern of continuous arrangement, starting with all channel elements at the first spatial location determined by the height and width dimensions, and then storing all channel elements at the next adjacent spatial location.

[0054] In some embodiments, the tensor to be processed can also be stored in the data storage format N(C / x)DHW(xC), where N represents the batch dimension, D represents the depth dimension, W represents the width dimension, H represents the height dimension, C represents the channel number dimension, and x is a positive integer greater than 1. This format is often referred to as interleave mode. Similar to NDHWC, but with the key difference that the number of channels is divided into multiple groups containing x channels (e.g., x can be a multiple of 2 such as 8, 16, or 32), and each group of x channels is locally arranged according to the NDHWC format.

[0055] It is understandable that, regardless of whether the tensor to be processed is stored in the aforementioned NDHWC or N(C / x)DHW(xC) data storage format, from the perspective of the user or upper-level software, they can both be regarded as a linear layout storage of a multidimensional array. That is, macroscopically, it manifests as a large contiguous storage structure that can be described by a single instruction containing the starting address and the dimensions of each dimension. Furthermore, different linear storage methods only result in different strides of each dimension in the physical memory address. The different loading methods resulting from the above differences can be further decomposed into loading instructions by the state machine described below. Therefore, the embodiments of this disclosure do not limit the specific method of linear storage.

[0056] Block-based storage refers to dividing tensors into multiple multidimensional data blocks with specific shapes and sizes during data transfer or caching, and writing these data blocks into memory at a granular level. This layout abandons the storage characteristic of tensors folding across the entire tensor dimension, and instead achieves the aggregation of data within each data block in physical memory, thereby improving the efficiency of parallel reading and computation in hardware.

[0057] In some embodiments, the preset shape size of the data block is determined based on the matrix size supported by the processor's arithmetic logic unit in a single computation cycle. Since the throughput capacity of the hardware path of the arithmetic logic unit within the computing core (e.g., a tensor core or multiply-accumulate array executing MMA multiply-accumulate instructions) is fixed within a single clock cycle or a specific execution cycle, it can only receive matrix block data of a specific size. Therefore, in some embodiments, the preset shape size of the data block can be the same as the matrix size supported in a single computation cycle. For example, if the hardware architecture of the arithmetic logic unit is designed to support 16×16 two-dimensional matrix multiplication in a single operation, the shape size of the data block can be preset to 16×16 in two dimensions; if the hardware architecture supports three-dimensional tensor convolution, for example, supporting a single throughput of 8×8×4 data, the preset shape size of the data block can be set to 8×8×4. In this way, each data block moved to the second memory can exactly fill the computation path of the arithmetic logic unit, avoiding redundant data readings or idle waiting of the computing unit. In other embodiments, the preset shape and size of the data block can be a predetermined multiple or fraction (such as 1 / 2, 1 / 3, etc.) of the matrix size supported by a single computation cycle. Those skilled in the art can set the shape and size of the data block according to the actual situation so that the size of the data block is aligned with the computational granularity of the computational unit in a desired manner.

[0058] In some embodiments, the data in each block dimension of a data block corresponds to continuous data in the tensor dimensions corresponding to the block dimensions of the tensor to be processed. Here, tensor dimension refers to the original dimension of the tensor to be processed at the macroscopic level (e.g., the five dimensions N, D, H, W, and C mentioned above). Block dimension refers to the internal dimension of the small data block itself (e.g., the block's height, width, and channels). The fact that the data in each block dimension of a data block corresponds to continuous data in the tensor dimensions corresponding to the block dimensions of the tensor to be processed means that, physically, a data block is a "solid slice" of the original massive tensor in a local space. Each row or column of data within the block is also a small, contiguous segment of data in the corresponding dimension of the original tensor, rather than discrete data with skipped sampling.

[0059] In some implementations, the tensor to be processed can be stored in the NDHWC data storage format, where the tensor dimensions correspond to the five dimensions of NDHWC. In some examples, the data block can be a three-dimensional data block with block dimensions representing H, W, and C. In this case, each block dimension of the three-dimensional data block maps to a corresponding tensor dimension of the tensor to be processed. Specifically, the data in the C block dimension of the data block corresponds to a continuous channel data segment in the original tensor; the data in the W block dimension corresponds to a continuous width data segment in the original tensor; and the data in the H block dimension corresponds to a continuous height data segment in the original tensor. In other words, the data block is a tiny three-dimensional continuous entity segmented from the macroscopic five-dimensional tensor. It is understood that the block dimension of the data block is at least 1, and the block dimension is less than or equal to the total number of tensor dimensions of the tensor to be processed (e.g., the data block can be a one-dimensional vector, a two-dimensional planar matrix, or a multi-dimensional hypercube with a maximum dimension not exceeding the total number of tensor dimensions). The size of the data block in the block dimension is less than or equal to the size of the tensor dimension in the tensor to be processed corresponding to the block dimension.

[0060] Figure 2 Exemplary steps of method 200 are shown.

[0061] In step S202, a loading instruction is obtained to load the tensor to be processed from the first memory to the second memory.

[0062] In step S204, the shape and size of the tensor to be processed and the initial starting coordinates in the coordinate system determined by the tensor to be processed are determined.

[0063] In step S206, starting from the initial starting coordinates, the loading instruction is executed by sequentially executing Z sub-instructions, where Z is the total number of sub-instructions and is an integer greater than 1.

[0064] Therefore, this disclosure automatically converts larger-granularity loading instructions into multiple smaller-granularity sub-instructions without changing the linear data arrangement instructions on the software side. Through the execution of the sub-instructions, the linearly stored tensor data is automatically loaded at the data block level for subsequent storage and computation. This eliminates software rearrangement overhead while achieving the intuitiveness of software programming and the efficiency of hardware block computation.

[0065] Specific examples of this disclosure will be described in detail below.

[0066] In step S202, a loading instruction is obtained to load the tensor to be processed from the first memory to the second memory.

[0067] Typically, the first memory is a large-capacity memory located outside the chip or on the chip, while the second memory is usually a high-speed temporary storage memory located close to the computing unit.

[0068] In some embodiments, it can be achieved through Figure 1 The instruction dispatch module or instruction parsing unit in the general-purpose graphics processor (e.g., the instruction scheduler at the front end of a streaming processor cluster) obtains the instruction to load the tensor to be processed from mass storage to high-speed memory. In some examples, the first memory may be... Figure 1 The high-bandwidth memory (HBM), global cache, or system main memory (DDR RAM) in the memory can be the secondary memory. Figure 1 The first and second memories can be defined as buffers within a streaming processor cluster, shared memory within a computing unit, or register files directly connected to arithmetic logic units. Those skilled in the art can determine the specific definitions of the first and second memories based on the actual situation.

[0069] It is understood that the "Load" instruction mentioned in the embodiments of this disclosure can also be broadly understood as a data transfer instruction. For example, the instruction to load the tensor to be processed can also be to perform a reverse operation (Store operation), that is, to move or write the tensor to be processed from high-speed memory to mass storage; or, the instruction can also be an instruction to control the data migration of the tensor to be processed between two memories with the same read / write speed (e.g., different global cache levels). In other words, the method of this disclosure can be applied to any scenario involving the format conversion of tensor data from linear layout to block layout. The physical specifications, capacity, and access rate type of the first and second memories do not limit or affect the state machine operation and specific implementation of the data loading method of this disclosure.

[0070] In step S204, the shape and size of the tensor to be processed and the initial starting coordinates in the coordinate system determined by the tensor to be processed are determined.

[0071] The loading instruction obtained in step S202 may include parameter information such as the specific shape and size of the tensor to be processed. In some embodiments, the shape and size of the tensor to be processed includes the number of tensor dimensions (e.g., five dimensions: N, D, H, W, and C) and the length of the data elements contained in each specific tensor dimension. Each tensor dimension may contain a different number of data elements of varying lengths. For example, the width W dimension of a tensor to be processed may contain 128 elements, the height H dimension may contain 64 elements, and the channel C dimension may contain 32 elements, etc. The arrangement order and length information of these dimensions together define the logically complete shape of the tensor to be processed.

[0072] The coordinate system defined by the tensor to be processed can be a multi-dimensional relative coordinate system. In some cases, since the tensor to be processed can be a local subset (i.e., the region of interest) of the original full data in a large-capacity memory (e.g., a very high-resolution image or a complete set of feature maps), in order to accurately locate this local subset, it is usually necessary to use a specific element in the original full data (e.g., the top-left vertex or the starting storage address of the original tensor) as the origin of the coordinate system (i.e., the absolute coordinate 0 point), and establish a relative coordinate system using the various dimensions of the tensor (such as the W, H, and C axes) as coordinate axes.

[0073] The initial starting coordinates are the spatial starting point of the tensor to be processed in this relative coordinate system. Specifically, the initial starting coordinates can be a set of multidimensional coordinate values ​​(e.g., ...). c_coord_b, w_coord_b, h_coord_b, d_ coord_b, n_coord_b This set of coordinate values ​​precisely points to the relative position of the first data element of the tensor to be processed in the aforementioned multidimensional coordinate system. At the hardware level, by combining this multidimensional relative initial starting coordinate with the base address of the original tensor in the first memory and the step size of each dimension to perform address mapping calculation, the actual physical memory address of the starting point of the tensor to be processed in the first memory can be determined.

[0074] In step S206, starting from the initial starting coordinates, the loading instruction is executed by sequentially executing Z sub-instructions, where Z is the total number of sub-instructions and is an integer greater than 1.

[0075] Specifically, in order to enable the computing unit to perform operations at the specific data block granularity it supports, this disclosure automatically divides the aforementioned large-granularity load instruction into Z smaller-granularity sub-instructions.

[0076] In some embodiments, the first loaded data block may be located at the starting position of the tensor to be processed. Therefore, the initial starting coordinates determined in step S204 are used as the starting coordinates of the data block of the first sub-instruction in these Z sub-instructions.

[0077] In the embodiments of this disclosure, the starting coordinates of the data block serve to locate the relative position of the data block within the original linear tensor (e.g., in the first memory). Specifically, using the starting coordinates of the data block, hardware (e.g., a direct memory access controller within a streaming processor cluster, or a dedicated memory read unit) can accurately calculate the physical address of the first element of the data block in the first memory, thereby starting from that physical address and beginning local data loading of the tensor to be processed at the data block granularity according to the size of the data block itself.

[0078] The sequential execution of Z sub-instructions in step S206 refers to the hardware unit, after successfully loading the current i-th data block (i is an index number greater than or equal to 1 and less than Z), dynamically calculating and generating the new starting coordinates of the next (i+1)-th data block to be loaded (i.e., generating the execution parameters for the next sub-instruction) using the shape and size information of the current data block and the shape and size information of the tensor to be processed. This process continues until all Z segmented data blocks have been loaded. This serial pipeline-style iterative calculation and reading process constitutes the complete process of sequentially executing Z sub-instructions.

[0079] To more clearly illustrate the execution process of the above single sub-instruction, Figure 3 An exemplary flowchart of a method 300 for loading data by the starting coordinates of a data block according to an embodiment of the present disclosure is shown.

[0080] like Figure 3 As shown, for the i-th subinstruction out of Z subinstructions (where i is an index greater than or equal to 1 and less than Z), perform the following operation: In step S302, the data of the i-th data block is returned starting from the starting coordinates of the data block, so that the data in the i-th data block is stored continuously in the second memory.

[0081] In step S304, the starting coordinates of the data block of the (i+1)th sub-instruction are determined based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in each block dimension, and the remaining size of the tensor to be processed in each corresponding tensor dimension.

[0082] The total shape and size of the Z data blocks corresponding to the Z sub-instructions is not less than the shape and size of the tensor to be processed. This is because when dividing a linear tensor into block layouts, if the total size of the tensor to be processed in a certain dimension (such as the W or H dimension) cannot be divided evenly by the size of that data block, the last loaded Z-th data block may exceed the physical boundary of the original tensor to be processed. For the parts that exceed the boundary (e.g., invalid data or non-interesting data), subsequent processing can be carried out by supplementing predetermined values ​​(e.g., padding with zeros). Therefore, the total size of these Z data blocks after being combined can exactly cover, or slightly exceed, the actual shape and size of the original tensor to be processed.

[0083] In embodiments of this disclosure, the remaining size refers to the length of unloaded valid data remaining in a specific tensor dimension (e.g., height H or width W) of the tensor to be processed, excluding the element data that has been successfully requested and loaded by a previous sub-instruction. remain_copy(Variable representation). In some hardware execution scenarios, the remaining size can be equivalently reflected as how far the current coordinate position of the current data block is from the physical boundary of the tensor to be processed in that dimension.

[0084] By executing method 300 on the i-th sub-instruction out of Z sub-instructions, the data block corresponding to the sub-instruction is identified and located in a linear space starting from the initial starting coordinates. Specifically, step S302 directly extracts the data block at the corresponding position and writes it continuously into the target memory (second memory), realizing the physical format conversion from linear arrangement to highly cohesive block arrangement; while step S304 utilizes multi-dimensional coordinate jump (i.e., based on the comparison of the current block starting coordinates, block shape size and tensor boundary / remaining size) to automatically plan the precise jump landing point (i.e., the starting coordinates of the next data block) for the next sub-instruction.

[0085] pass Figure 3 The steps shown in this disclosure completely offload the complex memory extraction operation of converting multidimensional linear arrays into block matrices to the hardware for automatic processing, eliminating the huge overhead of inserting rearrangement instructions on the software side. Secondly, by predicting the starting coordinates of the next instruction based on the current coordinates and the remaining size, embodiments of this disclosure provide an efficient address generator, enabling the hardware to continuously provide the required data blocks to the computing unit with minimal clock cycle overhead, regardless of the complexity of the tensor's dimensions.

[0086] Figure 4 An exemplary flowchart of a method 400 for determining the starting coordinates of a data block in a first block dimension according to an embodiment of the present disclosure is shown. Figure 4 As shown, method 400 specifically includes steps S402 and S404.

[0087] In step S402, in response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is greater than or equal to the size of the data block in the first block dimension, the coordinates of the first block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined as the sum of the coordinates of the first block dimension in the starting coordinates of the data block of the i-th sub-instruction and the size of the data block in the first block dimension.

[0088] In step S404, in response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is less than the size of the data block in the first block dimension, the coordinates of the first block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined as the coordinates of the first block dimension in the initial starting coordinates.

[0089] Specifically, the first block dimension can generally be understood as the lowest or fastest changing dimension of the data block during the iterative partitioning process of data reading and writing. It's important to note that which tensor dimension in the original tensor to be processed (e.g., NDHWC format) the first block dimension specifically maps to is specified by the preset shape and size of the data block and the priority traversal order of the hardware architecture. For example, the first block dimension can be any dimension among all tensor dimensions corresponding to the tensor to be processed, such as width (W), height (H), or channel (C). In some examples, coordinate stepping and boundary checks are preferentially initiated from the lowest block dimension (e.g., the first block dimension) of the data block.

[0090] In some embodiments, the starting coordinates of the data block of the (i+1)th sub-instruction are dynamically updated based on the changes in the starting coordinates of the data block of the current i-th sub-instruction and the remaining size of the tensor to be processed in the corresponding dimension. Specifically, the coordinate update judgment can start from the first block dimension, comparing the remaining unprocessed size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension of the data block with the fixed size of the data block in the first block dimension. As mentioned above, the size of the data block in a specific block dimension is determined by a preset shape size. In some embodiments, this preset shape size applies to all data blocks segmented from the current tensor to be processed, that is, all data blocks corresponding to all sub-instructions adopt a uniform preset shape size (e.g., a fixed M×N matrix size). This uniform size design simplifies the step accumulation of the hardware state machine and avoids complex size recalculation.

[0091] When the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is greater than or equal to the size of the data block in the first block dimension (corresponding to step S402), this means that in the current tensor dimension (e.g., the current row or column), there are still sufficient continuous data elements to be extracted from the tensor to be processed, and the physical boundary of the tensor to be processed in this dimension has not yet been reached. In this case, a coordinate "translation" operation is performed: the starting coordinates of the data block of the (i+1)th sub-instruction are determined to be the sum of the starting coordinates of the data block of the current i-th sub-instruction in the first block dimension and the preset size of the data block in this dimension. Intuitively, this is similar to a fixed-size reading window sliding forward one data block span in the current tensor dimension, thus smoothly pointing to the reading starting point of the next adjacent data block.

[0092] When the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is smaller than the size of the data block in the first block dimension (corresponding to step S404), this means that in the current tensor dimension, the remaining valid data is insufficient to fill the next data block, or the boundary of the tensor to be processed in that dimension has been completely reached. At this point, it is not possible to continue stepping in the same direction in that dimension, otherwise data from non-interesting regions will be read. Therefore, the state machine can perform a reset operation: the starting coordinates of the data block of the (i+1)th sub-instruction in the first block dimension will be reset or returned to the basic coordinate values ​​corresponding to that first block dimension in the initial starting coordinates. This operation causes the coordinates in the first block dimension to return to the starting point of the row where the tensor to be processed is located.

[0093] Figure 5 An exemplary flowchart of a method 500 for determining the starting coordinates of a data block in a block dimension higher than a first dimension, according to an embodiment of the present disclosure, is shown. The second block dimension has a lower priority in hardware traversal than the first block dimension, but a higher priority in spatial dimension hierarchy (e.g., if the first block dimension is the X-axis containing the width W, then the second dimension could be the Y-axis containing the height H).

[0094] like Figure 5 As shown, method 500 includes steps S502 and S504, wherein step S504 further includes sub-steps S5041 and S5042.

[0095] In step S502, in response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is greater than or equal to the size of the data block in the first block dimension, the coordinates of the second block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined as the coordinates of the second block dimension in the starting coordinates of the data block of the ith sub-instruction.

[0096] Specifically, step S502 describes a coordinate-preserving scenario. When there is still sufficient data in the lowest dimension (the first block dimension) for the sliding window to continue moving forward, the position of the coordinates in the higher dimension (the second block dimension) does not need to change. Intuitively, assuming the first block dimension corresponds to a row-major dimension from left to right and the second dimension corresponds to a column-major dimension from top to bottom, when it is possible to move to the right to read the next data block in the current data row, the downward row number of that data block (i.e., the coordinates of the second block dimension) remains unchanged. This simplifies the computational load of the hardware in most regular stepping cycles.

[0097] In step S504, in response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is less than the size of the data block in the first block dimension, the coordinates of the second block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined.

[0098] Just like a combination Figure 4 As described, when the first dimension touches the tensor boundary, causing its coordinates to be reset, this reset action acts as a carry signal, triggering the evaluation and updating of the coordinates of the second dimension, which is a higher-order adjacent dimension. Specifically, determining the coordinates of the second dimension in the starting coordinates of the data block of the (i+1)th sub-instruction includes executing steps S5041 and S5042.

[0099] In step S5041, in response to determining that the remaining size of the tensor to be processed in the second tensor dimension corresponding to the second block dimension is greater than or equal to the size of the data block in the second block dimension, the coordinates of the second block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined as the sum of the coordinates of the second block dimension in the starting coordinates of the data block of the i-th sub-instruction and the size of the data block in the second block dimension.

[0100] Step S5041 describes the scenario where a carry is successful and a dimension switch occurs. When a carry signal is received from the first dimension, it is determined whether there is enough remaining data in the second tensor dimension to accommodate the next data block. If there is sufficient remaining space, the second dimension is incremented by the size of one data block. Combined with the reset action of the first dimension, this is physically manifested as the sliding window returning to the beginning of the row and moving down by the height of a full row of data blocks, thus beginning continuous extraction of a new row.

[0101] In step S5042, in response to determining that the remaining size of the tensor to be processed in the second tensor dimension corresponding to the second block dimension is less than the size of the data block in the second block dimension, the coordinates of the second block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined as the coordinates of the second block dimension in the initial starting coordinates.

[0102] Step S5042 describes the scenario where an out-of-bounds reset also occurs in the second dimension. When it is determined that the remaining valid data in the second tensor dimension is insufficient to fill the next data block (i.e., the physical boundary of the second tensor dimension has been reached), it is impossible to continue stepping in that dimension, and the coordinates of the second dimension need to be reset back to the initial starting coordinates.

[0103] Understandably, the above description illustrates a general multi-dimensional judgment criterion for coordinate determination during block traversal in a multi-dimensional tensor space. Similarly, when the second dimension triggers a reset (i.e., step S5042 occurs), this reset action serves as a new carry signal, passed to the higher third dimension. If the remaining size of the third dimension is sufficient, the third dimension advances; if not, the third dimension also resets, and the process continues to carry to higher dimensions.

[0104] Therefore, the above-mentioned combination judgment based on two adjacent block dimensions can serve as a basic loop unit, seamlessly extending and fully covering three-dimensional data block traversal scenarios that include three dimensions such as width, height, and depth, and even tensor traversal of higher dimensions.

[0105] To more intuitively illustrate the specific parameter changes during this carry process, the following will take a data block containing three dimensions, X, Y, and Z, as an example and illustrate in detail the dynamic update and transformation process of each dimension coordinate under all possible boundary triggering conditions through a specific coordinate update parameter table.

[0106] Table 1

[0107] Table 1 shows a data block start coordinate update table according to an embodiment of the present disclosure.

[0108] In the example in Table 1, the data block is a three-dimensional data block represented by XYZ dimensions. Here, X dimension represents the lowest block dimension (i.e., the fastest changing dimension, corresponding to the first block dimension mentioned above), Y dimension represents the second lowest block dimension (corresponding to the second block dimension mentioned above), and Z dimension represents the highest block dimension. It is understood that if the tensor to be processed is stored in NDHWC format, XYZ can be flexibly mapped to any combination of three consecutive or adjacent dimensions in NDHWC (e.g., mapped to W, H, D, or C, W, H, etc.), and this disclosure does not impose any limitations on this.

[0109] To facilitate understanding, the parameters in Table 1 will be explained first.

[0110] The "dimension coordinates" (e.g., x_coord, y_coord, z_coord) in Table 1 represent the starting coordinates of the data block of the current i-th sub-instruction in each block dimension. After the coordinate transformation condition is determined, the new coordinates represented in the "dimension coordinates updated to" column will be used as the starting coordinates of the data block of the next (i+1)-th sub-instruction in the corresponding block dimensions.

[0111] block_x, block_y, and block_z represent the preset shape and size of the data block in each corresponding block dimension.

[0112] remain_copy_x, remain_copy_y, and remain_copy_z represent the remaining size of the tensor to be processed in each corresponding tensor dimension, that is, the length of valid data that has not yet been loaded from the physical boundary.

[0113] x_coord_b, y_coord_b and z_coord_b represent the initial starting coordinates of the tensor to be processed under the macroscopic multi-dimensional coordinate system, that is, the basic coordinate position that needs to be returned when the data block is reset (such as line feed or plane change).

[0114] With reference to Table 1, the update process of the starting coordinates is detailed as follows.

[0115] For the lowest-dimensional X dimension, the remaining size in the X dimension is first evaluated. If the remaining size of the X dimension is greater than or equal to the data block size of the X dimension (that is, remain_copy_x >= block_x), it indicates that the current line has not been read to the boundary, then the X coordinate is translated forward by the size of one data block in the X dimension, and the starting coordinate is updated to x_coord + block_x. If the remaining size of the X dimension is smaller than the data block size of the X dimension (that is, remain_copy_x < block_x), it indicates that the boundary is touched. At this time, the X coordinate triggers a reset operation, returns to the initial starting coordinate of the line head in this dimension (that is, updated to x_coord_b), and sends a carry signal to a higher dimension.

[0116] For the Y dimension, which is the second lowest dimension, the update of its coordinate depends on whether the X dimension is reset, that is, whether a carry signal is generated. Specifically, if the X dimension has not touched the boundary (that is, remain_copy_x >= block_x), no carry signal is generated, and the Y coordinate remains unchanged (that is, updated to y_coord). If the X dimension triggers a reset (that is, remain_copy_x < block_x), and the remaining size of the Y dimension itself is greater than or equal to the data block size of the Y dimension (that is, remain_copy_y >= block_y), the Y coordinate steps down by the size of one data block in the Y dimension (that is, updated to y_coord + block_y), so as to realize line feed in the physical sense. If the X dimension triggers a reset (that is, remain_copy_x < block_x), but the remaining size of the Y dimension itself is also insufficient (that is, remain_copy_y < block_y), it indicates that all data in the current two-dimensional plane (XY plane) have been read. At this time, the Y coordinate also triggers a reset operation (that is, updated to y_coord_b), and sends a carry signal to the higher dimension Z.

[0117] For the highest-dimensional Z dimension, the update condition for its coordinate is more complicated: it requires that the X and Y dimensions lower than the Z dimension must be reset simultaneously to trigger the coordinate change on the Z dimension. As long as the X dimension is not reset (i.e., remain_copy_x>=block_x) or the Y dimension is not reset (i.e., remain_copy_y>= block_y), this means that the traversal of the two dimensions lower than the Z dimension has not been completed, and no carry signal transmitted to the Z dimension is generated, so the Z coordinate always remains unchanged (i.e., updated to z_coord). When and only when both the X dimension and the Y dimension trigger reset (i.e., remain_copy_x<block_x and remain_copy_y<block_y), if the remaining size of the Z dimension itself is greater than or equal to the data block size of the Z dimension (i.e., remain_copy_z>= block_z), the Z coordinate will step forward by the size of one data block in the Z dimension (i.e., updated to z_coord +block_z), realizing entry into the next two-dimensional slice or surface change in the physical sense. If both the X dimension and the Y dimension trigger reset, and the remaining size of the Z dimension is also insufficient (i.e., remain_copy_z<block_z), the Z coordinate is reset at this time (i.e., updated to z_coord_b). In the context of three-dimensional block partitioning, this usually means that the traversal allocation of starting coordinates has been completed for all data blocks in the tensor to be processed.

[0118] Through the coordinate update process shown in Table 1, the data loading method of the present disclosure does not need to rely on any loop control instructions at the software side. Only relying on the parallel comparison process between the remaining size and the preset block size, continuous, gridded starting physical coordinates of data blocks can be automatically generated efficiently in any dimensional space. The large tensor originally distributed in an interleaved or linear manner is decomposed into data blocks that are most easily processed by computing cores, thereby completing the efficient rearrangement from a linear layout to a blocked layout in the data handling stage.

[0119] To further describe the dynamic update process when handling boundary truncation and coordinate carry, a specific example in which the size of each dimension cannot be divided exactly will be used for description below. It should be particularly noted that in the underlying hardware comparison process, the parameter "remaining size ( remain_copy )" does not refer to the total volume of the entire tensor that has not been handled, but refers to the remaining effective length of the starting coordinate of the current data block from the physical boundary of the tensor on a specific coordinate axis. In other words, this parameter is equal to the total size of the tensor in the dimension minus the current coordinate value.

[0120] Table 2

[0121] Table 2 shows an example update table of starting coordinates of sub-instructions according to an embodiment of the present disclosure.

[0122] As shown in Table 2, the total shape and size of the tensors to be processed in the first memory is 7×4×5, that is, the total dimensions in the X, Y, and Z tensor dimensions are 7, 4, and 5, respectively, and the initial starting coordinates are set to (0, 0, 0). At the same time, in order to adapt to the hardware throughput of the arithmetic logic unit, the preset shape and size of the three-dimensional data block is set to 3×3×2, that is, block_x=3, block_y=3, block_z=2.

[0123] Figures 6A-6C An exemplary schematic diagram of sub-instruction start coordinate update according to at least one embodiment of the present disclosure is shown.

[0124] Combine Table 2 and Figure 6A As shown, at the beginning of the traversal on the Z=0 plane, the first sub-instruction extracts a data block at (0, 0, 0). At this point, the remaining size of the X dimension is 7 (i.e., 7 minus 0), which is greater than or equal to the step size of 3. Therefore, the X coordinate is shifted forward and updated to 3. The second sub-instruction extracts a data block at (3, 0, 0). At this point, the remaining size of the X dimension becomes 4 (i.e., 7 minus 3), which is still greater than or equal to 3. The X coordinate continues to be shifted and updated to 6.

[0125] like Figure 6B As shown, when the third sub-instruction is executed and the data block is extracted at (6, 0, 0), the remaining size in the X dimension is only 1 (i.e., 7 minus 6). Since 1 is less than the block size 3, it indicates that the physical boundary has been reached in the X direction. At this time, an out-of-bounds reset in the X dimension and a carry judgment in the Y dimension are triggered: the X coordinate is reset to the initial value 0; at the same time, the current Y coordinate is 0, and its remaining size is 4 (i.e., 4 minus 0), which is greater than or equal to the block size 3, so the Y coordinate is updated downwards to 3. Since the Y dimension is not reset, the Z coordinate remains unchanged at 0. Thus, the starting coordinate of the fourth sub-instruction jumps to (0, 3, 0), completing the line break operation on the two-dimensional plane.

[0126] On the new data row with Y=3, the state machine continues to perform translation operations. The 4th and 5th sub-instructions extract data blocks at (0, 3, 0) and (3, 3, 0) respectively, causing the X coordinates to be updated to 3 and 6 respectively. When the 6th sub-instruction extracts a data block at (6, 3, 0), the remaining size of the X dimension becomes 1 again, triggering an X dimension reset to 0. At this time, the state machine re-evaluates the carry condition of the Y dimension: the current Y coordinate is 3, and its remaining size is only 1 (i.e., the total size 4 minus 3). Since the remaining size 1 is less than the block size 3, this indicates that the Y dimension has also completely crossed the physical boundary, thus triggering an out-of-bounds reset of the Y dimension and a carry process to the higher dimension Z. In this carry, the Y coordinate is reset to 0. The carry condition of the Z dimension is evaluated: the current Z coordinate is 0, and its remaining size is 5 (i.e., 5 minus 0), which is greater than or equal to the block size 2. Therefore, the Z coordinate is updated to 2 in the deeper dimension. Thus, the starting coordinates of the 7th sub-instruction were updated to (0, 0, 2), realizing a face swapping operation in a physical sense.

[0127] After entering the new plane with Z=2, the above two-dimensional scanning process is repeated. Sub-instructions 7 to 12 will extract data blocks sequentially at (0,0, 2), (3, 0, 2), (6, 0, 2), (0, 3, 2), (3, 3, 2), and (6, 3, 2). The position of the data block of the 8th sub-instruction in the tensor to be processed is as follows. Figure 6C As shown.

[0128] After processing the 12th subinstruction, the remaining dimensions in both the X and Y dimensions are less than their corresponding block dimensions, triggering a simultaneous reset of the X and Y coordinates to 0. At this point, the Z dimension is evaluated: the current Z coordinate is 2, and the remaining dimension is 3 (i.e., 5 minus 2), which is greater than or equal to the block size of 2. Therefore, the Z coordinate is updated again to 4, causing the starting coordinate of the 13th subinstruction to jump to (0, 0, 4).

[0129] In the final Z=4 plane, sub-instructions 13 through 18 extract data blocks sequentially at (0, 0, 4), (3, 0, 4), (6, 0, 4), (0, 3, 4), (3, 3, 4), and (6, 3, 4). After the data extraction of the 18th sub-instruction is completed, the remaining size in the X dimension is 1, triggering an X reset; the remaining size in the Y dimension is 1, triggering a Y reset; at this point, the Z dimension is evaluated, and the current Z coordinate is 4, with a remaining size of only 1 (i.e., 5 minus 4), which is less than the block size of 2. This indicates that the Z dimension has also reached the physical boundary of the highest dimension, thus triggering the final reset of the Z dimension. At this point, the coordinates of the X, Y, and Z dimensions are all reset back to (0, 0, 0), ending the carry loop. This means that the splitting and loading process of the 7×4×5 tensor, totaling 18 sub-instructions, is complete.

[0130] After the above detailed deduction of the dynamic generation process of the starting coordinates of the data block through specific three-dimensional examples, in order to illustrate the idea of ​​automatically decomposing the tensor loading instruction into multiple data block-level sub-instructions from an intuitive perspective, the following explanation is provided in conjunction with specific figures.

[0131] Figure 7 A schematic diagram illustrating data loading at the data block granularity according to an embodiment of the present disclosure is shown.

[0132] like Figure 7 As shown, the "tensor to be processed" on the left represents a macroscopic data structure stored in the first memory (e.g., high-bandwidth memory). In the software or hardware loading mode of related technologies, since this tensor is typically arranged in a specific linear format (e.g., NDHWC) in physical memory, if one attempts to directly extract a local sub-tensor spanning multiple rows, columns, or faces (e.g., ...) from it using a single instruction (e.g., a non-contiguous read instruction), the result may be difficult to obtain. Figure 7 The area marked by the shaded area on the left often faces serious memory address discontinuity problems, resulting in extremely low memory access efficiency.

[0133] However, in the embodiments of this disclosure, through the aforementioned coordinate update and stepping process (as described in Table 1 and related embodiments), a whole macroscopic instruction that is originally complex, large and physically non-contiguous is broken down into several microscopic sub-instructions that exist in the form of data blocks.

[0134] like Figure 7 As shown by the solid arrow connecting the left and right sides, the shaded area represents an independent data block whose starting coordinates are calculated by a state machine. The hardware only needs to use these automatically generated sub-instruction coordinates to perform gridded coverage and extraction of the entire tensor to be processed according to a specific dominant direction (e.g., an X-dimensional priority mode that prioritizes scanning along the X-axis, or a Y-dimensional priority mode that prioritizes scanning along the Y-axis).

[0135] Figure 7 The "data block" on the right visually illustrates the data structure after being loaded into the second memory (e.g., a cache buffer). Tensor elements that might have been physically scattered in the first memory due to row or plane spanning, once divided and loaded into data blocks by these independent sub-instructions, will be rearranged tightly within the second memory according to the preset shape and size of the data block, achieving contiguous storage (e.g., ...). Figure 7 The broken-line arrows inside the data block on the right indicate that data is written continuously and compactly in the cache.

[0136] This process of breaking down data loading instructions not only eliminates the huge computational overhead and programming complexity of rearranging multidimensional arrays in upper-level software, but more importantly, it transforms the complex non-contiguous memory access operations in the first memory into a series of efficient continuous read and write operations on specific small data blocks. As a result, computing units (such as arithmetic logic units or tensor cores) can directly and continuously obtain well-arranged, continuous, and appropriately sized matrix block data from the second memory without interruption, thereby greatly improving the hardware's computational utilization.

[0137] The following section will delve further into the micro-level, explaining the specific sub-instructions that are broken down (i.e., for each sub-instruction). Figure 7 (Any shaded data block in the memory), how the hardware actually performs the physical data loading from the first memory to the second memory, and handles possible boundary overflows.

[0138] In step S302 above, the data of the i-th data block is returned starting from the data block's starting coordinates, ensuring that the data in the i-th data block is stored continuously in the second memory. The macroscopic transformation and derivation rules for the starting coordinates of each data block have been described in detail above. The following will explain, according to embodiments of this disclosure, how the hardware, after determining the specific starting coordinates of a data block, correctly returns the specific data in the i-th data block based on those coordinates and stores it continuously in the second memory.

[0139] It's important to understand that because the original tensors to be processed are stored contiguously in the first memory according to a specific linear format (e.g., NDHWC), when retrieving a multidimensional data block, the elements within that block are often not completely contiguous in the physical address space of the first memory. Therefore, to load the i-th data block completely into the second memory, the hardware does not send a single coarse read instruction, but rather sends multiple requests sequentially to load the i-th data block. By sequentially and compactly writing the sub-data returned from the first memory by each request into the second memory, the complete data block is finally pieced together in the second memory.

[0140] Specifically, when partitioning these requests, the hardware dynamically evaluates the physical continuity of data blocks across various dimensions, identifying the dimensional boundaries where data blocks become discontinuous in physical memory. If the size relationship between the i-th data block and the tensor to be processed in the first tensor dimension prevents continuous loading of the data block in the first tensor dimension, but allows continuous loading in dimensions below the first tensor dimension, the hardware will partition the request along the tensor dimension to be partitioned. This partitioning occurs when the first tensor dimension is the same as or adjacent to the tensor dimension to be partitioned.

[0141] Specifically, the first tensor dimension can be understood as the tensor dimension where the physical addresses are not contiguous when the hardware attempts to load the data block continuously from the first memory. The tensor dimension to be divided is the tensor dimension on which the hardware ultimately decides to divide the loading task into multiple independent requests in order to ensure that each request can obtain an absolutely continuous segment of physical data.

[0142] In actual data arrangement, the relationship between the first tensor dimension and the tensor dimension to be partitioned depends strictly on the data storage format (e.g., NDHWC format or interleaved format) of the tensor to be processed in the first memory and the ascending order of its physical address. According to embodiments of this disclosure, this relationship is manifested as the first tensor dimension being the same as the tensor dimension to be partitioned, or the first tensor dimension being adjacent to the tensor dimension to be partitioned.

[0143] To illustrate this dimensional relationship more clearly, the following detailed explanation is provided in conjunction with specific storage format examples.

[0144] In some embodiments, it is assumed that the tensor to be processed is stored in the first memory in NDHWC format. In this format, the channel C dimension changes the fastest in terms of physical memory address, followed by the width W dimension. If, when the hardware attempts to load a data block, it finds that the length of the data block in the W dimension exceeds the physical boundary of the original tensor in the W dimension, then the W dimension is the first tensor dimension causing the data discontinuity. In this case, since the C dimension below the W dimension is still completely contiguous (i.e., the C dimension contains a complete set of channel data), the hardware will choose the dimension immediately below the W dimension that maintains absolute continuity as the partition boundary. In this situation, the first tensor dimension causing discontinuity is the W dimension, while the tensor dimension to be partitioned, which the hardware actually requests to extract a continuous row of channel data from, is adjacent to the W dimension. For example, in this scenario, it could be a lower-level C dimension or a higher-level H dimension, depending on the direction from which the hardware performs the partitioning. Therefore, this corresponds to the case where the first tensor dimension is adjacent to the tensor dimension to be partitioned. By splitting along adjacent dimensions, it can be ensured that each request can cover a continuous, untrunculated valid data segment in memory.

[0145] In other embodiments, it is assumed that the tensor to be processed is stored in the first memory in an interleaved mode (e.g., N(C / x)DHW(xC) format). In this particular format, because channels are divided into multiple groups containing x elements, the continuity of physical addresses is broken at the lowest dimension (i.e., the xC dimension). If, when loading a data block, the hardware finds that it cannot read it continuously even at the lowest xC dimension (e.g., the number of channels required for the data block spans different interleaved groups), then the xC dimension itself is the first tensor dimension causing the discontinuity. Since the xC dimension is already the lowest granularity of the physical memory arrangement, the hardware cannot look for a lower-level continuous dimension. Therefore, it can only directly and rigidly divide the loading task into multiple smaller request fragments at this discontinuous xC dimension. In this case, the first tensor dimension causing the discontinuity is the xC dimension, and the tensor dimension to be partitioned by the hardware is still this xC dimension. Therefore, this corresponds to the case where the first tensor dimension and the tensor dimension to be partitioned are the same.

[0146] Using the above method, the hardware in this embodiment can autonomously adapt to various complex linear storage formats such as NDHWC or interleaved modes. Whether splitting requests into entire rows / columns on adjacent dimensions or fragmenting requests on the same lowest dimension, the hardware always ensures that the data corresponding to each load request is contiguous in physical address. This not only maximizes the single burst read efficiency of the first memory, thereby maximizing memory access bandwidth, but also lays the foundation for subsequent zero-padding operations for out-of-bounds requests.

[0147] This partitioning rule means that the data corresponding to each request is either entirely within the valid data range of the tensor to be processed (i.e., all within the first memory), or entirely outside the valid data range of the tensor to be processed (i.e., none within the first memory).

[0148] To address the two scenarios described above, this disclosure employs the following different processing methods. If all requested sub-data is located in the first memory, this indicates that the request has hit valid data within the tensor to be processed. In this case, the requested sub-data comes directly from the tensor to be processed and is read continuously from the first memory. If none of the requested sub-data is located in the first memory, this indicates that during the aforementioned data block coordinate transformation process, this portion of the data block has exceeded the physical boundary of the tensor to be processed.

[0149] like Figure 6BIn the scenario shown, when the starting coordinates of the sub-instruction are (6,0,0) and the remaining size of the tensor to be processed in the X dimension is only 1, which cannot fill the 3×3×2 data block corresponding to the sub-instruction, the hardware does not initiate a real read operation on the first memory for this invalid request that exceeds the limit in the X dimension, so as to avoid wasting bus bandwidth or causing memory errors. Instead, it directly writes a predetermined value to the corresponding location in the second memory (e.g., performs zero-padding). In this way, the data block finally stored contiguously in the second memory not only contains the extracted valid tensor data, but also automatically completes the edge padding, so that the shape of the data block can maintain the predetermined shape and size, adapting to the throughput requirements of the subsequent arithmetic logic unit for the regular matrix.

[0150] To efficiently and automatically execute the aforementioned request partitioning based on the continuity dimension and the zero-padding judgment for out-of-bounds requests, this disclosure introduces a state machine mechanism in the hardware. The processes of sequentially sending multiple requests and determining data validity are all executed by this hardware state machine.

[0151] When using a state machine to extract the i-th data block, the state machine first initializes itself using the starting coordinates and preset shape and size of the data block. Specifically, based on the starting coordinates and shape and size of the data block, the first request for that data block is determined, and the initial state of the first request entering the state machine is also determined. For example, the initial state could be an assessment of whether the starting point of the data block falls to the left of the tensor boundary, inside the tensor, or to the right of the tensor boundary. After determining the first request and its initial state, the state machine, in conjunction with the initial state and the shape and size of the tensor to be processed, determines each subsequent request among the multiple requests, excluding the first request. The state machine automatically switches between a "zero-padding state" and a "read state" based on the execution status of the current request and the relative distance to the tensor boundary, until all data requests required for the i-th data block are generated and executed.

[0152] The following describes the actual working mechanism of a state machine. A state machine can be designed to contain three basic states: a first state (e.g., S0), a second state (e.g., S1), and a third state (e.g., S2). These three states correspond to the relative positions of the current request in the physical memory space. Specifically, the first state (S0) represents that the entire data segment pointed to by the current request falls to the "left" of the valid physical boundary of the tensor to be processed (i.e., an overflow has occurred); the second state (S1) represents that the entire data segment pointed to by the current request falls to the "inside" of the valid physical boundary of the tensor to be processed (i.e., valid data has been hit); and the third state (S2) represents that the entire data segment pointed to by the current request falls to the "right" of the valid physical boundary of the tensor to be processed (i.e., an overflow has occurred).

[0153] The state machine will automatically determine and switch between the three states mentioned above based on the current execution status of the request and the relative distance to the tensor boundary, and perform corresponding actions: The first step is the determination of the initial state and the execution of actions. When extracting the i-th data block, the state machine compares the first coordinate value of the starting coordinate of the data block in the first tensor dimension where discontinuity occurs (e.g., the W dimension where line breaks occur as mentioned above) with the starting boundary value (second coordinate value) of the original tensor in that dimension.

[0154] If the first coordinate value is less than the second coordinate value, it means that the starting point of the data block is located outside the "left" of the effective tensor range. At this time, the state machine determines the initial state and enters the first state (S0). In the first state (S0), the hardware action is to directly write a predetermined value (e.g., padding with zeros) to the corresponding continuous address range in the second memory, without sending a real memory read request to the first memory.

[0155] If the first coordinate value is greater than or equal to the second coordinate value, and less than the end boundary value of the original tensor in that dimension (the third coordinate value, i.e., the second coordinate value plus the size of the original tensor in that dimension), it means that the starting point of the data block falls "inside" the effective tensor range. At this time, the state machine determines that the initial state enters the second state (S1). In the second state (S1), the hardware actions are: assembling and sending a real memory read request to the first memory, and continuously writing the returned effective sub-data into the second memory.

[0156] If the first coordinate value is greater than or equal to the third coordinate value, it means that the starting point of the data block has completely crossed the "right" boundary of the effective tensor range. At this time, the state machine determines that the initial state has entered the third state (S2). In the third state (S2), the hardware action is similar to that in S0: zeros are directly added to the second memory.

[0157] Secondly, there is the dynamic switching process between states. After the current request is completed (whether zero padding or reading is performed in S0, S1 or S2), the state machine automatically calculates the coordinates of the next request based on the length of data processed by the request, and re-evaluates its relationship with the tensor boundary, thereby determining which state the next request will enter.

[0158] For example, if the current state is the first state (S0) and after zero padding, the coordinates of the next request are still less than the starting boundary of the valid tensor (i.e., have not yet entered the tensor), the state machine remains in the first state (S0) and continues to pad with zeros; if the coordinates of the next request cross the starting boundary and enter the tensor, the state machine jumps from the first state (S0) to the second state (S1) and begins to send the actual read request.

[0159] Similarly, if the current state is the second state (S1) and the coordinates of the next request are still inside the tensor after reading valid data, the state machine remains in the second state (S1); if the read operation causes the coordinates to cross the end boundary of the tensor, the state machine jumps from the second state (S1) to the third state (S2) and begins to fill the out-of-bounds part on the right with zeros.

[0160] Finally, if the current state is the third state (S2), since the right boundary has been crossed, all remaining requests for this data block in the current dimension will fall outside the valid area. Therefore, the state machine will always remain in the third state (S2) and continuously fill in zeros until all the sub-data requests required for the i-th data block in the current dimension have been generated and executed.

[0161] Through the rigorous definition of the above states (S0, S1, S2), and the explicit jump conditions based on the relationship between coordinates and boundary size, and the corresponding read / write / zero-padding actions, the hardware state machine of this disclosure can automatically pad invalid parts with zeros and extract valid parts completely when extracting any data blocks that may be partially out of bounds, and finally assemble a regular matrix block that meets the requirements of the computing unit in the second memory.

[0162] The data loading method disclosed herein splits a single linear load instruction into multiple precise data block start coordinates. For each segmented data block, a hardware state machine further breaks it down into consecutive physical memory requests, intelligently padding out-of-bounds data with zeros. Using this method, the embodiments of this disclosure overcome the reliance on software rearrangement instructions in related technologies, achieving a seamless conversion from large-capacity linear layout storage to high-speed cache block layout storage without additional software intervention.

[0163] Figure 8 An exemplary block diagram of a data loading apparatus according to an embodiment of the present disclosure is shown. It can be utilized... Figure 8 The system shown is used to execute Figure 2 Method 200 is described.

[0164] like Figure 8 As shown, in some embodiments, the data loading device 800 includes an instruction acquisition module 802, an instruction parsing module 804, and a sub-instruction execution module 806. In some embodiments, the functions of the instruction acquisition module 802 and the instruction parsing module 804 can be combined and implemented by a single integrated instruction parsing unit.

[0165] The instruction acquisition module 802 is used to acquire the loading instruction for loading the tensor to be processed from the first memory to the second memory.

[0166] The instruction parsing module 804 is used to determine the shape and size of the tensor to be processed and the initial starting coordinates in the coordinate system determined by the tensor to be processed.

[0167] The sub-instruction execution module 806 is used to execute the load instruction by sequentially executing Z sub-instructions, starting from the initial starting coordinates.

[0168] Where Z is the total number of sub-instructions and is an integer greater than 1. For the i-th sub-instruction among the Z sub-instructions, the following operations are performed: return the data of the i-th data block starting from the starting coordinates of the data block, so that the data in the i-th data block is stored continuously in the second memory. Determine the starting coordinates of the data block of the (i+1)-th sub-instruction based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in each block dimension, and the remaining size of the tensor to be processed in each corresponding tensor dimension. Where i is an index number less than Z, and the total shape size of the Z data blocks corresponding to the Z sub-instructions is not less than the shape size of the tensor to be processed.

[0169] It should be understood that Figure 8 The various modules or units of the device 800 shown can be used with reference to Figures 2 to 5 The steps in methods 200, 300, 400, and 500 described correspond to each other. Therefore, the operations, features, and advantages described above for these methods also apply to device 800 and its included modules and units. For the sake of brevity, some operations, features, and advantages are not repeated here.

[0170] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific actions performed by the modules discussed herein include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that performs the action, called or otherwise accessed by the specific module.

[0171] According to one aspect of this disclosure, a processor is also provided. Figure 9 An exemplary block diagram of a processor according to an embodiment of the present disclosure is shown. Figure 9 The processor 900 shown may include a first memory 902, a second memory 904, an instruction parsing unit 906, and a data loading unit 908.

[0172] The first memory 902 can be used to store tensors to be processed in a linear layout.

[0173] The second memory 904 can be used to store tensors to be processed in a block layout based on data blocks of a preset shape and size.

[0174] The instruction parsing unit 906 can be used to obtain a load instruction that loads the tensor to be processed from the first memory to the second memory.

[0175] The data loading unit 908 is coupled to the first memory, the second memory, and the instruction parsing unit. The data loading unit can be configured to execute the data loading method according to the embodiments of the present disclosure in response to a loading instruction.

[0176] In some embodiments, the data loading unit 908 may be implemented as a direct memory access controller, a load / store unit, or a dedicated tensor memory access unit. The data loading unit 908 may internally have a hardware unit that implements the state machine of this disclosure.

[0177] It is understood that the processor 900 can be implemented to perform the data loading method according to embodiments of the present disclosure. For details regarding the specific implementation of the data loading method, please refer to the description above, which will not be repeated here.

[0178] According to one aspect of this disclosure, an electronic device is also provided. Figure 10 An exemplary block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 10 As shown, the electronic device 1000 includes a memory 1020, a processor 1040, and a computer program stored in the memory. The memory 1020 and the processor 1040 may be coupled via an internal bus or other communication mechanism. The processor 1040 is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0179] In some embodiments, the processor 1040 may employ, for example... Figure 1 or Figure 9 The processor's microarchitecture design is shown.

[0180] Figure 11 A block diagram of an example computer device 2000 according to an embodiment of the present disclosure is shown.

[0181] The computer device 2000 may specifically correspond to the electronic device or processing system in the foregoing embodiments. It should be noted that... Figure 11 The components of the computer device 2000 shown are merely exemplary and not restrictive. Depending on the actual application requirements, the computer device 2000 may have other components or omit some components.

[0182] like Figure 11As shown, computer device 2000 includes processing device 2010. Processing device 2010 can perform various appropriate actions and processes according to computer-executable instructions stored in read-only memory (ROM) 2020 or loaded from storage device 2080 into random access memory (RAM) 2030. For example, processing device 2010 can be configured to execute the above-referenced... Figures 2 to 7 The described data loading or data storage method. Various programs and data required for the operation of the computer device 2000 can also be temporarily stored in RAM 2030. The processing device 2010, ROM 2020, and RAM 2030 are interconnected and exchange data via bus 2040. Input / output (I / O) interface 2050 is also connected to bus 2040.

[0183] In some embodiments, the processing device 2010 may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor (TPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. In specific implementations of embodiments of this disclosure, the processing device 2010 may have a built-in or coupled data loading unit and hardware state machine logic as described above to directly process tensor block loading tasks from mass storage (such as system memory) to on-chip cache (such as local cache) at the underlying level.

[0184] Typically, the following devices can be connected to the I / O interface 2050: input devices 2060 including, for example, touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 2070 including, for example, liquid crystal displays (LCDs), light-emitting diode displays (LEDs), speakers, vibrators, etc.; storage devices 2080 including, for example, magnetic tapes, hard disks, solid-state drives (SSDs), flash memory cards, etc.; and communication devices 2090.

[0185] The storage device 2080 can serve as a high-capacity, non-volatile storage medium for persistently storing the operating system, applications, and raw tensor data to be processed, such as large-scale neural network weights and massive input feature maps. The communication device 2090 allows the computer device 2000 to exchange data with other devices (such as cloud servers, other computing nodes in a distributed computing cluster, etc.) via a network, either wirelessly or via wired communication. The communication device 2090 may include a local area network (LAN) card, a wireless local area network (WLAN) card, a Bluetooth module, a near field communication (NFC) module, a cellular communication module, etc.

[0186] Although Figure 11A computer device 2000 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the computer device 2000 may alternatively implement or possess more or fewer devices. For example, when the computer device 2000 is used as an AI computing accelerator card or server node in a data center, it may not include input devices 2060 and output devices 2070 for human-computer interaction such as touch screens and keyboards, but may mainly consist of a processing unit 2010, high-bandwidth memory, and high-speed communication interfaces.

[0187] The functions of the systems, apparatuses, modules, or units described above herein can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on. Furthermore, the methods described in this disclosure can also be implemented by software or microcode containing computer-executable instructions that, when executed by the processing device 2010, cause the machine to perform the operations defined in this disclosure.

[0188] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided. Figure 12 An exemplary block diagram of a computer-readable storage medium according to embodiments of the present disclosure is shown. Figure 12 As shown, a computer-readable storage medium 3000 stores computer-executable instructions 3010, which, when executed by a processor, implement the steps of any of the method embodiments described above.

[0189] In some embodiments, the computer-readable storage medium 3000 may include any combination of one or more computer program products. For example, without limitation, the non-transitory computer-readable storage medium may include volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0190] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0191] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of this disclosure is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A data loading method for converting a tensor to be processed from linear layout storage to block layout storage based on data blocks of a preset shape and size, wherein, The data in each block dimension of the data block corresponds to the continuous data in each tensor dimension of the tensor to be processed, which corresponds to each block dimension respectively. The method includes: Obtain a load instruction to load the tensor to be processed from the first memory to the second memory; Determine the shape and size of the tensor to be processed and the initial starting coordinates in the coordinate system determined by the tensor to be processed; Starting from the initial coordinates, the load instruction is executed by sequentially executing Z sub-instructions, where Z is the total number of sub-instructions and is an integer greater than 1. For the i-th sub-instruction among the Z sub-instructions, the following operation is performed: Starting from the beginning coordinates of the data block, the data of the i-th data block is returned, so that the data in the i-th data block is stored contiguously in the second memory. The starting coordinates of the data block for the (i+1)th sub-instruction are determined based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in each block dimension, and the remaining size of the tensor to be processed in the corresponding tensor dimensions. Where i is an index number less than Z, and the total shape size of the Z data blocks corresponding to the Z sub-instructions is not less than the shape size of the tensor to be processed.

2. The data loading method according to claim 1, wherein, The size of the data block in the block dimension is less than or equal to the size of the tensor dimension in the tensor to be processed that corresponds to the block dimension.

3. The data loading method according to claim 1, the method further includes using the initial starting coordinates as the starting coordinates of the data block of the first sub-instruction among the Z sub-instructions.

4. The data loading method according to claim 1, wherein, The determination of the starting coordinates of the data block for the (i+1)th sub-instruction based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in the first block dimension, and the remaining size of the tensor to be processed in the first tensor dimension includes: In response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is greater than or equal to the size of the data block in the first block dimension, the coordinate of the first block dimension in the starting coordinate of the data block of the (i+1)th sub-instruction is determined as the sum of the coordinate of the first block dimension in the starting coordinate of the data block of the i-th sub-instruction and the size of the data block in the first block dimension; In response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is less than the size of the data block in the first block dimension, the coordinates of the first block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined as the coordinates of the first block dimension in the initial starting coordinates.

5. The data loading method according to claim 4, wherein, The block dimension also includes a second block dimension higher than the first block dimension. Determining the starting coordinates of the data block for the (i+1)th sub-instruction based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in the first block dimension, and the remaining size of the tensor to be processed in the first tensor dimension further includes: In response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is greater than or equal to the size of the data block in the first block dimension, the coordinates of the second block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined as the coordinates of the second block dimension in the starting coordinates of the data block of the i-th sub-instruction; In response to determining that the remaining size of the tensor to be processed in the first tensor dimension corresponding to the first block dimension is less than the size of the data block in the first dimension, the following operations are performed: In response to determining that the remaining size of the tensor to be processed in the second tensor dimension corresponding to the second block dimension is greater than or equal to the size of the data block in the second block dimension, the coordinates of the second block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined as the sum of the coordinates of the second block dimension in the starting coordinates of the data block of the ith sub-instruction and the size of the data block in the second block dimension; In response to determining that the remaining size of the tensor to be processed in the second tensor dimension corresponding to the second block dimension is less than the size of the data block in the second block dimension, the coordinates of the second block dimension in the starting coordinates of the data block of the (i+1)th sub-instruction are determined as the coordinates of the second block dimension in the initial starting coordinates.

6. The data loading method according to any one of claims 1-5, wherein, The step of returning the data of the i-th data block starting from the starting coordinates of the data block, so that the data in the i-th data block is stored continuously in the second memory, includes: Multiple requests for loading the i-th data block are sent sequentially, and the sub-data returned by each request is written sequentially into the second memory to load the data block into the second memory; Wherein, in response to the size relationship between the i-th data block and the tensor to be processed in the first tensor dimension such that when loading the data block, it cannot be loaded continuously in the first tensor dimension but can be loaded continuously in each dimension below the first tensor dimension, the requested partitioning is performed on the tensor dimension to be partitioned, and each requested sub-data is located in the first memory or not located in the first memory at all. Wherein, the first tensor dimension is the same as the tensor dimension to be divided, or the first tensor dimension is adjacent to the tensor dimension to be divided.

7. The data loading method according to claim 6, wherein, In response to the fact that all the requested sub-data is located in the first memory, the requested sub-data comes from the tensor to be processed and is stored continuously in the first memory; in response to the fact that none of the requested sub-data is located in the first memory, a predetermined value is written to the second memory.

8. The data loading method according to claim 6, wherein, The first request among the plurality of requests and the initial state of the first request entering the state machine are determined based on the starting coordinates of the data block of the i-th data block and the shape and size of the data block. In combination with the initial state, based on the shape and size of the tensor to be processed, the state machine is used to determine each of the plurality of requests other than the first request.

9. The data loading method according to any one of claims 1 to 5, wherein, The tensor to be processed is stored in the NDHWC data storage format, where N represents the batch dimension, D represents the depth dimension, W represents the width dimension, H represents the height dimension, and C represents the number of channels dimension.

10. The data loading method according to any one of claims 1 to 5, wherein, The tensor to be processed is stored in the data storage format N(C / x)DHW(xC), where N represents the batch dimension, D represents the depth dimension, W represents the width dimension, H represents the height dimension, C represents the channel number dimension, and x is a positive integer greater than 1.

11. The data loading method according to claim 1, wherein, The preset shape and size of the data block are determined based on the matrix size supported by the processor's arithmetic logic unit in a single calculation cycle.

12. A data loading apparatus for converting a tensor to be processed from linear layout storage to block layout storage based on data blocks of a preset shape and size, wherein, The data in each block dimension of the data block corresponds to the continuous data in each tensor dimension of the tensor to be processed, which corresponds to each block dimension respectively. The apparatus includes: The instruction acquisition module is used to acquire a loading instruction to load the tensor to be processed from the first memory to the second memory; The instruction parsing module is used to determine the shape and size of the tensor to be processed and the initial starting coordinates in the coordinate system determined by the tensor to be processed; The sub-instruction execution module is used to execute the load instruction by sequentially executing Z sub-instructions, starting from the initial starting coordinates, where Z is the total number of sub-instructions and is an integer greater than 1. For the i-th sub-instruction among the Z sub-instructions, the following operation is performed: Starting from the beginning coordinates of the data block, the data of the i-th data block is returned, so that the data in the i-th data block is stored contiguously in the second memory. The starting coordinates of the data block for the (i+1)th sub-instruction are determined based on the starting coordinates of the data block of the i-th sub-instruction, the size of the data block in each block dimension, and the remaining size of the tensor to be processed in the corresponding tensor dimensions. Where i is an index number less than Z, and the total shape size of the Z data blocks corresponding to the Z sub-instructions is not less than the shape size of the tensor to be processed.

13. A processor, comprising: The first memory is used to store the tensors to be processed in a linear layout; The second memory is used to store the tensor to be processed in a block layout based on data blocks of a preset shape and size; The instruction parsing unit is used to obtain a loading instruction for loading the tensor to be processed from the first memory to the second memory; as well as A data loading unit is coupled to the first memory, the second memory, and the instruction parsing unit, and the data loading unit is configured to execute the data loading method according to any one of claims 1-11 in response to the loading instruction.

14. An electronic device, comprising: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the data loading method according to any one of claims 1-11.

15. A non-transitory computer-readable storage medium, wherein, The non-transitory computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the data loading method according to any one of claims 1-11.

Images