Data processing method, data loading method and device, equipment, and storage medium

Abstract

This disclosure provides a data processing method, a data loading method, an apparatus, a device, and a storage medium. The method may include: obtaining a data processing instruction for performing a quantization operation on tensor data in memory; obtaining a data row range and a data column range associated with the tensor data; for the tensor data, sequentially performing the following processing on each row in the data row range: requesting the tensor data row of the current row in the data column range from memory; obtaining the scaling factor row corresponding to the quantization operation of the tensor data row, and storing the obtained scaling factor row in a register at the row position corresponding to the current row; and in response to determining that the scaling factors stored in the register form a scaling factor block with a predetermined storage layout, sending the scaling factor block to a buffer in a burst transfer manner, wherein one burst transfer corresponds to one base address, so that the scaling factors in the scaling factor block are stored contiguously in the buffer.

Description

Technical Field

[0001] This disclosure relates to a data processing method, a data loading method and apparatus, device, and storage medium. Background Technology

[0002] In general computing, computing units typically process large amounts of data when performing computational tasks, and this data is generally stored in memory or buffers. The way data is stored and retrieved in memory or buffers affects memory access bandwidth, and consequently, the hardware utilization of the computing unit.

[0003] In some computing scenarios, data can be quantized to reduce storage overhead or improve computational efficiency. During quantization, a scaling factor (SF) may be generated that is associated with the quantized data. The way the scaling factor is stored and loaded also affects the overall data access efficiency. Summary of the Invention

[0004] This disclosure provides a data processing method, a data loading method, an apparatus, a computer device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a data processing method is provided, the method comprising: obtaining a data processing instruction for performing a quantization operation on tensor data in memory; obtaining a data row range and a data column range associated with the tensor data; and, for each row in the data row range, sequentially performing the following processing: requesting the tensor data row of the current row of the tensor data within the data column range from memory; in response to obtaining the tensor data row returned from memory, obtaining a scaling factor row corresponding to the quantization operation of the tensor data row, and storing the obtained scaling factor row in a register at the row position corresponding to the current row; and in response to determining that the scaling factors stored in the register form a scaling factor block with a predetermined storage layout, sending the scaling factor block to a buffer in a burst transfer manner, wherein a burst transfer corresponds to a base address, so that the scaling factors in the scaling factor block are stored contiguously in the buffer.

[0006] According to another aspect of this disclosure, a data loading method is provided, the method comprising: receiving a data loading instruction for loading scaling factors from a buffer, wherein the scaling factors are generated via a quantization operation performed on tensor data, and the scaling factors are stored in the buffer in units of scaling factor blocks using a predetermined storage layout, wherein scaling factors in the same scaling factor block are stored contiguously in the buffer, and each scaling factor in the scaling factor block corresponds to a base address; and responding to the data loading instruction, performing the following operations: determining the number of scaling factors to be loaded per row in an execution cycle based on the data format and dot product operation scale associated with the data loading instruction; and repeatedly accessing the base address of the same scaling factor block in one or more consecutive execution cycles, based on the number of scaling factors to be loaded, to load the scaling factors in the same scaling factor block.

[0007] According to another aspect of this disclosure, an apparatus for a data processing method is provided, the apparatus comprising: an instruction receiving unit configured to receive a data processing instruction for performing a quantization operation on tensor data in memory; a data range determining unit configured to receive a data row range and a data column range associated with the tensor data; a data processing unit configured to perform the following processing sequentially on each row in the data row range for the tensor data: a data requesting unit configured to request a tensor data row in the data column range for the current row of the tensor data from memory; a scaling factor acquisition unit configured to, in response to receiving a tensor data row returned from memory, obtain a scaling factor row corresponding to the quantization operation of the tensor data row, and store the obtained scaling factor row in a register at the row position corresponding to the current row; and a sending unit configured to, in response to determining that the scaling factors stored in the register form a scaling factor block with a predetermined storage layout, send the scaling factor block to a buffer in a burst transmission manner, wherein each burst transmission corresponds to a base address, so that the scaling factors in the scaling factor block are stored continuously in the buffer.

[0008] According to another aspect of this disclosure, an apparatus for a data loading method is provided, the apparatus comprising: an instruction receiving unit configured to receive a data loading instruction for loading scaling factors from a buffer, wherein the scaling factors are generated via a quantization operation performed on tensor data, and the scaling factors are stored in the buffer in units of scaling factor blocks using a predetermined storage layout, wherein scaling factors in the same scaling factor block are stored contiguously in the buffer, and each scaling factor in the scaling factor block corresponds to a base address; a data processing unit configured to perform the following operations in response to the data loading instruction: a scaling factor determining unit configured to determine the number of scaling factors to be loaded in each row within an execution cycle based on the data format and dot product operation scale associated with the data loading instruction; and a scaling factor access unit configured to repeatedly access the base address of the same scaling factor block within one or more consecutive execution cycles based on the number of scaling factors to be loaded, to load the scaling factors within the same scaling factor block.

[0009] According to another aspect of this disclosure, a computer device is provided, the computer device comprising: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the aforementioned method.

[0010] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, causes the processor to perform the aforementioned method.

[0011] According to another aspect of this disclosure, a computer program product is provided, which includes a computer program that, when executed by a processor, causes the processor to perform the aforementioned method.

[0012] According to the method proposed in this disclosure, scaling factors are grouped into scaling factor blocks using a predetermined storage layout, and writing can be completed in a single burst transfer, greatly improving bandwidth utilization and transmission efficiency. During the loading phase, thanks to the storage mode of scaling factor blocks, the hardware only needs to access the base address of the scaling factor block to load the entire data block, significantly reducing the number of address seeks and instruction cycle overhead, thus achieving highly efficient data loading.

[0013] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0014] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram of the structure of a general-purpose graphics processing unit (GPU) is shown. Figure 2 A schematic diagram of storing two-dimensional array data is shown; Figure 3 A schematic flowchart of a data processing method according to an embodiment of the present disclosure is shown; Figure 4 A schematic flowchart of a data loading method according to an embodiment of the present disclosure is shown; Figure 5 A schematic diagram of a predetermined storage layout according to an embodiment of the present disclosure is shown; Figure 6 A schematic diagram of an apparatus for a data processing method according to an embodiment of the present disclosure is shown; Figure 7 A schematic diagram of an apparatus for a data loading method according to an embodiment of the present disclosure is shown; Figure 8 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0018] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0019] The data processing and data loading methods disclosed herein are related to GPUs. Figure 1 A schematic diagram of the structure of a general-purpose graphics processing unit (GPU) is shown.

[0020] refer to Figure 1 The GPU module 100 shown includes one or more streaming multiprocessors (e.g., streaming multiprocessor 102), and one or more scheduling and distribution modules (e.g., thread block scheduling and distribution module 101) for distributing computing tasks to the streaming multiprocessors. The streaming multiprocessor 102 may internally include one or more execution cores (e.g., the one or more cores shown), and a scheduling and distribution module (e.g., thread bundle scheduling and distribution module 103) for issuing instructions to the cores. The cores may internally include one or more arithmetic logic units, floating-point arithmetic units, dedicated tensor computation units (e.g., multiplication units for performing matrix multiplication and accumulation operations), etc.

[0021] GPUs can store and process tensor data. Generally, tensor data can be viewed as a generalization of multidimensional arrays: zero-dimensional tensors are scalars, one-dimensional tensors are vectors, two-dimensional tensors are matrices, and higher-dimensional tensors correspond to higher-dimensional features. The GPU module 100 can have multiple storage levels to store tensor data or related parameters. For example, tensor data can be stored in video memory 109 (e.g., HBM) and moved to the vicinity of one or more cores via on-chip storage such as L2 cache 108 and L1 cache / shared memory 107 for computation; intermediate results during computation can be temporarily stored in register 106. In some examples, the movement of tensor data from video memory 109 to L1 cache / shared memory 107 can be asynchronously performed by a tensor memory accelerator (TMA) or a data movement unit with similar functionality, with the tensor data's dimensions (e.g., shape, span, etc.) as parameters.

[0022] For common matrix operations in GPU computing (such as Generalized Matrix Multiplication (GEMM) or Matrix Multiplication and Accumulation (MACC) calculations), tensor data is typically represented and used in computation in the form of the aforementioned multidimensional array. After an instruction is issued to one or more cores, the computation unit within the core can multiply and accumulate the corresponding elements of the tensor data (such as quantized data in Mxf8 format) to complete the corresponding operation.

[0023] Figure 1 This is merely one exemplary GPU architecture; however, those skilled in the art will recognize other GPU architectures, and this disclosure is not intended to limit them.

[0024] Figure 2 This diagram illustrates a method for storing data in a two-dimensional array. Logically, this two-dimensional array is defined by both row and column dimensions. Figure 2 Each grid cell represents a basic storage unit.

[0025] In some examples, Figure 2 Each grid cell in the tensor can represent a scaling factor of the stored data. As mentioned earlier, tensor data can be quantized to reduce its storage overhead. Generally, quantization maps high-precision tensor data (e.g., FP32, BF16) to low-precision, scaled tensor data (e.g., Mxf8). This quantization process can produce... Figure 2 The scaling factor is shown.

[0026] Below, for reference Figure 3 A schematic flowchart 300 is provided to describe a data processing method according to an embodiment of the present disclosure.

[0027] At S301, a data processing instruction is obtained, which is used to perform quantization operations on the tensor data in memory.

[0028] At S302, obtain the range of data rows and the range of data columns associated with the tensor data.

[0029] For tensor data, perform the following processing on each row in the data row range in sequence: At S303, a request is made to memory for the current row of tensor data within the range of data columns.

[0030] At S304, in response to obtaining the tensor data row returned from memory, the scaling factor row corresponding to the quantization operation of the tensor data row is obtained, and the obtained scaling factor row is stored in the register at the row position corresponding to the current row.

[0031] At S305, in response to determining that the scaling factors stored in the register form a scaling factor block with a predetermined storage layout, the scaling factor block is sent to the buffer in a burst transfer. In such an embodiment, one burst transfer corresponds to one base address, so that the scaling factors in the scaling factor block are stored contiguously in the buffer.

[0032] According to embodiments of this disclosure, previously multiple, discrete storage operations can be transformed into fewer, high-bandwidth burst transmissions. Compared to traditional linear, fragmented storage models, embodiments of this disclosure construct scaling factor blocks conforming to a predetermined storage layout in registers, and store scaling factors in the form of scaling factor blocks in a buffer. According to embodiments of this disclosure, the throughput of scaling factors sent from the computation core to the buffer is significantly improved through the predetermined storage layout and burst transmission mechanism. Moreover, since scaling factors are stored contiguously in the buffer in units of scaling factor blocks, subsequent computation units only need to access a single base address to read the entire scaling factor block when loading it, greatly simplifying address calculation complexity and optimizing downstream loading efficiency.

[0033] Return to reference Figure 2 The diagram illustrates that scaling factors can be stored in a buffer. In related technologies, buffers employ linear storage. However, this approach suffers from access efficiency bottlenecks when processing multidimensional data.

[0034] For example, inside the buffer, Figure 2 The scaling factors shown follow a row-major linear arrangement rule, meaning that all scaling factors in the first row are arranged consecutively in physical address space until the end of the row, followed immediately by the starting data of the next row. If a computational unit in the core needs to read 64 rows of scaling factors, because the physical addresses are linearly expanded in row-major order, the starting addresses of each row are not contiguous in the buffer. Therefore, to obtain these 64 rows of scaling factors, the GPU needs to initiate an addressing request for each row separately. This means that the hardware needs to perform 64 read operations to access 64 addresses, which will consume 64 execution cycles.

[0035] Furthermore, during the quantization process, the scaling factors generated by the quantization operation are also arranged in a row-major linear manner. The number of scaling factors in each row is very small relative to the number of tensor data in its corresponding row (for example, the tensor data obtained after quantization of a row is about 1KB, while the scaling factor corresponding to that row is only a few bytes). As a result, the actual effective scaling factor data carried in each bus transmission accounts for a very small percentage of the total bus bandwidth, leading to a significant waste of bandwidth.

[0036] According to one or more embodiments of this disclosure, a predetermined storage layout for scaling factors is proposed, and a data processing method and a data loading method corresponding to this predetermined storage layout are provided. This enables scaling factors to be sent to a buffer in a predetermined storage layout, forming scaling factor blocks. Multiple scaling factors in a scaling factor block are then sent to the buffer in a single burst transmission, thereby fully utilizing data transmission bandwidth and improving the transmission efficiency of scaling factors. Furthermore, when loading scaling factors from the buffer, since scaling factors are stored in the buffer in units of scaling factor blocks, only the base address of the scaling factor block needs to be accessed to load all scaling factors in that block, thereby improving data loading efficiency and reducing instruction cycle overhead.

[0037] In some examples, the data processing method of this disclosure can be executed by a Tensor Memory Accelerator (TMA) or a data transport unit with similar functionality. The TMA is capable of asynchronously transporting data between storage levels according to instructions with tensor data dimensions as parameters, and is therefore suitable as the execution entity of the data processing method described in this disclosure. In such examples, the aforementioned data processing instructions can be TMA instructions sent to the TMA, which then parses and splits the instructions, and performs subsequent operations such as data requests, scaling factor generation and transmission.

[0038] In some examples, the data processing instructions can be TMA instructions, which instruct the TMA to perform quantization operations and generate corresponding scaling factors during data transfer.

[0039] In some examples, when the data processing instruction is a TMA instruction, the splitting of the data processing instruction based on a predetermined column granularity can be accomplished by the TMA scheduling logic, so that each split data processing instruction corresponds to one TMA data transfer operation. This splitting method ensures that the scaling factor generated by each TMA data transfer precisely matches the assembly requirements of the predetermined storage layout.

[0040] The above steps will be described in detail below with reference to some non-limiting embodiments.

[0041] At S301, a data processing instruction is obtained, which is used to perform quantization operations on the tensor data in memory.

[0042] Data processing instructions can serve as trigger signals for quantization pipelines, defining and initiating quantization operations on tensor data stored in memory.

[0043] In some examples, tensor data can include row and column dimensions, to Figure 2 The illustrated method is stored in memory.

[0044] In some examples, the tensor data can be stored in memory in a linear fashion. For instance, the tensor data can be stored in memory in a row-major linear fashion.

[0045] In some examples, data processing instructions can specify the starting address in memory of the tensor data to be quantized, as well as information such as the row and column dimensions of the tensor data. For instance, a data processing instruction might specify the following information for the tensor data to be loaded: starting address 0x7FFF0000, row dimension 1024, column dimension 2048. This means that the first tensor data to be loaded is located at address 0x7FFF0000, containing a total of 1024 rows, each with 2048 elements. If the tensor data is stored in memory in a row-major linear fashion, the column dimension determines the step size for jumping from one row to the next.

[0046] In some examples, data processing instructions can indicate relevant parameters for the quantization operation, such as the quantization format, which may include the original data format of the tensor data (e.g., high-precision fp32 or bf16) and the target quantization data format (e.g., low-precision Mxf8, Mxf4, S4).

[0047] In some examples, data processing instructions may specify a predetermined column granularity for splitting the instruction. Such exemplary embodiments will be described in detail below.

[0048] At S302, obtain the range of data rows and the range of data columns associated with the tensor data.

[0049] In some examples, the data row range can include all row dimensions of the tensor data to be loaded, such as from the first row to the last row. In other examples, the data row range can include a portion of the row dimensions of the tensor data to be loaded, such as from the first row to a specific row or from a specific row to the last row.

[0050] In some examples, the data column range can include all column dimensions of the tensor data to be loaded, such as from the first column to the last column. In other examples, the data column range can include a portion of the column dimensions of the tensor data to be loaded, such as from the first column to a specific column or from a specific column to the last column.

[0051] Furthermore, in the embodiment described above where scaling factors are sent to the buffer as scaling factor blocks in a predetermined storage layout, the granularity of splitting data processing instructions can be adaptively adjusted in conjunction with this predetermined storage layout during the tensor data loading phase.

[0052] For example, prior to performing S302, this disclosure may further include: splitting the data processing instructions at least once along the column dimension based on a predetermined column granularity to obtain at least two split data processing instructions. In such an embodiment, the data column range is obtained based on the column dimension range indicated by the split data processing instructions.

[0053] Generally, in related technologies, data processing instructions are typically processed in a linear, continuous manner. For example, tensor data is stored in memory in a row-major linear fashion. Taking the NHW linear model as an example, N represents the batch size fetched in one calculation, H represents the height of the matrix, and W represents the width of the matrix. In this model, the first row of tensor data corresponds to memory address 0, the second row to memory address 1, and the third row to memory address 2. If the data processing instruction instructs to process these three rows of tensor data, then each row needs to be extracted sequentially. For example, the three addresses need to be sent to memory using three sub-instructions, each sub-instruction extracting an entire row of tensor data.

[0054] In this row-by-row processing method, the scaling factors corresponding to the loaded tensor data are arranged in a linear and continuous manner. For example, the first row of tensor data corresponds to the first row of scaling factors, and the second row of tensor data corresponds to the second row of scaling factors. Under this processing method, the resulting scaling factors are arranged in a shape that extends continuously along the row direction. However, in typical matrix operation scenarios, the multiplication unit often needs to consume the scaling factors corresponding to multiple rows of data simultaneously in a single operation. This means that loading scaling factors requires accessing multiple rows of scaling factors at the same time. In the linear and continuous arrangement, the scaling factors of different rows extend sequentially in physical space, and the starting addresses of each row's scaling factors are separate. To obtain multiple rows of scaling factors, the hardware needs to initiate addressing requests for each row separately. Therefore, the number of execution cycles required to access the scaling factors of multiple rows increases linearly with the number of rows, resulting in low bandwidth utilization. Similarly, in the quantization stage, since the scaling factors generated from multiple rows of tensor data also need to be written sequentially to separate addresses, bandwidth is also difficult to fully utilize.

[0055] In contrast, according to one or more embodiments of this disclosure, scaling factors are no longer stored in a linear, row-oriented manner. Instead, data processing instructions are split along the column dimension. This allows the scaling factors of multiple rows simultaneously consumed by the multiplication unit in a single operation to be compactly organized and stored contiguously within the same scaling factor block in physical space, with scaling factors within the block sharing a single base address. Under this splitting method, the scaling factors of multiple rows participating in the operation are compactly organized and stored contiguously within the same scaling factor block in physical space, with scaling factors within the block sharing a single base address. This transforms the original access process requiring multiple addressings across multiple rows into a continuous read of a single base address: only one base address access is needed to obtain the scaling factors of all rows within the entire block, significantly reducing the number of addressing attempts and execution cycles. Correspondingly, in the writing phase, the multi-row scaling factors generated by the quantization operation can also be written to the buffer in blocks, using a single burst transfer, significantly improving the utilization of bus bandwidth. Therefore, significant bandwidth and cycle efficiency gains can be achieved whether sending scaling factors to or loading scaling factors from the buffer.

[0056] Continuing with the exemplary embodiment of column-based splitting described earlier, tensor data can be split along its column dimensions according to a predetermined column granularity. This splitting transforms the original data processing instructions into multiple split data processing instructions. Each split data processing instruction (or sub-instruction) corresponds to a specific range of split data columns in the tensor data to be loaded. In such an example, the range of data columns in S302 can be obtained based on the column dimension range indicated by the split data processing instructions. It is understood that in other examples where splitting is not required, the range of data columns in S302 can correspond to the column range of the original data instructions.

[0057] Continuing with the previous example of column-based splitting, as a specific non-limiting embodiment, tensor data of [1024, 2048] is to be loaded from memory, where the row dimension is 1024 rows and the column dimension is 2048 columns. Based on a predetermined column granularity of 512 columns, the data processing instructions are split into four sub-instructions along the column dimension. In such an embodiment, sub-instruction 1 indicates a column dimension range of 0 to 511, sub-instruction 2 indicates a column dimension range of 512 to 1023, sub-instruction 3 indicates a column dimension range of 1024 to 1535, and sub-instruction 4 indicates a column dimension range of 1536 to 2047. At this point, the data column range obtained by each sub-instruction during the execution phase of S302 can be determined based on these split column dimension ranges.

[0058] In this example, by loading tensor data using this splitting method, each split data processing instruction can independently assemble one or more complete scaling factor blocks in its register during its own execution, without waiting for scaling factors generated by subsequent instructions to be pieced together across instructions. Once assembled, the entire scaling factor block can be immediately written to the buffer via a single burst transfer. Accordingly, the space in the register used to temporarily store scaling factors can be released in a timely manner, the bandwidth of a single burst transfer can be fully utilized, and each scaling factor block written to the buffer is a complete block in physical space composed of consecutive scaling factors generated by the same subinstruction, facilitating subsequent block-level loading based on base addresses.

[0059] For example, instructing a data processing instruction to split at least once in the column dimension to obtain at least two split data processing instructions may include: obtaining the first column coordinates of the first split data processing instruction based on the column coordinates indicated by the data processing instruction; and obtaining the second column coordinates of the second split data processing instruction by offsetting the first column coordinates by a first preset offset.

[0060] Based on this example, the column coordinates indicated by the split data processing instruction can be located by recursively adding a first preset offset to the column coordinates indicated by the data processing instruction. Specifically, splitting the data processing instruction at least once along the column dimension yields two adjacent split data processing instructions along the column dimension. The column coordinates of these two adjacent instructions differ by the first preset offset. Furthermore, the first column coordinate of the first split data processing instruction along the column dimension can be obtained based on the column coordinates of the data processing instruction, and the second column coordinate of the adjacent second split data processing instruction can be obtained based on the first column coordinates offset by the first preset offset.

[0061] For example, a data processing instruction can specify the following parameters for tensor data: starting column coordinate C_start = 0, column dimension W = 1024, and first preset offset: Offset = 512. First, the data processing instruction is split along the column dimension based on a predetermined column granularity, resulting in a first split data processing instruction and a second split data processing instruction, which are adjacent along the column dimension. The first column coordinate of the first split data processing instruction is based on the starting column coordinate, i.e., first column coordinate = 0. The data column range loaded by the first split data processing instruction is from column 0 to column 511. The second column coordinate of the second split data processing instruction is obtained by offsetting the first column coordinate by the first preset offset, i.e., second column coordinate = 0 + 512 = 512. The data column range loaded by the second split data processing instruction is from column 512 to column 1023.

[0062] Furthermore, in some examples, the first preset offset corresponds to the number of tensor data points corresponding to a predetermined column granularity. For example, if the predetermined column granularity is 512 columns, then the first preset offset corresponds to 512 columns of tensor data. In such embodiments, the column ranges of tensor data covered by adjacent first and second split data processing instructions in the column dimension can be interconnected. The first split data processing instruction covers the column range corresponding to the predetermined column granularity starting from its first column coordinate; to ensure that the column range of the second split data processing instruction begins precisely at the end of the coverage of the first split data processing instruction, the second column coordinate should be offset relative to the first column coordinate by the number of tensor data points corresponding to the predetermined column granularity. Thus, the first preset offset corresponds to the number of tensor data points corresponding to the predetermined column granularity. By determining the column coordinates of the split data processing instructions in this way, address calculation can be simplified; simply adding the preset offset to the column coordinates corresponding to the current instruction allows for rapid location of the column coordinates corresponding to the next instruction.

[0063] In some examples, the predetermined column granularity can be determined based on the quantization format indicated by the data processing instructions. Specifically, different quantization formats correspond to different scaling ratios (i.e., how many tensor data points share a scaling factor), and different predetermined column granularities can be configured for different quantization formats. Since the maximum capacity of a single bus burst is fixed (e.g., 256 bytes), the total scaling factor generated by each sub-instruction during its execution can be set not to exceed this maximum. Simultaneously, to ensure that each sub-instruction can independently trigger a complete burst (without waiting for scaling factors from subsequent instructions to be pieced together), the number of scaling factors generated by each sub-instruction can be set equal to the number of scaling factors corresponding to the burst's capacity. In such examples, the minimum column granularity satisfying the above conditions can be determined based on the scaling ratio specified by the quantization format (how many tensor data points correspond to one scaling factor).

[0064] For example, for the Mxf8 format (where the scaling factor is 1 for every 32 tensor data points), if a scaling factor is used as follows... Figure 5The illustrated 64-row * 4-scaling-factor predefined storage layout, since 4SF corresponds to 128 tensor data in Mxf8 format, that is, the range of quantized tensor data columns is 128 * 8 = 1k col, which corresponds exactly to one burst transfer. Therefore, the range of tensor data columns before quantization corresponding to this 1k col can be used as the first column granularity. For example, if the tensor data before quantization is in bf16 format, then the range of tensor data columns before quantization corresponding to 1k col is 2k col, so the first column granularity (e.g., the column range corresponding to 2k tensor data) can be used as the predefined column granularity. Similarly, for Mxf4 format (where the scaling ratio is 1 scaling factor for every 32 tensor data, but the tensor data bit width is smaller), the second column granularity (e.g., the column range corresponding to 4k tensor data) can be used as the predefined column granularity.

[0065] In some examples, the predetermined column granularity can also be determined based on the area or number of registers on the chip. For example, if the chip has four registers, for the Mxf8 format, the predetermined column granularity can be determined as 8k col (2k col * 4) based on the aforementioned 2k col. Furthermore, in this case, the corresponding scaling factor rows will be stored in different registers. This exemplary embodiment will be described in detail below.

[0066] By configuring the predefined column granularity in relation to the quantization format and / or register area, it is possible to ensure that each subinstruction independently and completely produces a scaling factor block under different quantization formats. This ensures that the hardware benefits of a single base address and a single burst are always available during the write and load phases, without degrading to cross-instruction piecing or scattered writing due to changes in quantization format.

[0067] The following continues with details of an example example based on this disclosure. As previously described, for tensor data, steps S303 and S304 can be performed sequentially on each row in the data row range.

[0068] For example, in step S303, a tensor data row corresponding to the current row of the tensor data within the data column range is requested from memory. Since the data column range of the tensor data has already been obtained, a row of tensor data (i.e., a tensor data row) corresponding to each row within that data column range can be requested from memory sequentially. For example, if the data column range of the tensor data is from column 0 to column 511, and the data row range is from row 0 to row 63, for each row, the corresponding tensor data row can be requested from memory. For example, the tensor data row of row 0 within columns 0 to 511 can be requested, the tensor data row of row 1 within columns 0 to 511 can be requested, and so on.

[0069] At S304, in response to obtaining the tensor data row returned from memory, the scaling factor row corresponding to the quantization operation of the tensor data row is obtained, and the obtained scaling factor row is stored in the register at the row position corresponding to the current row.

[0070] According to embodiments of this disclosure, after obtaining the tensor data row returned from memory, the scaling factor row corresponding to its quantization operation can be obtained based on the row coordinate information of the current tensor data row.

[0071] In some examples, the scaling factor rows corresponding to the tensor data rows can be calculated. In other examples, the scaling factor rows corresponding to the tensor data rows can be retrieved from memory. Further, in embodiments of this disclosure, the obtained scaling factor rows are not arbitrarily written to the register, but rather stored in the register at a specific row position corresponding to the row coordinates of the current row. This storage method allows for the construction, row by row, of a scaling factor set logically consistent with the row dimension layout of the original tensor data within the register, according to the row coordinates of the tensor data. This provides a basis for subsequently assembling the scaling factors in the register into scaling factor blocks using a predetermined storage layout.

[0072] As a non-limiting concrete example, a data processing request may include requesting a row of tensor data from row 10, columns 0 through 511, from memory. In such an example, in response to returning the tensor data row from memory, a scaling factor row corresponding to the quantization operation of that tensor data row can be calculated. If the scaling ratio corresponding to the quantization format indicated by the data processing instruction is 32:1, i.e., one scaling factor for every 32 tensor data, then 16 scaling factors constituting the scaling factor row are calculated. The obtained scaling factor row is then stored in a register at the row location corresponding to row 10.

[0073] At S305, in response to determining that the scaling factors stored in the register form a scaling factor block with a predetermined storage layout, the scaling factor block is sent to the buffer in a burst transfer. In such an embodiment, one burst transfer corresponds to one base address, so that the scaling factors in the scaling factor block are stored contiguously in the buffer.

[0074] This step utilizes registers as temporary storage points for scaling factors, assembling the scattered scaling factor rows according to a predetermined storage layout, and then pushing them to the buffer all at once using a burst transfer, an efficient storage method. Compared to a scheme that immediately stores a scaling factor row to the buffer after obtaining it, resulting in a large number of scattered write operations, according to the embodiments of this disclosure, these scaling factors can be temporarily stored in registers until the number of scaling factors accumulated in the registers reaches the data volume that a single burst transfer of the bus can carry. At this point, a burst transfer operation is triggered to write the entire block to the buffer. As a result, the proportion of effective scaling factor data carried in each bus transfer is significantly increased, and the transmission bandwidth can be fully utilized. At the same time, since the size of the predetermined storage layout corresponds to the carrying capacity of a single burst transfer, the scaling factors temporarily stored in the registers meet the burst transfer triggering condition once the predetermined storage layout is satisfied, without the need for additional judgment.

[0075] Burst transmission refers to storing a continuous sequence of data at a single address without needing to store data separately for each address. For example, based on the base address of a burst transmission, a block of scaling factors is sent to a buffer all at once, and the scaling factors within the same block are stored consecutively in the buffer due to this burst transmission method.

[0076] A pre-defined storage layout can be a storage layout with a predetermined number of rows and columns. Figure 5 A schematic diagram of a predetermined storage layout according to an embodiment of the present disclosure is shown. Figure 5 The predetermined storage layout has a 64-row, 4-column scaling factor. According to one or more embodiments of this disclosure, the size of this predetermined storage layout can correspond to the amount of data carried in a single burst transfer. For example, when each scaling factor is 1B, the size of a scaling factor block is 64×4=256B, corresponding to the amount of data carried in a single burst transfer. In some examples, the scaling factor can be... Figure 5 A predetermined storage layout is used to form a scaling factor block, which is then sent to a buffer storage. The buffer may contain one or more such scaling factor blocks.

[0077] During quantization, each row of tensor data (e.g., on the order of approximately 1KB) produces a scaling factor that is only a few bytes long (e.g., 4 bytes). If written line by line in a traditional linear manner, the effective scaling factor data accounts for less than 2% of the bus bandwidth in each bus transfer, resulting in a significant waste of bus bandwidth. By assembling scaling factors into blocks with a predetermined storage layout and storing them in a buffer, bandwidth utilization can be maximized. Moreover, this method simplifies subsequent scaling factor loading. Since the scaling factors are stored in blocks in the buffer, subsequent access only requires accessing the base address of the scaling factor block to access all scaling factors within the entire block, achieving efficient loading and access.

[0078] In some examples, the number of rows in the predefined storage layout corresponds to the number of rows in the matrix output by the multiplication unit, and the number of columns in the predefined storage layout corresponds to the channel width of the multiplication unit.

[0079] For example, in matrix operations, a multiplication unit (e.g., one or more matrix multipliers) outputs a batch of matrix results each time it performs a calculation. This batch of results contains a certain number of matrix rows, and each row of the matrix requires a corresponding scaling factor during the dequantization stage. For example, if the multiplication unit outputs 16 rows of matrix results each time, then 16 corresponding scaling factors need to be retrieved simultaneously during the dequantization stage. In an exemplary embodiment of this disclosure, the number of rows in a predetermined storage layout can be set to be equal to the number of matrix rows output by the multiplication unit at one time. Continuing with the previous example, the predetermined storage layout can be set to have 16 rows. Thus, all the scaling factors required by the multiplication unit for one output are concentrated within the same scaling factor block. The dequantization stage only needs to access a single base address of this scaling factor block to retrieve all the required scaling factors, without having to load across multiple scaling factor blocks, thus avoiding the additional cycle overhead of cross-block address indexing.

[0080] For example, the number of columns in the predetermined storage layout can correspond to the channel width of the multiplication unit. For example, in matrix operations, the number of tensor data that the multiplication unit can process at one time can be determined based on the channel width of the multiplication unit. For example, a channel width of 128 means that the multiplication unit can process the multiplication and addition of 128 tensor data points from two sets of matrices in one clock cycle. For example, the channel width of the multiplication unit indicates the number of tensor data points that the multiplication unit can process in one clock cycle. For example, a channel width of 128 means that the multiplication unit can process the multiplication and addition of 128 tensor data points from two sets of matrices in one clock cycle. Accordingly, the number of scaling factors corresponding to the tensor data processed by the multiplication unit in one clock cycle is also determined based on the scaling ratio indicated by the data processing instructions (i.e., how many tensor data points share a scaling factor). For example, with a channel width of 128 and a scaling ratio of 32 tensor data points sharing a scaling factor, the tensor data processed by the multiplication unit in one clock cycle corresponds to 4 scaling factors. In embodiments of this disclosure, the number of columns in the predetermined storage layout can be set to this number. Continuing the example above, each row of the predetermined storage layout can be set to 4 scaling factors. This ensures that all scaling factors in each row of the scaling factor block correspond exactly to the tensor data processed by the multiplication unit in one clock cycle, thus preventing any unused scaling factors or missing scaling factors in tensor data within a single clock cycle.

[0081] According to embodiments of this disclosure, a predetermined storage layout of the scaling factor block can be matched with the computational power of the multiplication unit to improve data throughput efficiency.

[0082] In some examples, the number of scaling factors in a scaling factor row can be based on a predetermined quantization format and a range of data columns; and wherein storing the obtained scaling factor row into the register at the row position corresponding to the current row includes: in response to determining that the number of scaling factors in the scaling factor row is a multiple of the number of columns in a predetermined storage layout based on a predetermined quantization format, dividing the scaling factor row into multiple groups of scaling factors corresponding to the multiple, and storing each of the multiple groups of scaling factors into a different register.

[0083] In embodiments of this disclosure, the number of scaling factors in a scaling factor row can be based on a predetermined quantization format and data column range. For example, the scaling factor row corresponds to the quantization operation of a tensor data row, and the data column range (or, in other words, the predetermined column granularity of the corresponding data processing instruction) determines the total number of tensor data in the tensor data row. Dividing the two gives the total number of scaling factors produced after quantizing the tensor data row. For instance, the data processing instruction might indicate that the quantization format is Mxf8, and that 32 tensor data share one scaling factor. The tensor data row returned from memory is row 0, and the data column range is columns 0 to 255. That is, based on the data column range, this row 0 contains 256 tensor data. Further, based on the predetermined quantization format, it can be determined that these 256 tensor data correspond to 8 scaling factors, therefore the scaling factor row contains 8 scaling factors.

[0084] In embodiments of this disclosure, at S305, the scaling factors in the register are grouped into scaling factor blocks according to a predetermined storage layout. Correspondingly, when the number of scaling factors in a scaling factor row is a multiple of the number of columns in the predetermined storage layout, instead of writing the entire row of scaling factors into a single register at once, the scaling factor row can be divided into multiple groups of scaling factors corresponding to the multiple, and stored in different registers respectively. This allows for the pre-completion of synchronous construction of multiple scaling factor blocks on the register side, thereby supporting the parallel burst transmission of multiple subsequent scaling factor blocks. For example, if the number of columns in the predetermined storage layout is 4, meaning each row in the predetermined storage layout has 4 scaling factors, then for the 8 scaling factors in the aforementioned scaling factor row, the scaling factor row can be divided into 2 groups and stored in 2 registers respectively, serving as the 0th row of the predetermined storage layout for two scaling factor blocks.

[0085] By temporarily storing scaling factor rows containing multiple scaling factors in multiple registers in the aforementioned distributed storage manner, the register area on the chip can be fully utilized, achieving a highly compact physical layout of storage resources while meeting the requirements of parallel transmission.

[0086] In some aspects, the data processing method 300 may further include: receiving a data loading instruction for loading scaling factors from a buffer. In such an embodiment, the scaling factors are stored in the buffer in units of scaling factor blocks; in response to the data loading instruction, performing the following operations: determining the number of scaling factors to be loaded per line in an execution cycle based on the data format and dot product operation size associated with the data loading instruction; and, based on the number of scaling factors to be loaded, repeatedly accessing the base address of the same scaling factor block in one or more consecutive execution cycles to load scaling factors within the same scaling factor block.

[0087] In some aspects, the data processing method 300 may further include: a data loading instruction further used to instruct an inverse quantization operation on the loaded scaling factors, and wherein determining the number of scaling factors to be loaded per row in an execution cycle based on the data format and dot product scale associated with the data loading instruction includes: determining the amount of tensor data to be used in a single dot product operation cycle based on the data format and dot product scale; and determining the number of scaling factors to be loaded per row in an execution cycle based on the determined amount of data to be used.

[0088] In some aspects, the data processing method 300 may further include: after the scaling factors within the same scaling factor block have been loaded, determining the base address of the next scaling factor block to be loaded by applying a second preset offset to the base address.

[0089] In some aspects, the data processing method 300, by applying a second preset offset to the base address to determine the base address of the next scaling factor block to be loaded, may include: the second preset offset corresponding to the size of the scaling factor block.

[0090] The above embodiments will be Figure 4 The details are elaborated in the text, and will not be repeated here.

[0091] Below, for reference Figure 4 Here is a schematic flowchart describing a data loading method 400 according to an embodiment of the present disclosure.

[0092] At S401, a data load instruction is received for loading scaling factors from the buffer. In such an embodiment, the scaling factors are generated via a quantization operation performed on the tensor data, and the scaling factors are stored in the buffer in units of scaling factor blocks. In such an embodiment, scaling factors within the same scaling factor block are stored contiguously in the buffer, and each scaling factor in each scaling factor block corresponds to a base address.

[0093] At S402, the number of scaling factors to be loaded for each row in an execution cycle is determined based on the data format and dot product operation scale associated with the data loading instruction.

[0094] At S403, based on the number of scaling factors to be loaded, the base address of the same scaling factor block is repeatedly accessed in one or more consecutive execution cycles to load the scaling factors within the same scaling factor block.

[0095] Since scaling factors are stored in the buffer as scaling factor blocks with a predetermined storage layout, when loading scaling factors from the buffer, it is only necessary to access the base address of the scaling factor block to load all scaling factors in the scaling factor block, thereby improving data loading efficiency and reducing instruction cycle overhead.

[0096] The above steps will be explained in detail below.

[0097] At S401, a data load instruction is received for loading scaling factors from the buffer. In such an embodiment, the scaling factors are generated via a quantization operation performed on the tensor data, and the scaling factors are stored in the buffer in units of scaling factor blocks. In such an embodiment, scaling factors within the same scaling factor block are stored contiguously in the buffer, and each scaling factor in each scaling factor block corresponds to a base address.

[0098] like Figure 3 As described in S305, the data processing method 300 includes sending scaling factor blocks of a predetermined storage layout to a buffer. Therefore, correspondingly, in the buffer, scaling factors are stored in units of scaling factor blocks, and the scaling factors within each scaling factor block are stored contiguously, with each scaling factor in each block corresponding to a base address. Upon receiving a data load instruction, these scaling factors stored in the form of scaling factor blocks can be loaded from the buffer.

[0099] In some examples, the data loading instruction can specify the data format to be used in matrix operations, such as Mxf8, Mxf4, etc.

[0100] In some examples, the data load instruction can specify one or more base addresses of the scaling factor block to be loaded from the buffer.

[0101] In some examples, data loading instructions can be used to instruct the loading scaling factor to be dequantized.

[0102] In some examples, the data loading instruction can indicate the size of the dot product operation.

[0103] In some examples, the data load instruction can specify the amount of data to be dequantized; for example, the data load instruction can specify the amount of data as col = 256B.

[0104] At S402, the number of scaling factors to be loaded for each row in an execution cycle is determined based on the data format and dot product operation scale associated with the data loading instruction.

[0105] For example, this step may include: determining the amount of tensor data to be used in a single dot product operation cycle based on the data format and the scale of the dot product operation; and determining the number of scaling factors to be loaded for each row in an execution cycle based on the determined amount of data to be used.

[0106] In such an example, a data load instruction could instruct a dequantization operation based on the loaded scaling factor. For instance, a data load instruction instructs a dequantization operation based on the loaded scaling factor. Figure 5 The scaling factor in column 0 shown in the figure also performs dequantization on 64*32 Mxf8 format tensor data.

[0107] For example, the dequantization operation (which includes loading scaling factors) is driven by a dot product operation (or matrix multiplication). The dot product scale indicates the multiplication unit's ability to process dot products within a single execution cycle. Since the dequantization operation is driven by the dot product, this capability limits how much tensor data needs to be dequantized within a single execution cycle. In some examples, the dot product cycle corresponds to the execution cycle. In other examples, the loading scaling factor operation can be decoupled from the dot product operation in the pipeline, and the execution cycle and the dot product cycle can be different. In some examples, the dot product scale can be dp32, dp64, etc. Furthermore, the data format limits the bit width of each tensor data. In some examples, the data format can be the aforementioned Mxf8, Mxf4, etc. If the data loading instruction indicates the amount of data to be dequantized, the data format limits the number of tensor data corresponding to that amount of data.

[0108] For example, if the chip's bus width is 32 bytes, with a data format of Mxf8 and a dot product operation size of dp32, 32 bytes of tensor data can be used in one execution cycle, corresponding to 32 Mxf8 format tensors. Therefore, the amount of tensor data to be used in a single dot product operation cycle is determined to be 32. As another example, if the chip's bus width is 32 bytes, with a data format of Mxf4 and a dot product operation size of dp32, 64 Mxf4 format tensors are consumed in one execution cycle. Therefore, the amount of tensor data to be used in a single dot product operation cycle is determined to be 64.

[0109] Further, in embodiments of this disclosure, the number of rows in the scaling factor block is set to match or be equal to the number of matrix output rows covered by the multiplication units that can operate in parallel. In such an example, the scaling factors corresponding to all matrix output rows consumed in parallel by the multiplication units within one execution cycle can be distributed across all rows of the scaling factor block. Based on this, and further combined with the aforementioned amount of tensor data to be used within a single execution cycle, the number of scaling factors to be loaded in each row of the scaling factor block within that execution cycle can be determined. For example, in the case of a data format of Mxf8 and a dot product operation size of dp32, based on the determination that the amount of tensor data to be used within a single dot product operation cycle is 32, then one scaling factor needs to be loaded within one execution cycle. Based on this, if the chip has four 16*32 multiplication units that can compute dp32 in parallel, and the scaling factor block has 64 rows (e.g., ...), ... Figure 5 (As shown in the layout), each row of the scaling factor block can load one scaling factor within one execution cycle, meaning all 64 rows of the scaling factor block are loaded simultaneously, with one scaling factor loaded per row. That is, cycle 0 can load 64 rows of SF0, cycle 1 can load 64 rows of SF1, and so on. For example, with a data format of Mxf4 and a dot product operation size of dp32, based on the determined tensor data volume to be used within a single dot product operation cycle being 64, then two scaling factors need to be loaded within one execution cycle. Furthermore, if the chip has four 16*32 multiplication units that can compute dp32 in parallel, and the scaling factor block has 64 rows (e.g., ... Figure 5 (As shown in the layout), each row of the scaling factor block can load two scaling factors within one execution cycle, meaning that all 64 rows of the scaling factor block are loaded simultaneously, with each row loading two scaling factors. In other words, cycle0 can load 64 rows of SF0 and SF1, cycle1 can load 64 rows of SF2 and SF3, and so on.

[0110] Those skilled in the art can configure various parameters such as data format, dot product operation scale, and computing power of multiplication unit, as well as hardware configuration, according to actual conditions, and this disclosure does not impose any restrictions on these.

[0111] At S403, based on the number of scaling factors to be loaded, the base address of the same scaling factor block is repeatedly accessed in one or more consecutive execution cycles to load the scaling factors within the same scaling factor block.

[0112] As described above, according to some exemplary embodiments of this disclosure, the number of scaling factors to be loaded in each execution cycle is determined at S402, and the scaling factors are stored in the buffer in units of scaling factor blocks, with scaling factors in the same scaling factor block having a base address. Thus, this base address can be repeatedly accessed in one or more consecutive execution cycles to load scaling factors in the same scaling factor block.

[0113] For example, it has been determined that two scaling factors will be loaded per line in each execution cycle, and the base address of the scaling factor block is addr, and the predetermined storage layout of the scaling factor block is 64 rows and 4 columns (e.g. Figure 5 (See the layout shown). Therefore, in the first execution cycle, SF0 and SF1 of each row can be loaded from addr; in the second execution cycle, SF2 and SF3 of each row can be loaded from addr.

[0114] In some examples, after the scaling factors within the same scaling factor block have been loaded, the base address of the next scaling factor block to be loaded can be determined by applying a preset offset to the base address.

[0115] For example, the scaling factor blocks in the buffer can be arranged linearly, such as storing the first scaling factor block first, then the second, and so on, in a row-major direction. In such an example, the base address of the next scaling factor block to be loaded can be determined by applying a preset offset to the base address of the scaling factor block.

[0116] In some examples, the preset offset may correspond to the size of the scaling factor block. Alternatively, the size of the scaling factor block may be associated with the predetermined storage layout described above. For example, the predetermined storage layout has... Figure 5 The layout shown indicates that the size of the scaling factor block can be 256 scaling factors. If a scaling factor has 1B, then the size of the scaling factor block is 256B. Furthermore, the preset offset can also be a corresponding 256B. For example, if it has been determined that two scaling factors will be loaded per line in each execution cycle, and the base address of the scaling factor block is addr, the predetermined storage layout of the scaling factor block is... Figure 5The example shows a 64-row, 4-column scaling factor, with each scaling factor being 1B. In this example, in the first execution cycle, SF0 and SF1 of each row can be loaded from addr; in the second execution cycle, SF2 and SF3 of each row can be loaded from addr. In the third execution cycle, a preset offset of 256B can be applied to the base address addr to determine the base address addr+256B of the next scaling factor block to be loaded. Continuing with this example, SF4 and SF5 of each row can be loaded based on the address addr+256B, and so on.

[0117] By loading scaling factor blocks from the buffer as described above, this disclosure significantly improves throughput, reducing address indexing, which originally required numerous cycles, to a mere few cycles of continuous loading operations. For GPUs, this efficient scaling factor loading mechanism enables multiplication units to process dequantized data at extremely high frequencies, even full load, ensuring that multiplication units do not idle while waiting for scaling factors to load, thereby greatly improving the execution efficiency of computationally intensive tasks such as deep learning.

[0118] Furthermore, on the write side, this disclosure significantly increases the proportion of effective scaling factor data in each bus transfer compared to related techniques that use fragmented writing, by assembling multiple scaling factors into scaling factor blocks and then sending them to the buffer in a single burst transmission, thus enabling full utilization of the bus bandwidth during the write phase.

[0119] Below, for reference Figure 5 A schematic diagram of a predetermined storage layout 500 according to an embodiment of the present disclosure is provided. The predetermined storage layout 500 has a scaling factor of 64 rows and 4 columns. Figure 5 The details have already been explained above and will not be repeated here. Figure 5 Taking the 64-row, 4-column scaling factor block layout shown as an example, in related technologies, it takes about 64 execution cycles to obtain 64 rows of scaling factors for address indexing. However, according to the embodiments of this disclosure, it is only necessary to initiate an access to a single base address of the scaling factor block once to retrieve the entire block of 64 rows of scaling factors in the same execution cycle, reducing the execution cycle from the order of O(N) to O(1).

[0120] Below, for reference Figure 6 This is a schematic diagram illustrating an apparatus 600 for a data processing method according to an embodiment of the present disclosure.

[0121] The apparatus 600 may include an instruction receiving unit 601, a data range determining unit 602, a data processing unit 603, a data requesting unit 604, a scaling factor obtaining unit 605, and a sending unit 606. In addition to these units, the apparatus 600 may also include other components; however, since these components are not relevant to the content of this embodiment, their illustrations and descriptions are omitted here. Furthermore, the specific details of the operations performed by the apparatus 600 according to this embodiment are consistent with those described above. Figure 3 The details described are the same, so repeated descriptions of the same details are omitted here to avoid repetition.

[0122] The instruction receiving unit 601 can be configured to obtain data processing instructions, which are used to perform quantization operations on tensor data in memory.

[0123] The data range determination unit 602 can be configured to obtain the data row range and data column range associated with the tensor data.

[0124] The data processing unit 603 may include a data request unit 604 and a scaling factor acquisition unit 605. It can be configured to, for tensor data, sequentially perform the following processing on each row in the data row range via the data request unit 604 and the scaling factor acquisition unit 605. The data request unit 604 can be configured to request the tensor data row of the current row within the data column range from memory. The scaling factor acquisition unit 605 can be configured to, in response to obtaining the tensor data row returned from memory, obtain the scaling factor row corresponding to the quantization operation of the tensor data row, and store the obtained scaling factor row in the register at the row position corresponding to the current row.

[0125] The sending unit 606 can be configured to send the scaling factor block to the buffer in a burst transmission manner in response to determining the scaling factor stored in the register, which is arranged in a predetermined storage layout. In such an embodiment, one burst transmission corresponds to one base address, so that the scaling factors in the scaling factor block are stored contiguously in the buffer.

[0126] Below, for reference Figure 7 This is a schematic diagram illustrating an apparatus 700 for a data loading method according to an embodiment of the present disclosure.

[0127] The apparatus 700 may include an instruction receiving unit 701, a data processing unit 702, a scaling factor determining unit 703, and a scaling factor accessing unit 704. In addition to these units, the apparatus 700 may also include other components; however, since these components are not relevant to the content of this embodiment, their illustrations and descriptions are omitted here. Furthermore, the specific details of the operations performed by the apparatus 700 according to this embodiment are consistent with those described above. Figure 4 The details described are the same, so repeated descriptions of the same details are omitted here to avoid repetition.

[0128] The instruction receiving unit 701 can be configured to receive a data loading instruction for loading scaling factors from a buffer. In such an embodiment, the scaling factors are generated via a quantization operation performed on the tensor data, and the scaling factors are stored in the buffer in units of scaling factor blocks using a predetermined storage layout. In such an embodiment, scaling factors within the same scaling factor block are stored contiguously in the buffer, and each scaling factor in each scaling factor block corresponds to a base address.

[0129] The data processing unit 702 may include a scaling factor determination unit 703 and a scaling factor access unit 704, and may be configured to perform the following operations in response to a data loading instruction through the scaling factor determination unit 703 and the scaling factor access unit 704.

[0130] The scaling factor determination unit 703 can be configured to determine the number of scaling factors to be loaded for each row in an execution cycle based on the data format and dot product operation scale associated with the data loading instruction.

[0131] The scaling factor access unit 704 can be configured to repeatedly access the base address of the same scaling factor block in one or more consecutive execution cycles, based on the number of scaling factors to be loaded, in order to load the scaling factors within the same scaling factor block.

[0132] The following describes a computer device according to the present invention, wherein a computer program is stored in a memory on the computer device, and when the computer program is executed by a processor, the processor performs the aforementioned data processing method and / or data loading method.

[0133] Below, for reference Figure 8 A schematic diagram illustrating a computer-readable storage medium according to embodiments of the present disclosure is provided.

[0134] For example, such as Figure 8 As shown, the storage medium 800 can be a non-transitory computer-readable storage medium on which one or more computer programs 801 can be stored non-transitory. For example, when the computer program 801 is executed by a processor, it can perform one or more steps of the data loading method or one or more steps of the data processing method described above.

[0135] For example, storage media may include memory cards in smartphones, storage components in tablet computers, hard drives in personal computers, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), and portable compact disc read-only memory (CD). ROM, flash memory, or any combination of the above storage media, or other applicable storage media.

[0136] The following describes a computer program product according to the present invention, which includes a computer program that, when executed by a processor, causes the processor to perform the aforementioned data processing method and / or data loading method. For example, the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD). ROM, USB storage, flash memory, etc. One or more computer programs can be stored on a computer-readable storage medium, and the processor can run these programs to perform its various functions. Various application programs and data can also be stored on the storage medium.

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0139] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0140] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0141] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A data processing method, characterized in that, The method includes: Obtain data processing instructions, which are used to perform quantization operations on tensor data in memory; Obtain the data row range and data column range associated with the tensor data; For the tensor data, perform the following processing sequentially on each row in the data row range: Request the current row of the tensor data from the memory within the range of the data columns; and In response to obtaining the tensor data row returned from the memory, a scaling factor row corresponding to the quantization operation of the tensor data row is obtained, and the obtained scaling factor row is stored in the register at the row position corresponding to the current row; and In response to determining that the scaling factors stored in the register form a scaling factor block with a predetermined storage layout, the scaling factor block is sent to a buffer in a burst transfer manner, wherein the burst transfer corresponds to a base address, so that the scaling factors in the scaling factor block are stored contiguously in the buffer.

2. The method according to claim 1, characterized in that, Also includes: Based on a predetermined column granularity, the column dimension range indicated by the data processing instruction is split at least once to obtain at least two split data processing instructions, wherein the data column range is obtained based on the column dimension range indicated by the split data processing instructions.

3. The method according to claim 2, characterized in that, Splitting the column dimension range indicated by the data processing instruction at least once to obtain at least two split data processing instructions includes: The first column coordinates of the first split data processing instruction are obtained based on the column coordinates indicated by the data processing instruction; and The second column coordinates of the second split data processing instruction are obtained by offsetting the first column coordinates by a first preset offset.

4. The method according to claim 3, characterized in that, The first preset offset corresponds to the number of tensor data corresponding to the predetermined column granularity.

5. The method according to any one of claims 1-4, characterized in that, The tensor data is stored in memory in a row-major linear manner.

6. The method according to any one of claims 1-4, characterized in that, The number of rows in the predetermined storage layout corresponds to the number of rows in the matrix output by the multiplication unit, and the number of columns in the predetermined storage layout corresponds to the channel width of the multiplication unit.

7. The method according to claim 6, characterized in that, The number of scaling factors in the scaling factor row is based on a predetermined quantization format and the range of the data columns; and Storing the obtained scaling factor row into the register at the row position corresponding to the current row includes: In response to determining, based on the predetermined quantization format, that the number of scaling factors in the scaling factor row is a multiple of the number of columns in the predetermined storage layout, the scaling factor row is divided into multiple groups of scaling factors corresponding to the multiple; Each of the multiple scaling factors is stored in a different register.

8. The method according to any one of claims 1-4, characterized in that, Also includes: A data loading instruction is received for loading scaling factors from the buffer, wherein the scaling factors are stored in the buffer in units of scaling factor blocks using the predetermined storage layout; as well as Based on the data loading instruction, perform the following operations: Based on the data format and dot product operation scale associated with the data loading instruction, determine the number of scaling factors to be loaded for each row within an execution cycle; and Based on the number of scaling factors to be loaded, the base address of the same scaling factor block is repeatedly accessed in one or more consecutive execution cycles to load the scaling factors within the same scaling factor block.

9. The method according to claim 8, characterized in that, Based on the data format and dot product operation scale associated with the data loading instruction, the number of scaling factors to be loaded for each row within an execution cycle is determined, including: Based on the data format and the scale of the dot product operation, determine the amount of tensor data to be used in a single dot product operation cycle; Based on the determined tensor data volume, determine the number of scaling factors to be loaded for each row in the said execution cycle.

10. The method according to claim 8, characterized in that, Also includes: After the scaling factors within the same scaling factor block are loaded, the base address of the next scaling factor block to be loaded is determined by applying a second preset offset to the base address.

11. The method according to claim 10, characterized in that, The second preset offset corresponds to the size of the scaling factor block.

12. A data loading method, characterized in that, The method includes: A data loading instruction is received for loading a scaling factor from a buffer, wherein the scaling factor is generated via a quantization operation performed on tensor data, and the scaling factor is stored in the buffer in units of scaling factor blocks using a predetermined storage layout, wherein the scaling factors in the same scaling factor block are stored contiguously in the buffer, and the scaling factor in each scaling factor block corresponds to a base address. Based on the data loading instruction, perform the following operations: Based on the data format and dot product operation scale associated with the data loading instruction, determine the number of scaling factors to be loaded for each row within an execution cycle; and Based on the number of scaling factors to be loaded, the base address of the same scaling factor block is repeatedly accessed in one or more consecutive execution cycles to load the scaling factors within the same scaling factor block.

13. The method according to claim 12, characterized in that, Based on the data format and dot product operation scale associated with the data loading instruction, the number of scaling factors to be loaded for each row within an execution cycle is determined, including: Based on the data format and the scale of the dot product operation, determine the amount of tensor data to be used in a single dot product operation cycle; Based on the determined tensor data volume, determine the number of scaling factors to be loaded for each row in the execution cycle.

14. The method according to claim 12 or 13, characterized in that, Also includes: After the scaling factors within the same scaling factor block have been loaded, the base address of the next scaling factor block to be loaded is determined by applying a preset offset to the base address.

15. The method according to claim 14, characterized in that, The preset offset corresponds to the size of the scaling factor block.

16. An apparatus for a data processing method, characterized in that, The device includes: The instruction receiving unit is configured to receive a data processing instruction, which is used to perform a quantization operation on tensor data in memory. The data range determination unit is configured to obtain the data row range and data column range associated with the tensor data; The data processing unit includes a data request unit and a scaling factor acquisition unit. The data processing unit is configured to perform the following processing sequentially on each row of the data row range for the tensor data: The data request unit requests tensor data rows from the memory for the current row of the tensor data within the range of the data column. In response to obtaining the tensor data row returned from the memory, the scaling factor acquisition unit obtains the scaling factor row corresponding to the quantization operation of the tensor data row, and stores the obtained scaling factor row into the register at the row position corresponding to the current row; and The sending unit is configured to, in response to determining that the scaling factors stored in the register are arranged in a predetermined storage layout to form a scaling factor block, send the scaling factor block to a buffer in a burst transmission manner, wherein the burst transmission corresponds to a base address, so that the scaling factors in the scaling factor block are stored contiguously in the buffer.

17. An apparatus for a data loading method, characterized in that, The device includes: The instruction receiving unit is configured to receive a data loading instruction for loading a scaling factor from a buffer, wherein the scaling factor is generated by a quantization operation performed on tensor data, and the scaling factor is stored in the buffer in units of scaling factor blocks using a predetermined storage layout, wherein the scaling factors in the same scaling factor block are stored contiguously in the buffer, and each scaling factor in the scaling factor block corresponds to a base address. The data processing unit includes a scaling factor determination unit and a scaling factor access unit, and the data processing unit is configured to perform the following operations in response to the data loading instruction: The scaling factor determination unit determines the number of scaling factors to be loaded for each row within an execution cycle, based on the data format and dot product operation scale associated with the data loading instruction; and The scaling factor access unit repeatedly accesses the base address of the same scaling factor block within one or more consecutive execution cycles, based on the number of scaling factors to be loaded, in order to load the scaling factors within the same scaling factor block.

18. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-15.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1-15.

20. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1-15.

Images