Data processing method, electronic device, storage medium and program product

Abstract

The embodiment of the invention relates to a data processing method, electronic equipment, a storage medium and a program product. The method comprises the following steps: receiving a first instruction for writing target data in an on-chip memory of a computing unit into an off-chip memory; generating metadata for the target data through a metadata generation component, and storing the metadata in an on-chip memory, the metadata describing one or more statistical values corresponding to at least part of data in the target data; writing the target data into an off-chip memory through a direct memory access component; obtaining metadata from the on-chip memory in response to receiving a second instruction to perform a target process for the target data using the computing unit, the target process being related to a target statistical value of the one or more statistical values; and executing target processing for the target data based on the target statistical value in the metadata by using the calculation unit. In this way, the data processing efficiency can be improved.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, an electronic device, a computer-readable storage medium, and a computer program product for data processing. BACKGROUND

[0002] With the rapid development of computer technology, the application scenarios of generative models are becoming more and more extensive, for example, in information retrieval, dialogue assistants, code generation, and the like. The user experience of generative models is largely dominated by response latency. In the inference process of a generative model, the prefill stage of the Transformer architecture determines the "first token generation time", which in turn directly determines the absolute time of the response of the generative model. SUMMARY

[0003] In a first aspect of the present disclosure, a method for data processing is provided. The method comprises: receiving a first instruction to write target data in an on-chip memory of a computing unit to an off-chip memory; generating, via a metadata generation component, metadata for the target data and storing the metadata in the on-chip memory, the metadata generation component being disposed between the on-chip memory and a direct memory access component, the metadata describing one or more statistical values corresponding to at least part of the target data; writing, via the direct memory access component, the target data to the off-chip memory; in response to receiving a second instruction to execute a target process for the target data using the computing unit, obtaining the metadata from the on-chip memory, the target process being related to a target statistical value in the one or more statistical values; and executing, using the computing unit, the target process for the target data based on the target statistical value in the metadata.

[0004] In a second aspect of the present disclosure, an electronic device is provided. The electronic device comprises at least one computing unit; and at least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit. The instructions, when executed by the at least one computing unit, cause the electronic device to perform the method of the first aspect.

[0005] In a third aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon computer-executable instructions that are executable by a computing unit to implement the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0007] The scheme provided by the present disclosure sets a metadata generation component between the on-chip memory and the direct memory access component, for generating metadata for target data in the on-chip memory in the process of sending the target data to the direct memory access component, and returning the metadata to the on-chip memory; then the calculation unit can perform a target processing process for the target data based on the target statistical value in the metadata, thereby reducing the secondary acquisition and calculation process of the target data, saving data processing resources, and improving data processing efficiency.

[0008] It should be understood that the content described in this part is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail some embodiments thereof with reference to the annexed drawings in which: Figure 1 A schematic diagram showing an example environment in which embodiments according to the present disclosure can be implemented is shown; Figure 2 A flowchart showing an example process for data processing according to some embodiments of the present disclosure is shown; Figure 3 A schematic structural block diagram of a metadata generation component according to some embodiments of the present disclosure is shown; Figure 4 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0010] Embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, the embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are merely for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0011] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document and any type of embodiment can be included under any section / subsection. Furthermore, embodiments described in any section / subsection can be combined with any other embodiment described in the same section / subsection and / or in a different section / subsection in any manner.

[0012] In the description of embodiments of the disclosure, the term "includes" and its similar terms are understood to be open-ended, i.e., "includes but is not limited to". The term "based on" is understood to be "based, at least in part, on". The term "one embodiment" or "the embodiment" is understood to be "at least one embodiment". The term "some embodiments" is understood to be "at least some embodiments". The following can also include other explicit and implicit definitions. The terms "first", "second", etc. can refer to different or the same objects. The following can also include other explicit and implicit definitions.

[0013] In some embodiments of the disclosure, data of users, acquisition and / or use of data, etc. can be involved. These aspects comply with the corresponding laws and regulations and relevant provisions. In some embodiments of the disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user is aware of and confirms. Accordingly, when implementing various embodiments of the disclosure, the type of data or information that can be involved, the scope of use, the use scenario, etc. should be notified to the user and authorized by the user in a proper manner according to the relevant laws and regulations. The specific notification and / or authorization manner can vary according to the actual situation and application scenario, and the scope of the disclosure is not limited in this respect.

[0014] In the specification and embodiments of the disclosure, if personal information processing is involved, it will be processed on the premise of legality (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and only within the prescribed or agreed range. Users refuse to process personal information other than the necessary information required for basic functions, which will not affect the user's use of basic functions.

[0015] As mentioned above, the user experience of generative models is largely dominated by response latency. In the inference process of generative models, the prefill stage of the Transformer architecture determines the "first token generation time", which directly determines the absolute time of the generative model response.

[0016] In the prefill data flow of the Transformer architecture, intermediate activations between different data layers are often counted and normalized in the last dimension in units of rows or channels. For example, the per-row maximum subtraction required by Softmax, the mean or square mean required by RMSNorm or other Norm, the min / max or quantile interval required by quantization / dequantization, etc.

[0017] In existing implementations, these statistics described above usually need to be calculated again by downstream operators after the upstream operators have written data to external storage; if the downstream operators also need to perform quantization processing on the data (for example, quantization after subtracting the maximum value, deriving Scale according to min / max, etc.), at least one complete read traversal overhead will be generated, which will cause significant bandwidth and delay burden in data processing under long sequence (large seqlen) and large channel dimension (large hidden_dim). Although kernel fusion and online algorithms can reduce the temporary round trip of data within the operator, when the intermediate results of data processing need to be landed on external storage and multiple subsequent data processing (such as matrix multiplication, Softmax, RMSNorm, quantization / dequantization, activation clipping, KV compression, etc.) share the same or related statistics, repeated read and repeated calculation are still inevitable.

[0018] Embodiments of the present disclosure provide a scheme for data processing. The scheme includes: receiving a first instruction to write target data in an on-chip memory of a computing unit to an off-chip memory; generating, via a metadata generation component, metadata for the target data and storing the metadata in the on-chip memory, the metadata generation component being disposed between the on-chip memory and a direct memory access component, the metadata describing one or more statistical values corresponding to at least part of the target data; writing, via the direct memory access component, the target data to the off-chip memory; in response to receiving a second instruction to perform a target processing on the target data using the computing unit, obtaining the metadata from the on-chip memory, the target processing being related to a target statistical value in the one or more statistical values; and performing, using the computing unit, the target processing on the target data based on the target statistical value in the metadata.

[0019] In this way, embodiments of the present disclosure generate metadata for the target data by disposing a metadata generation component between the on-chip memory and the direct memory access component, for generating metadata for the target data during sending the target data in the on-chip memory to the direct memory access component and returning the metadata to the on-chip memory; then the computing unit can perform the target processing on the target data based on the target statistical value in the metadata, thereby reducing the secondary acquisition and calculation process of the target data, saving data processing resources, and improving data processing efficiency.

[0020] The scheme extracts the required statistics in a "streaming online" manner in the path of writing the on-chip memory to the off-chip memory, and provides the metadata to the on-chip memory for the computing unit to perform subsequent data processing, thereby effectively eliminating or reducing additional data read traversal, reducing unnecessary consumption of data processing, and improving data processing efficiency.

[0021] Various example implementations of the subject innovation are described in further detail below in connection with the accompanying drawings.

[0022] Example Environment Figure 1 A schematic diagram illustrating an example environment 100 in which embodiments of the present disclosure can be implemented is shown. As shown, the example environment 100 can include a computing unit 110. The computing unit 110 can include a processor core 111 and an on-chip memory 112. Figure 1

[0023] In some embodiments, the computing unit 110 can be a standalone electronic device, or a software unit or hardware unit embedded in an electronic device. As an example, such an electronic device can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile handset, a tablet computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a palmtop computer, a portable gaming terminal, a VR / AR device, a Personal Communication System (PCS) device, a personal navigation device, a Personal Digital Assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination thereof, including accessories and peripherals of such devices, or any combination thereof. In some embodiments, the electronic device can also support any type of interface to a user (such as "wearable" circuitry, etc.).

[0024] The computing unit 110 can store at least one of data to be processed, intermediate data during processing, and result data after processing, in the on-chip memory 112.

[0025] Referring to Figure 1 As shown, the example environment 100 can also include an off-chip memory 120, and the computing unit 110 can also include a direct memory access component 114. The direct memory access component 114 can be configured to obtain data stored in the on-chip memory 112 and store the obtained data in the off-chip memory 120.

[0026] As an example, the off-chip memory 120 can be an external random access memory, such as an external dynamic random access memory (DRAM), including but not limited to a DDR series memory, a LPDDR memory, a GDDR memory, or a HBM memory, or some combination thereof, without limitation.

[0027] ​In the scheme of the present disclosure, between the on-chip memory 112 and the direct memory access component 114, a metadata generation component 113 is arranged. As an example, the target data in the on-chip memory 112 needs to pass through the metadata generation component 113 in the process of flowing to the direct memory access component 114, and the metadata generation component 113 generates the metadata corresponding to the target data in the process of flowing of the target data. For example, the metadata generation component 113 can perform at least one statistical processing on the target data to generate the corresponding metadata.

[0028] The metadata generation component 113 can store the generated metadata into the on-chip memory 112 for the computing unit 110 to perform subsequent data calculation and other processing processes.

[0029] In some scenarios, after generating the metadata, the metadata generation component 113 can also send the generated metadata to the direct memory access component 114, and store the target data and the metadata corresponding to the target data into the off-chip memory 120 via the direct memory access component 114, so that other computing units or data processing devices can directly obtain the corresponding content in the metadata to perform corresponding data processing processes.

[0030] As an example, the computing unit 110 can support at least part of the data processing process of the generative model or the language model. For example, for a generative model with a multi-layer Transformer cascade structure, each layer can be deployed with a Softmax operator and an RMSNorm operator. Both the Softmax operator and the RMSNorm operator can utilize the metadata generated by the metadata generation component 113 in the computing unit 110 of the present disclosure to perform subsequent calculation or processing processes, so as to improve the data processing efficiency.

[0031] In some implementations, the matrix multiplication processing process in the generative model or the language model can also utilize the metadata generated by the metadata generation component 113 to realize data quantization acceleration and improve the data processing efficiency.

[0032] It should be understood that the structure and function of each element in the example environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0033] Some example embodiments of the present disclosure will be described below with continuous reference to the accompanying drawings.

[0034] Example process: Figure 2 A flowchart of an example process 200 of processing data according to some embodiments of the present disclosure is shown. The example process 200 can be implemented at the computing unit 110. The example process 200 will be described below with reference to Figure 1 .

[0035] As shown in FIG. 1, the computing unit 110 receives a first instruction of writing target data in the on-chip memory 112 of the computing unit 110 to the off-chip memory 120. Figure 2 As shown in FIG. 1, the computing unit 110 receives a first instruction of writing target data in the on-chip memory 112 of the computing unit 110 to the off-chip memory 120.

[0036] As an example, the computing unit 110 can utilize the processor core 111 or other instruction processing component to receive the first instruction of writing the target data in the on-chip memory 112 to the off-chip memory 120.

[0037] In some embodiments, the target data can be intermediate data generated by the language model in a pre-filling stage.

[0038] In block 220, the computing unit 110 generates metadata for the target data via the metadata generation component 113, and stores the metadata in the on-chip memory 112, the metadata generation component 113 being arranged between the on-chip memory 112 and the direct memory access component 114, the metadata describing one or more statistical values corresponding to at least part of the data in the target data.

[0039] In the present disclosure, the data in the on-chip memory 112 no longer directly flows to the direct memory access component 114, but the metadata generation component 113 is arranged between the on-chip memory 112 and the direct memory access component 114, and the data in the on-chip memory 112 needs to pass through the metadata generation component 113 before flowing to the direct memory access component 114. In this process, the metadata generation component 113 processes the target data flowing to the direct memory access component 114 to determine one or more statistical values corresponding to at least part of the data in the target data, and generates metadata corresponding to the target data according to the statistical values, and then writes the generated metadata into the on-chip memory 112.

[0040] In some embodiments, the metadata can indicate at least one of the following statistical values: maximum value, minimum value, cumulative value, sum of squares, mean value, variance, etc.

[0041] As an example, the metadata can also include data such as a histogram or percentile approximation value corresponding to the target data.

[0042] In this way, the embodiments of the present disclosure can perform multiple types of statistics on at least part of the data in the target data to determine corresponding multiple statistical values, thereby effectively enriching the data content of the metadata, enriching the types of processing that can be performed on the target data based on the metadata, and further improving the efficiency of subsequent processing using the metadata.

[0043] In some embodiments, the type of metadata is determined based on at least one subsequent processing to be performed on the target data.

[0044] As an example, such post-processing procedures can include, but are not limited to, any one of the following: finding a maximum value or a minimum value, finding a variance, a mean value, a square mean value, or a square sum, performing quantization or dequantization on at least part of the target data, etc.

[0045] In this way, the embodiments of the present disclosure can generate metadata of corresponding types based on the types of subsequent processing that the target data can perform, thereby ensuring the processing efficiency of subsequent processing, and avoiding the waste of computing and storage resources caused by generating unnecessary types of metadata.

[0046] In some implementation scenarios of the present disclosure, the metadata generated by the metadata generation component 113 can include not only the whole-dimension metadata of the target data in whole-dimension statistics, but also at least one item of local metadata corresponding to at least one group of local data after the target data is divided by at least one granularity.

[0047] In some embodiments, generating the metadata for the target data via the metadata generation component 113 between the on-chip memory 112 and the direct memory access component 114 includes: dividing, by the computing unit 110, the target data into a plurality of data slices using the metadata generation component 113; and determining the metadata for the target data based on the plurality of data slices.

[0048] In the process of writing the target data in the on-chip memory 112 to the off-chip memory 120, the computing unit 110 divides the target data using the metadata generation component 113 in the process of flowing through the metadata generation component 113 to obtain a plurality of data slices of the same data size, and then performs parallel statistics on the plurality of data slices to generate local metadata corresponding to each data slice, and can also generate whole-dimension metadata corresponding to the target data, thereby obtaining metadata associated with the target data.

[0049] In this way, the present solution can not only generate corresponding whole-dimension metadata for the target data, but also generate local metadata corresponding to the data slices after division, thereby synchronously obtaining metadata corresponding to different data granularities of the target data, effectively enriching the data content of the metadata, and enriching the types of processing that can be performed on the target data based on the metadata, further improving the efficiency of subsequent processing using the metadata.

[0050] In some embodiments, the target data includes a plurality of unit data corresponding to a preset data dimension, and dividing the target data into a plurality of data slices includes: dividing, by the computing unit 110, the target data into a plurality of data slices based on the preset data dimension, each data slice corresponding to a predetermined number of groups of unit data.

[0051] As an example, the preset data dimension can include a lowest data dimension corresponding to the target data. For example, the target data includes 128 data matrices of the same size, and the lowest data dimension can be that each data matrix is independently taken as a data slice, i.e., the number of data slices obtained by division can be 128.

[0052] In some scenarios, the preset data dimension can also include any dimension higher than the lowest data dimension and lower than the highest data dimension (the whole dimension data). Continuing to take the example that the target data includes 128 data matrices of the same size, the number of data slices corresponding to the preset data dimension can also be 64, 32, 16, 8, 4, 2, etc.

[0053] In some implementations, the computing unit 110 can divide the target data according to the preset data dimension to obtain a plurality of data slices corresponding to at least one data granularity (e.g., one data is one group, two data is one group, or more data is one group, etc.). The plurality of data slices corresponding to the same data granularity have the same number of data in the corresponding group of unit data.

[0054] Taking the maximum value or the minimum value as an example, the metadata generation component 113 can construct a corresponding comparator tree by arranging a plurality of levels of comparators. The data dimension and the data granularity corresponding to the plurality of comparators in each level are the same, i.e., the plurality of comparators have the same number of data in the corresponding group of unit data, and the plurality of metadata (e.g., maximum value or minimum value) output by the plurality of comparators correspond to the same data granularity in the target data.

[0055] The metadata generation component 113 utilizes the structure of the comparator tree, and can output at least one maximum value or minimum value in each level, thereby obtaining a plurality of metadata of different levels and the whole dimension metadata corresponding to the target data, effectively enriching the data content of the metadata, and enriching the processing type and the processing granularity based on the metadata to perform processing on the target data, further improving the efficiency of subsequent processing using the metadata, and effectively expanding the applicable scenarios of the metadata.

[0056] In this way, the embodiments of the present disclosure can divide the target data based on the preset data dimension, thereby effectively ensuring that the plurality of data slices of the same data dimension have the same number of data, and further ensuring the accuracy and effectiveness of the metadata of the same data dimension.

[0057] In some implementations, the computing unit 110 can mark the division marker corresponding to the plurality of data slices after dividing the target data into a plurality of data slices. For example, such a division marker can include a group descriptor and a boundary marker, etc.

[0058] As an example, the computing unit 110 can write the partition mark in association with the metadata when writing the metadata into the on-chip memory 112 by the metadata generation component 113, to construct a corresponding data index.

[0059] In some embodiments, determining the metadata for the target data based on the plurality of data slices includes that the computing unit 110 generates the metadata corresponding to a plurality of granularities based on the plurality of data slices, the granularity indicating a number of data slices used to generate a statistical value.

[0060] Continuing with the example of the comparator tree, the computing unit 110 can utilize the comparator tree in the metadata generation component 113 to first partition the target data at a minimum granularity, and then accumulate layer by layer, to achieve a plurality of different granularities of the target data, i.e., to obtain a plurality of data slices corresponding to a plurality of different granularities respectively. The metadata generation component 113 can obtain the metadata corresponding to each granularity synchronously in the process of utilizing the comparator tree to achieve a plurality of different granularities of the partition, so as to not only achieve the generation of metadata corresponding to a plurality of different granularities for the target data, but also ensure the efficiency of the generation of the metadata.

[0061] In this way, the embodiments of the present disclosure can generate metadata corresponding to a plurality of different granularities for the target data, effectively enrich the data content of the metadata, and enrich the types of processing that can be performed on the target data based on the metadata, further improving the efficiency of subsequent processing performed by utilizing the metadata.

[0062] Figure 3 A schematic structural block diagram of the metadata generation component 113 in some embodiments of the present disclosure is shown. The following exemplary description of the generation process of the metadata is made with reference to Figure 1 and Figure 3 .

[0063] As shown in Figure 3 , the metadata generation component 113 includes a multiplexer 310. As an example, the target data from the on-chip memory 112 is written into the off-chip memory 120 via the multiplexer 310 in a multi-path parallel input manner.

[0064] In the scheme of the present disclosure, the metadata generation component 113 can not only write the target data into the off-chip memory 120 via the multiplexer 310, but also generate the metadata corresponding to the target data. For example, the metadata generation component 113 can also generate the metadata corresponding to at least one granularity of the target data in the process of the flow of the target data, and such metadata can include at least one of the following statistical values: maximum value, minimum value, cumulative value, mean value, variance, etc.

[0065] In some scenarios, for example, the data scale of the target data is too large or the available storage space of the on-chip memory 112 is small, the metadata generation component 113 can also write the generated metadata to the off-chip memory 120.

[0066] As an example, the metadata generation component 113 can also include a packager 320. The metadata generation component 113 sends the target data from the on-chip memory 112 to the packager 320 via the multiplexer 310, and also sends the generated metadata to the packager 320, which packs the metadata together with the target data for writing to the off-chip memory 120. For example, the packager 320 can write the packed data to the off-chip memory 120 via the direct memory access component 114.

[0067] In some scenarios, for example, the metadata includes a maximum value or a minimum value, the metadata generation component 113 can include a comparator tree 330 and a first register group 331. The target data from the on-chip memory 112 is input to the comparator tree 330 in a multi-parallel manner, and a plurality of maximum values corresponding to different granularities of the target data are output. The comparator tree 330 can write the comparison results of each layer of comparators (i.e., the plurality of maximum values output by the layer) to the first register group 331 and to the on-chip memory 112.

[0068] In some scenarios, the comparator tree 330 can only compare a fixed number of inputs at a time. For example, if the comparator tree 330 supports a maximum of 128 inputs, it can only calculate the maximum value or the minimum value of 128 inputs at a time. If the target data exceeds 128 inputs, the comparator tree 330 needs to perform multiple calculation processes. In this case, the comparator tree 330 can store the results corresponding to each group of 128 inputs in the first register group 331, and then perform the next group of 128 inputs. In this way, the final calculation result is obtained by taking the results of each group of 128 inputs as the input of the comparator tree 330 again, and the metadata corresponding to different granularities of the target data is generated.

[0069] The present disclosure aims to protect the deployment and application of the metadata generation component 113 in the computing unit 110, and does not aim to emphasize the specific implementation structure of the comparator tree 330. In actual applications, the specific implementation structure of the comparator tree 330 can refer to related technologies, which will not be described here.

[0070] Taking the pre-padding operation process in the Transformer architecture model as an example, the Softmax operator mainly performs a process of converting a set of real numbers (usually referred to as logits) into a probability distribution, including: first performing an exponential operation on the input data, and then normalizing the result to make the sum equal to 1, thereby forming an effective probability distribution. In actual applications, in order to ensure numerical stability, the maximum value is usually subtracted from the input data before the exponential operation and normalization.

[0071] In combination with the scheme of the present disclosure, in a scenario where it is necessary to perform processing on target data by using a Softmax operator, the calculation unit 110 can use the metadata generation component 113 to count the maximum value in the target data as at least part of the data content in the metadata corresponding to the target data in the process of writing the target data to the off-chip memory 120, and write it back to the on-chip memory 112. Then the Softmax operator can obtain the corresponding maximum value from the on-chip memory 112 without the need to obtain the complete target data and perform the process of counting the maximum value, thereby effectively reducing the reading consumption of the target data and improving the processing efficiency of the Softmax operator.

[0072] The following takes the data structure corresponding to the target data of the Softmax operator as an example to exemplarily describe the process of counting the maximum value in the target data by the metadata generation component 113.

[0073] The data structure input and output by the Softmax operator is [batch_size, num_head, q_length, k_length], where batch_size represents the batch size of the data, num_head represents the number of attention heads, q_length represents the query sequence length, and k_length represents the key sequence length.

[0074] The metadata generation component 113 takes the last dimension [k_length] of the input data (i.e., target data) [batch_size, num_head, q_length, k_length] as the maximum value, that is, the [k_length] corresponding to each [batch_size, num_head, q_length] is taken as the minimum input unit of the comparator tree 330 (i.e., the input data of the first layer of the comparator tree 330). The maximum values corresponding to the minimum data granularity are output by the first layer of the comparator, and are stored in the first register group 331 and used as the input data of the second layer of the comparator. The maximum values corresponding to the second granularity are counted by the second layer of the comparator, and are stored in the first register group 331 and used as the input data of the third layer of the comparator. In this way, the comparator tree 330 is used to count the maximum values corresponding to different data granularities of the target data layer by layer until the maximum value corresponding to the whole dimension of the target data is output, which is stored in the first register group 331 and can also be written into the on-chip memory 112. In this process, the maximum values corresponding to the same data granularity in the comparator tree 330 can be counted in parallel.

[0075] After the metadata generation component 113 counts the maximum value of the target data, the Softmax operator can directly perform data processing based on the maximum value. For example, the Softmax operator subtracts the maximum value from all values of [k_length], and then calculates the output according to the standard formula. As an example, the calculation formula of the Softmax operator can be expressed as: ;

[0076] For the scenario in which the metadata includes the minimum value, the comparator tree 330 and the first register group 331 described above can also be used. The process of generating the minimum value by the metadata generation component 113 can refer to the process of generating the maximum value described above, which will not be described here.

[0077] In some embodiments of the present disclosure, all the maximum values and / or minimum values written into the first register group 331 can also be sent to the packer 320 in the process of being written into the on-chip memory 112, and the target data and the maximum values and / or minimum values corresponding to different granularities thereof are packed together by the packer 320 and sent to the direct memory access component 114 to be written into the off-chip memory 120.

[0078] As an example, in response to the storage space of the on-chip memory 112 being less than a first threshold, or in response to the data scale of the target data or the metadata (e.g., the plurality of maximum values and / or minimum values) being greater than a second threshold, the metadata generation component 113 can no longer write the generated metadata to the on-chip memory 112, but write the metadata (e.g., the plurality of maximum values and / or minimum values) to the off-chip memory 120 together with the target data after being packaged by the packager 320, so as to save the storage space of the on-chip memory 112.

[0079] In some embodiments, taking the metadata including the cumulative values as an example, the metadata generation component 113 can include an adder tree 340 and a second register group 341. The target data from the on-chip memory 112 is input into the adder tree 340 in a multi-way parallel manner to obtain a plurality of cumulative values corresponding to a plurality of different granularities of the target data. The adder tree 340 writes the plurality of cumulative values output by each layer of adders to the second register group 341, and can also write to the on-chip memory 112.

[0080] In some scenarios, the comparator tree 330 can only compare a fixed number of inputs at a time. Taking the comparator tree 330 supporting a maximum of 128-way input as an example, it can only count the cumulative values of 128-way input data at a time. If the target data exceeds 128-way, the adder tree 340 needs to perform multiple counting processes. In this case, the adder tree 340 can store the execution results corresponding to every 128-way input into the second register group 341, and then perform the next set of 128-way input; and so on, and finally take the execution results corresponding to each set of 128-way input as the input of the adder tree 340 again to obtain the final counting result, and generate the metadata corresponding to the target data at different granularities.

[0081] The present disclosure aims to protect the deployment and application of the metadata generation component 113 in the computing unit 110, and does not aim to emphasize the specific implementation structure of the adder tree 340. In actual applications, the specific implementation structure of the adder tree 340 can refer to related technologies, which will not be described here.

[0082] In some scenarios, taking the cumulative values including the sum of squares as an example, the metadata generation component 113 can further include a first multiplier 342. The first multiplier 342 can be used to calculate the square value of any data in the target data. The target data from the on-chip memory 112 is input into the first multiplier 342 in a multi-way parallel manner, and the square value of each data in the target data is calculated one by one. Then, the adder tree 340 is used to perform cumulative calculation on the square values corresponding to the target data at different granularities to determine a plurality of sums of squares of the target data at different granularities. The adder tree 340 can also write the plurality of sums of squares at each granularity to the second register group 341 and the on-chip memory 112.

[0083] Continuing with the example of the pre-filling operation in the Transformer architecture model, the RMSNorm operator can normalize the features of the target data using the root mean square of the target data. The data structure of the input and output of the RMSNorm operator is [batch_size, seqlen, hidden_dim], where batch_size represents the batch size, seqlen represents the sequence length, and hidden_dim represents the hidden dimension.

[0084] Taking the RMSNorm operator as an example, the process of calculating the root mean square of the last dimension [hidden_dim] of the target data [batch_size, seqlen, hidden_dim] can be expressed as: ; In the formula, is a preset value to prevent data instability caused by division by zero.

[0085] As an example, the final output result of the RMSNorm operator can be expressed as , where is a learned scaling parameter.

[0086] As can be seen, in the process of calculating the root mean square of the target data by the RMSNorm operator, the sum of squares of the target data needs to be calculated first, then the mean is calculated, and finally the root mean square value is calculated. The sum of squares of the target data can be generated by the metadata generation component 113 to reduce the data acquisition and statistical processing process of the RMSNorm operator and improve the data processing efficiency.

[0087] As an example, the metadata generation component 113 can utilize the first multiplier 342 to square the last dimension [hidden_dim] of the input data (i.e., the target data) [batch_size, seqlen, hidden_dim], that is, the first multiplier 342 calculates the corresponding square value for each [hidden_dim], and then takes the calculated square value as the input of the adder tree 340. That is, the first multiplier 342 takes the calculated square value as the minimum input unit of the adder tree 340 (i.e., the input data of the first layer adder of the adder tree 340). The multiple square sums corresponding to the minimum data granularity are output by the first layer adder and stored in the second register group 341 and taken as the input data of the second layer adder; the multiple square sums corresponding to the second granularity are counted by the second layer adder and stored in the second register group 341 and taken as the input data of the third layer adder; and so on and so forth, the adder tree 340 is utilized to count the multiple square sums corresponding to different data granularities of the target data layer by layer until the square sum corresponding to the whole dimension of the target data is finally output and stored in the second register group 341 and can also be written into the on-chip memory 112. In this process, the multiple square sums corresponding to the same data granularity in the adder tree 340 can be counted in parallel.

[0088] In some embodiments, all the accumulated values (e.g., accumulated sums, square sums, etc.) written into the second register group 341 can also be sent to the packer 320 in the process of being written into the on-chip memory 112, and the target data and the multiple accumulated values corresponding thereto of different granularities are collectively packed by the packer 320 and sent to the direct memory access component 114 to be written into the off-chip memory 120.

[0089] As an example, in response to the storage space of the on-chip memory 112 being less than a first threshold, or in response to the data scale of the target data or the metadata (e.g., the multiple accumulated values) being greater than a second threshold, the metadata generation component 113 can no longer write the generated metadata (e.g., the multiple accumulated values) into the on-chip memory 112, but write the metadata (e.g., the multiple accumulated values) and the target data collectively via the packer 320 into the off-chip memory 120 to save the storage space of the on-chip memory 112.

[0090] In some scenarios of the present disclosure, the subsequent processing process for the target data can also include a quantization or dequantization process.

[0091] For example, in a scenario where the candidate processing procedure for the target data includes a quantization procedure, the quantization procedure can include a matrix multiplication or convolution operation procedure in a model. Before quantization, the data type of the matrix multiplication or convolution operation can be FP16 or BFP16. The quantization procedure can convert the target data into low-precision data (e.g., INT8, INT4, or FP8, generally, a data type of less than or equal to 8 bits can be referred to as a low-precision data format), then perform the matrix multiplication or convolution operation using the converted low-precision data, and convert the operation result into a high-precision data format that is the same as the target data. In this process, both the conversion of the target data into low-precision data and the conversion of the operation result of the low-precision data into a high-precision data format need to use a scaling coefficient, and such a scaling coefficient is associated with the data granularity of the conversion of the target data into low-precision data.

[0092] As can be seen, for a scenario where quantization needs to be performed on target data, the metadata generation component 113 in the present solution can generate corresponding scaling coefficients in the process of writing the target data from the on-chip memory 112 to the off-chip memory 120. As an example, for target data, the metadata generation component 113 can calculate a scaling coefficient (also referred to as a quantization coefficient) corresponding to a maximum value of the entire tensor range of the target data, or calculate a scaling coefficient corresponding to a maximum value of every 128 / 64 / 32 / 16 data elements.

[0093] Thus, the present solution can effectively avoid repeated acquisition of target data for calculating a maximum value and a scaling coefficient in a quantization procedure, thereby saving data bandwidth and reducing end-to-end delay caused by excessive occupation of data bandwidth.

[0094] The metadata generation component 113 can use the comparator tree 330 to generate multiple maximum values corresponding to different granularities for the target data, and then input at least one maximum value associated with the scaling granularity of the target data to the scaling coefficient solver 351 to calculate at least one corresponding scaling coefficient.

[0095] Reference Figure 3As shown, the metadata generation component 113 can further include a scaling factor solver 351 and a second multiplier 352. The scaling factor solver 351 is configured to calculate the scaling factor corresponding to the target data and send the scaling factor to the second multiplier 352 and send the reciprocal of the scaling factor to the packer 320. The target data from the on-chip memory 112 is input to the second multiplier 352 in a multi-lane parallel manner, and then the second multiplier 352 performs multiplication operation on each data element in the target data using the scaling factor to convert the target data into a low-precision data format, and sends the calculation result (the low-precision data corresponding to the target data) to the packer 320 via the multiplexer 310. The target data, the low-precision data corresponding to the target data, and the reciprocal of the scaling factor are packed by the packer 320 and sent to the direct memory access component 114 for writing to the off-chip memory 120.

[0096] In this way, the execution subject of the matrix multiplication or convolution operation does not need to obtain the target data and calculate the corresponding scaling factor and perform data conversion when performing quantization on the target data, but can directly obtain the low-precision data corresponding to the target data and the reciprocal of the scaling factor from the off-chip memory 120 to perform the matrix multiplication or convolution operation, and convert the operation result into a data format with the same precision as the target data using the reciprocal of the scaling factor.

[0097] In the present solution, the scaling factor and the reciprocal of the scaling factor appear in pairs. The reciprocal of the scaling factor can also be sent by the scaling factor solver 351 to the second multiplier 352 to perform the first data conversion (e.g., which can be used in the dequantization process), and the scaling factor can be sent to the packer 320 to facilitate subsequent processing and then perform the second data conversion using the scaling factor to restore the data format to the same as the target data.

[0098] In some implementations, the scaling factor solver 351 can also calculate the scaling factor based on the variance of the target data first, and then calculate the scaling factor based on the variance. As an example, the adder tree 340 can obtain multiple square sums corresponding to different granularities of the target data after performing accumulation calculation of different granularities based on the square values output by the first multiplier 342. Then the scaling factor solver 351 can obtain at least one square sum of a corresponding granularity from the output of the adder tree 340 based on the scaling granularity of the target data for calculating the scaling factor and the reciprocal of the scaling factor.

[0099] In some implementations, if the storage space of the on-chip memory 112 is large enough, the second multiplier 352 can write the data converted based on the scaling factor (or the reciprocal of the scaling factor) and the reciprocal of the scaling factor (or the scaling factor) output by the scaling factor solver 351 into the on-chip memory 112 to facilitate other execution subjects to quickly obtain the corresponding data from the on-chip memory 112.

[0100] Referring back to Figure 2 As shown in block 230, the computing unit 110 writes the target data to the off-chip memory 120 via the direct memory access component 114.

[0101] By way of example, with reference to Figure 1 and Figure 3 As shown, the computing unit 110 can send the target data in the on-chip memory 112 to the direct memory access component 114 via the multiplexer 310 in the metadata generation component 113, and write the target data to the off-chip memory 120 by the direct memory access component 114.

[0102] In some embodiments, the metadata includes at least one quantization parameter for quantizing the target data, and writing the target data to the off-chip memory via the direct memory access component includes: the computing unit 110 performing a quantization process on the target data based on the at least one quantization parameter; and writing the quantized target data to the off-chip memory via the direct memory access component 114.

[0103] By way of example, the at least one quantization parameter can include a scaling factor or a scaling factor inverse corresponding to the target data.

[0104] In some scenarios, the computing unit 110 can utilize the metadata generation component 113 to perform a quantization process on the target data based on the at least one quantization parameter. With reference to Figure 3 As shown, the computing unit 110 can utilize the second multiplier 352 in the metadata generation component 113 to perform a quantization process on the target data based on the scaling factor to obtain low-precision data corresponding to the target data, or based on the scaling factor inverse to obtain high-precision data corresponding to the target data.

[0105] By way of example, in the process of writing the quantized target data to the off-chip memory 120 via the direct memory access component 114, the parameter inverse corresponding to the at least one quantization parameter can also be written to the off-chip memory 120.

[0106] Taking the scaling factor resolver 351 delivering the scaling factor to the second multiplier 352 as an example, the packer 320 in the metadata generation component 113 can pack the scaling factor inverse from the scaling factor resolver 351 and the target data quantization result output by the second multiplier 352, and write them to the off-chip memory 120 via the direct memory access component 114.

[0107] With the scaling coefficient solver 351 delivering the scaling coefficient inverse to the second multiplier 352 as an example, the packer 320 in the metadata generation component 113 can pack the scaling coefficient from the scaling coefficient solver 351 and the target data quantization result output by the second multiplier 352, and write into the off-chip memory 120 via the direct memory access component 114.

[0108] In this way, the embodiments of the present disclosure can perform quantization processing on the target data, and write the quantized target data into the off-chip memory, thereby effectively reducing the data reading and calculation consumption in the subsequent target data quantization process, and improving the subsequent data processing efficiency.

[0109] Returning to continue referring to Figure 2 As shown, at block 240, the computing unit 110 obtains the metadata from the on-chip memory 112 in response to receiving a second instruction to utilize the computing unit 110 to perform a target processing on the target data, the target processing being related to a target statistical value in one or more statistical values.

[0110] In the present solution, such a target processing can include but is not limited to: determining a probability distribution, normalization, matrix multiplication or convolution operation, etc. Such a target statistical value can include but is not limited to: maximum value, minimum value, sum of squares, mean value, variance, low-precision data and corresponding scaling coefficient inverse, high-precision data and corresponding scaling coefficient, etc.

[0111] As an example, in response to the second instruction indicating that the target processing is to determine a probability distribution, the computing unit 110 can obtain at least one maximum value corresponding to at least one granularity of the target data from the on-chip memory, such at least one maximum value being generated by the metadata generation component 113 (e.g. the comparator tree 330) and written into. In response to the second instruction indicating that the target processing is normalization, the computing unit 110 can obtain at least one sum of squares corresponding to at least one granularity of the target data from the on-chip memory, such at least one sum of squares being generated by the metadata generation component 113 (e.g. the adder tree 340) and written into. In response to the second instruction indicating that the target processing is matrix multiplication or convolution operation, the computing unit 110 can obtain the quantization result corresponding to the target data and the scaling coefficient or the scaling coefficient inverse from the on-chip memory.

[0112] In some embodiments, the on-chip memory also maintains the metadata and corresponding data index, and obtaining the metadata from the on-chip memory includes: the computing unit 110 determines at least one data index corresponding to the target data; and obtains the metadata corresponding to the at least one data index from the on-chip memory.

[0113] In the scheme, the metadata generation component 113 can synchronize the data index corresponding to the metadata to the on-chip memory 112 in the process of writing the metadata to the on-chip memory 112.

[0114] As an example, the data index is determined in the process of generating the metadata. For example, the data index can be determined by dividing the target data to obtain a plurality of data slices.

[0115] In this way, the embodiments of the present disclosure can quickly obtain the corresponding metadata based on the data index by maintaining the metadata and the corresponding data index by using the on-chip memory, and improve the metadata search and acquisition efficiency.

[0116] In some embodiments, the at least one data index indicates a batch identifier and a sequence identifier corresponding to the target data.

[0117] As an example, the batch identifier can indicate the generation batch of the target data corresponding to the metadata.

[0118] As an example, the sequence identifier can indicate sequence information of the target data, or can indicate a sequence position of the data slice corresponding to the metadata in the target data.

[0119] In this way, the embodiments of the present disclosure can quickly locate the metadata information corresponding to the target data from the data index by using the batch identifier and the sequence identifier, thereby quickly searching and acquiring the corresponding metadata, effectively improving the metadata search and acquisition efficiency, and ensuring the accuracy of the acquired metadata.

[0120] In block 250, the computing unit 110 performs target processing on the target data based on the target statistical value in the metadata.

[0121] In this way, the embodiments of the present disclosure set the metadata generation component between the on-chip memory and the direct memory access component, which is used to generate the metadata of the target data in the process of sending the target data in the on-chip memory to the direct memory access component, and return the metadata to the on-chip memory. Then, the computing unit can perform target processing on the target data based on the target statistical value in the metadata, thereby reducing the secondary acquisition and calculation process of the target data, saving data processing resources, and improving data processing efficiency.

[0122] The scheme of the present disclosure utilizes the metadata generation component set between the on-chip memory and the direct memory access component, and migrates the metadata generation process from the downstream operator side back to the data write-back process of writing the data in the on-chip memory to the off-chip memory, which can form a unified and decoupled general statistical and quantization service from specific operators, thereby effectively reducing redundant data read-back and repeated calculation.

[0123] In some embodiments, the process 200 can further include that the computing unit 110 writes the metadata into the off-chip memory 120 via the direct memory access component 114.

[0124] As an example, in response to determining that the storage space of the on-chip memory 112 is less than the first threshold, or in response to determining that the data size of the target data or the metadata is greater than the second threshold, the computing unit 110 can utilize the packer 320 in the metadata generation component 113 to pack the metadata together with the target data, and write the packed data into the off-chip memory 120 via the direct memory access component 114. In this way, the present scheme not only enables effective storage of the metadata, but also ensures the association between the metadata and the target data.

[0125] As an example, in the process of writing the metadata into the off-chip memory 120, the direct memory access component 114 can not only associate the metadata with the corresponding target data, but also construct a data index corresponding to the metadata by associating the partition markers corresponding to the plurality of data slices associated with the metadata, and write the data index into the off-chip memory 120. For example, the computing unit 110 can utilize the metadata generation component 113 to construct the corresponding data index and send it to the packer 320, which packs the data index, the metadata, and the target data together and writes them into the off-chip memory 120 via the direct memory access component 114.

[0126] In this way, the embodiments of the present disclosure can write the metadata into the off-chip memory, effectively enriching the storage path and storage method of the metadata, thereby ensuring that the metadata is effectively stored and ensuring the effectiveness of the metadata.

[0127] The following is a comprehensive reference Figure 1 and Figure 3 The data structure generated by the model in the inference process is an intermediate activation tensor of [bs, seqlen, hidden_dim], which is taken as an example to describe the implementation process provided by the present disclosure for data processing. In the target data, bs represents the batch size, seqlen represents the sequence length, and hidden_dim represents the hidden dimension.

[0128] As an example, such a compute unit 110 can be a linear projection of a certain feed-forward layer or attention output of the model. The processor core 111 in the compute unit 110 stores the target data in the on-chip memory 112 after generating the target data. In response to receiving a first instruction to write the target data to the off-chip memory 120, the compute unit 110 can read the target data out of the on-chip memory 112 and stream the target data to the direct memory access component 114 via the metadata generation component 113. As an example, the compute unit 110 can write the target data in rows to the off-chip memory 120, and such a row can be defined as a contiguous slice in the hidden_dim dimension.

[0129] In this process, the metadata generation component 113 can directly send the target data to the direct memory access component 114 using the multiplexer 310. Meanwhile, the metadata generation component 113 can generate metadata corresponding to at least one data granularity of the target data using at least one of the comparator tree 330, the first multiplier 342 and the adder tree 340, the scaling coefficient solver 351, and the second multiplier 352. The metadata generation component 113 can write the metadata back to the on-chip memory 112, or can send the metadata to the direct memory access component 114 together with the target data via the packer 320 for writing to the off-chip memory 120.

[0130] In some implementations, the metadata generation component 113 can divide the target data into a plurality of data slices according to the number of allowed input paths of the comparator tree 330 or the adder tree 340 to obtain the plurality of data slices corresponding to the data granularity; or can divide the target data into a plurality of data slices according to the number of layers configured by the comparator tree 330 or the adder tree 340 to obtain the plurality of data slices corresponding to the data granularity.

[0131] It should be noted that the metadata generation component 113 can determine the group descriptor and the boundary marker corresponding to each data slice in the target data in the process of dividing the target data into a plurality of data slices, and then the metadata generation component 113 can construct a data index corresponding to each data slice based on the group descriptor and the boundary marker, in combination with the batch marker and the sequence marker of the target data, to obtain the data index corresponding to the metadata and the target data.

[0132] In some implementations, in the default case, the statistics axes are outer-layered by each pair of indices of [bs, seqlen] and inner-layered by hidden_dim, so that the start and end of each row segment can be indicated by the length field or EOS marker carried by the metadata generation component 113. For sequences with padding, the metadata generation component 113 can carry a mask bit, which the metadata generation component 113 can ignore the masked elements or fill the padding with values that are invalid for subsequent budgeting (e.g., fill with -inf for max or softmax) in the process of generating metadata. For variable-length sequences, the metadata generation component 113 can define the grouping termination of data segments by length tags or EOS tags in the form of TLV.

[0133] In the case of performing whole-dimension statistics on the target data, the comparator tree 330 maintains the maximum and minimum values on the whole hidden_dim dimension online, the adder tree 340 accumulates the sum of the row online, and the mean and variance accumulations in the form of Welford can be maintained in parallel; in the case of dividing the target data into multiple data segments, the hidden_dim dimension is divided into multiple data segments of 32 or 64 or 128 in size by the hardware configuration of the comparator tree 330 or the adder tree 340, and the comparator tree 330 and the adder tree 340 can set programmable “taps” on the corresponding levels to output the intermediate maximum, minimum, and partial sum of each data segment on the pipeline, while continuing to aggregate upwards to form the whole-dimension statistics. Through this “intermediate node visible” structure design, the same statistics pipeline can output both the whole-dimension statistics result at the group tail and the block statistics result at the boundary of the data segment, and the two types of results can be written to the on-chip memory 112 or the off-chip memory 120 at the same time and distinguished by different GroupID or labels.

[0134] To ensure numerical stability, the comparator tree 330 in the present scheme for statistics of minimum / maximum values can adopt NaN robust rules (such as ignoring NaN or propagating according to IEEE754); the adder tree 340 in the process of statistics accumulation and mean / variance accumulation, etc. can adopt high-precision data paths (such as internal FP32 accumulation) and support Kahan / Neumaier compensation to reduce rounding errors. For scenarios that require more rich distribution information, the metadata generation component 113 can enable programmable histogram functions to perform linear or logarithmic scale binning on the data domain, and output the count of each bin at the group tail for subsequent calculation of approximate quantile.

[0135] In the scenario of quantization, the metadata generation component 113 can configure a second multiplier 352 with a programmable fixed-point packer 320 per channel (or per data slice). When the online statistics form min / max (or based on mean / std, quantile threshold) at the end of a certain group (or data slice), the metadata generation component 113 (e.g., with the scaling factor solver 351) can immediately derive the quantization parameters (e.g., scaling factors for FP8) for that group.

[0136] In the “on-the-fly quantization” mode, the target data is scaled by the second multiplier 352 per channel and encoded into INT8 / INT4 / FP8, etc. target format by the packer 320 before being written to the data area in compressed format, while the corresponding quantization parameters and statistics metadata are written to the metadata area in parallel.

[0137] In the “metadata only” mode, the metadata written back to the off-chip memory 120 is in floating-point data structure, but the group quantization parameters and statistics metadata are already available for the subsequent operators to directly read for on-demand quantization or normalization.

[0138] In both modes, the metadata generation component 113 can generate an “completion notification” at the end of a group and ensure the sequential relationship between “target data write completion” and “metadata visibility” through the doorbell or fence mechanism, so as to guarantee that the execution subject of the subsequent processing can obtain the matching metadata in a deterministic timing when reading the target data. To adapt to the high-concurrency and multi-queue out-of-order write SoC form, the metadata generation component 113 can also allocate a corresponding identifier for each data slice and record the start and end addresses or handles of the data area in the metadata, so that the downstream operators can still obtain and use the metadata in an unambiguous manner in a cross-channel, multi-core environment.

[0139] In one numerical configuration example, the computing unit 110 injects target data to the metadata generation component 113 with a parallelism of 128 ways per shot. The depth of the comparator tree 330 is about log2(128)=7 levels, and the intermediate nodes are introduced at the 4th, 5th, and 6th levels to align the 32 / 64 / 128 granularity block statistics (here, 32 granularity means that a statistical quantity is output for every 32 inputs); the adder tree 340 adopts a pipeline structure with the same level as the comparator tree 330 and completes the group sum in FP32 at the tree root while bypassing the Welford cumulative pair to form the mean and variance. The group descriptor sets the statistics axis to hidden_dim and sets the group length to hidden_dim or its sub-block length (32 / 64 / 128), and can optionally enable histogram to derive distribution information.

[0140] In the instant quantization mode, min / max statistics obtained by dividing data slices generate quantization parameters at the tail of the corresponding data slice, the second multiplier 352 scales the continuous data by data slices, and the packager 320 packs it into INT8 / INT4 (two-by-two or four-by-four packing to improve write channel efficiency), so that the data written back to the off-chip memory 120 is in compressed format; if FP8 is enabled, generate scaling factors by data slice granularity and re-encode floating-point numbers to FP8 E4M3 / E5M2.

[0141] The embodiment can form a data stream on each row [bs, seqlen, hidden_dim] (i.e., fixed bs and seqlen, along hidden_dim), trigger whole-dimension statistics and metadata writing at the end of the row, and form sub-block statistics and sub-block quantization parameters at the sub-block boundary of every 32 / 64 / 128 channels.

[0142] The scheme enables Softmax to directly read the maximum value of each row for value-stable subtraction and normalization, RMSNorm or other Norm to directly read the mean value or square mean value for normalization, and the quantization operator to directly read the scaling factor of different granularity, greatly reducing the repeated read-back of the original tensor.

[0143] Example device: As shown in Figure 4 The electronic device 400 is in the form of a general electronic device. The components of the electronic device 400 can include, but are not limited to, at least one computing unit 110, a memory 420, a storage device 430, one or more communication units 440, one or more input devices 450, and one or more output devices 460. The computing unit 110 can be an actual or virtual processor and can be capable of performing various processing according to programs stored in the memory 420. In a multi-processor system, multiple computing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 400.

[0144] The electronic device 400 typically includes multiple computer storage media. Such media can be any available media accessible by the electronic device 400, including both volatile and non-volatile media, removable and non-removable media. The memory 420 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 430 can be removable or non-removable media and can include machine-readable media such as a flash drive, a magnetic disk, or any other medium that can be capable of storing information and / or data and can be accessed within the electronic device 400.

[0145] The electronic device 400 can further include additional detachable / non-detachable, volatile / non-volatile storage media. Although not shown in FIG. 4, a disk drive for reading from or writing to a detachable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a detachable, non-volatile optical disk can be provided. In these cases, each drive can be connected to the bus (not shown) by one or more data media interfaces. The memory 420 can include a computer program product 425 having one or more program modules configured to carry out the various methods or acts of the various embodiments of the present disclosure. Figure 4

[0146] The communication unit 440 enables communication with other electronic devices through communication media. Additionally, the functionality of the components of the electronic device 400 can be implemented in a single computing cluster or a plurality of computer machines capable of communicating over a communication connection. As such, the electronic device 400 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0147] The input device 450 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 460 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 400 can also communicate with one or more external devices (not shown), such as a storage device, a display device, etc., one or more devices that enable a user to interact with the electronic device 400, or any devices (e.g., a network card, a modem, etc.) that enable the electronic device 400 to communicate with one or more other electronic devices, as needed, through the communication unit 440. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0148] According to an example implementation of the present disclosure, a computer readable storage medium having computer executable instructions stored thereon is provided, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0149] Various aspects of the disclosure can be described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, systems, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions. ​

[0150] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage viruses or other code or instructions implementing a functionally equivalent to that of the software manual reproduction process. The instructions can form an interface to other code or programs that or programs that can be stored in the computer readable storage medium. The instructions and / or various pieces of data and code described herein also can be stored on a computer readable storage medium whose examples store the programs for execution by a computer or that store instructions for implementation of any or all of the features having the same functionality described above. Accordingly, the computer readable storage medium includes a computer readable storage medium embodying a program of machine-readable instructions or logic or a computer program product.

[0151] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0152] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0153] The implementations of the disclosure have been described above with the intent to be illustrative rather than limiting. Although being shown and described in connection with certain implementations, it will be appreciated that various modifications and adaptations can occur to those skilled in the art. Therefore, the embodiments described herein are meant to be illustrative rather than limiting. The scope of the disclosure is intended to be limited only by the claims. The terms "comprises", "comprising", "including", "including", "having" and their conjugates mean that non-limiting inclusion, such that processes, methods, articles, or apparatuses that comprise, include, have or otherwise some feature are not to be interpreted as having necessarily excluded or excluding other features. The terms "comprises", "comprising", "including", "including", "having" and their conjugates mean that non-limiting inclusion, such that processes, methods, articles, or apparatuses that comprise, include, have or otherwise some feature are not to be interpreted as having necessarily excluded or excluding other features. The terms "coupled" and "connected", along with their derivatives, mean one or more instances of the item in question and the absence of a direct and physical connection between the items. It should be understood that many of the functional units described herein are merely illustrative and that the term "functional unit" is used herein to encompass a single functional unit, a combination of functional units, or a group of functional units.

Claims

1. A method for data processing, characterized in that, The method comprises: receiving a first instruction to write target data in an on-chip memory of a computing unit to an off-chip memory; generating, via a metadata generation component, metadata for the target data and storing the metadata in the on-chip memory, the metadata generation component being disposed between the on-chip memory and a direct memory access component, the metadata describing one or more statistical values corresponding to at least part of the target data; writing, via the direct memory access component, the target data to the off-chip memory; in response to receiving a second instruction to execute a target process for the target data with the computing unit, retrieving the metadata from the on-chip memory, the target process being related to a target statistical value among the one or more statistical values; and executing, with the computing unit, the target process for the target data based on the target statistical value in the metadata.

2. The method of claim 1, wherein, The type of the metadata is determined based on at least one subsequent process to be executed for the target data.

3. The method of claim 1, wherein, The generating, via the metadata generation component between the on-chip memory and the direct memory access component, of the metadata for the target data comprises: dividing the target data into a plurality of data slices; and determining the metadata for the target data based on the plurality of data slices.

4. The method of claim 3, wherein, The target data comprises a plurality of unit data corresponding to a preset data dimension, and the dividing the target data into a plurality of data slices comprises: dividing the target data into the plurality of data slices based on the preset data dimension, each data slice corresponding to a predetermined number of groups of unit data.

5. The method of claim 3, wherein, The determining the metadata for the target data based on the plurality of data slices comprises: generating the metadata corresponding to a plurality of granularities based on the plurality of data slices, the granularity indicating a number of data slices used to generate a statistical value.

6. The method of claim 1, wherein, The on-chip memory further maintains the metadata and corresponding data indices, and the retrieving the metadata from the on-chip memory comprises: determining at least one data index corresponding to the target data; and retrieving the metadata corresponding to the at least one data index from the on-chip memory.

7. The method of claim 6, wherein, The at least one data index indicates a batch identification and a sequence identification corresponding to the target data.

8. The method of claim 1, wherein, The method further comprises: writing, via the direct memory access component, the metadata to the off-chip memory.

9. The method of claim 1, wherein, The metadata indicates at least one statistical value of a maximum value, a minimum value, an accumulated value, a mean value, a variance, and a sum of squares.

10. The method of claim 1, wherein, The metadata comprises at least one quantization parameter used to quantize the target data, and the writing, via the direct memory access component, of the target data to the off-chip memory comprises: performing a quantization process on the target data based on the at least one quantization parameter; and writing, via the direct memory access component, the quantized target data to the off-chip memory.

11. The method of claim 1, wherein, The target data is intermediate data generated by a language model in a pre-population stage.

12. An electronic device, comprising: The electronic device comprises: at least one computing unit; and at least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit, the instructions when executed by the at least one computing unit cause the electronic device to perform the method according to any one of claims 1 to 11.

13. A computer-readable storage medium having stored thereon computer- executable instructions, wherein, The computer-executable instructions can be executed by a computing unit to implement the method according to any one of claims 1 to 11.

14. A computer program product, the computer program product being tangibly stored in a computer storage medium and comprising computer-executable instructions, the computer program product being characterized in that, The computer-executable instructions, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.

Images