Method for data processing, electronic device, storage medium and program product

Abstract

Embodiments of the present disclosure relate to a method, an electronic device, a storage medium and a program product for data processing. The method comprises: receiving a first instruction of writing target data in an on-chip memory of a computing unit to an off-chip memory; generating, via a metadata generation component, metadata for the target data and storing the metadata in the on-chip memory, the metadata describing one or more statistical values corresponding to at least part of the target data; writing, via a direct memory access component, the target data to the off-chip memory; in response to receiving a second instruction of performing a target processing on the target data using the computing unit, obtaining the metadata from the on-chip memory, the target processing being related to a target statistical value in the one or more statistical values; and performing, using the computing unit, the target processing on the target data based on the target statistical value in the metadata. In this way, embodiments of the present disclosure can improve the efficiency of data processing.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, an electronic device, a computer-readable storage medium, and a computer program product for data processing. BACKGROUND

[0002] With the rapid development of computer technology, the application scenarios of generative models are becoming more and more extensive, such as in information retrieval, dialogue assistants, code generation, and the like. The user experience of a generative model is largely dominated by the response latency. In the inference process of a generative model, the prefill stage of the Transformer architecture determines the "first token generation time", which in turn directly determines the absolute time of the response of the generative model. SUMMARY

[0003] In a first aspect of the present disclosure, a method for data processing is provided. The method comprises: receiving a first instruction to write target data in an on-chip memory of a computing unit to an off-chip memory; generating, via a metadata generation component, metadata for the target data and storing the metadata in the on-chip memory, the metadata generation component being disposed between the on-chip memory and a direct memory access component, the metadata describing one or more statistical values corresponding to at least part of the target data; writing, via the direct memory access component, the target data to the off-chip memory; in response to receiving a second instruction to execute a target process for the target data using the computing unit, obtaining the metadata from the on-chip memory, the target process being related to a target statistical value in the one or more statistical values; and executing, using the computing unit, the target process for the target data based on the target statistical value in the metadata.

[0004] In a second aspect of the present disclosure, an electronic device is provided. The electronic device comprises at least one computing unit; and at least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit. The instructions, when executed by the at least one computing unit, cause the electronic device to perform the method of the first aspect.

[0005] In a third aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon computer-executable instructions that are executable by a computing unit to implement the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0007] The scheme provided by the present disclosure sets a metadata generation component between the on-chip memory and the direct memory access component, for generating metadata for target data in the on-chip memory in the process of sending the target data to the direct memory access component, and returning the metadata to the on-chip memory; then the calculation unit can perform a target processing process for the target data based on the target statistical value in the metadata, thereby reducing the secondary acquisition and calculation process of the target data, saving data processing resources, and improving data processing efficiency.

[0008] It should be understood that the content described in this part of the content is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail some embodiments thereof with reference to the annexed drawings in which:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments according to the present disclosure can be implemented is shown;

[0011] Figure 2 A flowchart showing an example process for data processing according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic structural block diagram of a metadata generation component according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A block diagram of an electronic device capable of implementing a plurality of embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be noted that the headings provided herein are for convenience only and are not to be construed as limiting. Various embodiments are described herein, and any type of embodiment can be included under any section. Further, embodiments described in any section can be combined with any other embodiments described in the same section and / or a different section in any manner.

[0016] In the description of embodiments of the disclosure, the term "includes" and its similar terms are to be understood as open-ended, i.e., "including but not limited to". The term "based on" is to be understood as "based, at least in part, on". The term "one embodiment" or "the embodiment" is to be understood as "at least one embodiment". The term "some embodiments" is to be understood as "at least some embodiments". Other explicit or implicit definitions can also be included below. The terms "first", "second", etc. can refer to different or same objects. Other explicit and implicit definitions can also be included below.

[0017] Data of users, acquisition and / or use of data, etc. can be involved in embodiments of the disclosure. These aspects all comply with corresponding laws and regulations and relevant provisions. In embodiments of the disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are performed on the premise that the user is aware of and confirms. Accordingly, when implementing embodiments of the disclosure, the type of data or information that can be involved, the use range, the use scenario, etc. should be notified to the user and the authorization of the user should be obtained according to relevant laws and regulations through appropriate means. The specific notification and / or authorization mode can vary according to the actual situation and application scenario, and the scope of the disclosure is not limited in this respect.

[0018] In the specification and embodiments of the disclosure, if personal information processing is involved, it will be processed on the premise of legality (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and only within the prescribed or agreed range. Users refuse to process personal information other than the necessary information required for basic functions, which will not affect the user's use of basic functions.

[0019] As mentioned above, the user experience of the generative model is largely dominated by the response latency. In the inference process of the generative model, the prefill stage of the Transformer architecture determines the "first token generation time", which directly determines the absolute time of the generative model response.

[0020] In the pre-padded data flow of the Transformer architecture, the intermediate activations between different data layers are often in row or channel units, and statistics and normalization are performed on the last dimension. For example, the per-row maximum subtraction required by Softmax, the mean or square mean required by RMSNorm or other Norm, the min / max or quantile interval required by quantization / dequantization, etc.

[0021] In existing implementations, these statistics described above often need to be calculated again by the downstream operator after the upstream operator has written data to external memory; if the downstream operator also needs to perform quantization processing on the data (for example, quantization after subtracting the maximum value, deriving Scale according to min / max, etc.), at least one complete read traversal overhead will be generated, which will cause significant bandwidth and delay burden in the data processing process under long sequence (large seqlen) and large channel dimension (large hidden_dim). Although kernel fusion and online algorithms can reduce the temporary round trip of data within the operator, when the intermediate results of data processing need to be landed on external memory and multiple subsequent data processing (such as matrix multiplication, Softmax, RMSNorm, quantization / dequantization, activation clipping, KV compression, etc.) share the same or related statistics, repeated read-back and repeated calculation are still unavoidable.

[0022] Embodiments of the present disclosure provide a scheme for data processing. The scheme includes: receiving a first instruction to write target data in an on-chip memory of a computing unit to an off-chip memory; generating, via a metadata generation component, metadata for the target data and storing the metadata in the on-chip memory, the metadata generation component being disposed between the on-chip memory and a direct memory access component, the metadata describing one or more statistical values corresponding to at least part of the target data; writing, via the direct memory access component, the target data to the off-chip memory; in response to receiving a second instruction to perform a target processing on the target data using the computing unit, obtaining the metadata from the on-chip memory, the target processing being related to a target statistical value of the one or more statistical values; and performing, using the computing unit, the target processing on the target data based on the target statistical value in the metadata.

[0023] In this way, embodiments of the present disclosure set a metadata generation component between the on-chip memory and the direct memory access component, for generating metadata for target data in the on-chip memory in the process of sending the target data to the direct memory access component, and returning the metadata to the on-chip memory; then the computing unit can perform a target processing process for the target data based on the target statistical value in the metadata, thereby reducing the secondary acquisition and calculation process for the target data, saving data processing resources, and improving data processing efficiency.

[0024] The present solution extracts the required statistical quantity in a "streaming online" manner in the path of writing the on-chip memory to the off-chip memory, and provides it to the on-chip memory in the form of metadata for the computing unit to perform subsequent data processing processes, thereby effectively eliminating or reducing additional data read traversal, reducing unnecessary consumption of data processing, and improving data processing efficiency.

[0025] Various example implementations of the present solution are described in further detail below in conjunction with the accompanying drawings.

[0026] Example Environment:

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. As shown, the example environment 100 can include a computing unit 110. The computing unit 110 can include a processor core 111 and an on-chip memory 112. Figure 1

[0028] In some embodiments, the computing unit 110 can be a standalone electronic device, or a software unit or hardware unit embedded in an electronic device. As an example, such an electronic device can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile handset, a tablet computer, a notebook computer, a laptop computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a palmtop computer, a portable gaming terminal, a VR / AR device, a Personal Communication System (PCS) device, a personal navigation device, a Personal Digital Assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination thereof, including accessories and peripherals of such devices or any combination thereof. In some embodiments, the electronic device can also support any type of interface to the user (such as "wearable" circuitry, etc.).

[0029] ​The computing unit 110 can store at least one of data to be processed, intermediate data during processing, processed result data, etc. in the on-chip memory 112.

[0030] Referring to Figure 1 As shown, the example environment 100 can also include an off-chip memory 120, and the computing unit 110 can also include a direct memory access component 114. The direct memory access component 114 can be configured to obtain data stored in the on-chip memory 112 and store the obtained data in the off-chip memory 120.

[0031] As an example, the off-chip memory 120 can be an external random access memory, such as an external dynamic random access memory (DRAM), including but not limited to a DDR series memory, a LPDDR memory, a GDDR memory, or a HBM memory, or some combination thereof, without limitation.

[0032] In the scheme of the present disclosure, a metadata generation component 113 is arranged between the on-chip memory 112 and the direct memory access component 114. As an example, target data in the on-chip memory 112 needs to pass through the metadata generation component 113 in the process of flowing to the direct memory access component 114. The metadata generation component 113 generates metadata corresponding to the target data in the process of flowing of the target data. For example, the metadata generation component 113 can perform at least one statistical processing on the target data to generate corresponding metadata.

[0033] The metadata generation component 113 can store the generated metadata in the on-chip memory 112 for the computing unit 110 to perform subsequent data computing and other processing.

[0034] In some scenarios, after generating the metadata, the metadata generation component 113 can also send the generated metadata to the direct memory access component 114, and the direct memory access component 114 stores both the target data and the metadata corresponding to the target data in the off-chip memory 120, so that other computing units or data processing devices can directly obtain the corresponding content in the metadata to perform corresponding data processing.

[0035] As an example, the computing unit 110 can support at least part of the data processing process of a generative model or a language model. For example, for a generative model with a multi-layer Transformer cascade structure, each layer can be deployed with a Softmax operator and an RMSNorm operator. Both the Softmax operator and the RMSNorm operator can utilize the metadata generated by the metadata generation component 113 in the computing unit 110 of the present disclosure to perform subsequent computing or processing, so as to improve the data processing efficiency.

[0036] In some implementations, the matrix multiplication processing procedure in the generative model or the language model can also utilize the metadata generated by the metadata generation component 113 to achieve data quantization acceleration and improve data processing efficiency.

[0037] It should be understood that the structure and function of the various elements in the example environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0038] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0039] Example process:

[0040] Figure 2 A flowchart of an example process 200 of processing data according to some embodiments of the present disclosure is shown. The example process 200 can be implemented at the computing unit 110. The example process 200 will be described below with reference to Figure 1 .

[0041] As shown in Figure 2 , at block 210, the computing unit 110 receives a first instruction to write target data in the on-chip memory 112 of the computing unit 110 to the off-chip memory 120.

[0042] As an example, the computing unit 110 can utilize the processor core 111 or other instruction processing component to receive the first instruction to write the target data in the on-chip memory 112 to the off-chip memory 120.

[0043] In some embodiments, the target data can be intermediate data generated by the language model in a pre-filling stage.

[0044] At block 220, the computing unit 110 generates metadata for the target data via the metadata generation component 113, and stores the metadata in the on-chip memory 112, the metadata generation component 113 being disposed between the on-chip memory 112 and the direct memory access component 114, the metadata describing one or more statistical values corresponding to at least part of the data in the target data.

[0045] In the present disclosure, the data in the on-chip memory 112 no longer flows directly to the direct memory access component 114, but the metadata generation component 113 is arranged between the on-chip memory 112 and the direct memory access component 114, and the data in the on-chip memory 112 needs to pass through the metadata generation component 113 before flowing to the direct memory access component 114. In this process, the metadata generation component 113 processes the target data flowing to the direct memory access component 114 to determine one or more statistical values corresponding to at least part of the data in the target data, and generates metadata corresponding to the target data according to the one or more statistical values, and then writes the generated metadata into the on-chip memory 112.

[0046] In some embodiments, the metadata can indicate at least one of the following statistical values: maximum value, minimum value, cumulative value, sum of squares, mean value, variance, etc.

[0047] As an example, the metadata can also include histogram or percentile approximation value of the target data, etc.

[0048] In this way, the embodiments of the present disclosure can perform multiple types of statistics on at least part of the target data to determine corresponding multiple statistical values, thereby effectively enriching the data content of the metadata, enriching the processing types that can be performed on the target data based on the metadata, and further improving the efficiency of subsequent processing performed using the metadata.

[0049] In some embodiments, the type of the metadata is determined based on at least one subsequent processing to be performed on the target data.

[0050] As an example, such subsequent processing can include but is not limited to any of the following: finding the maximum value or the minimum value, finding the variance, the mean value, the square mean value, or the sum of squares, performing quantization or dequantization on at least part of the target data, etc.

[0051] In this way, the embodiments of the present disclosure can generate metadata of corresponding types based on the types of subsequent processing that can be performed on the target data, thereby ensuring the processing efficiency of the subsequent processing, and avoiding the waste of computing and storage resources caused by generating unnecessary types of metadata.

[0052] In some implementation scenarios of the present disclosure, the metadata generated by the metadata generation component 113 can include not only the whole-dimension metadata of the target data in whole-dimension statistics, but also at least one local metadata corresponding to at least one group of local data obtained by dividing the target data according to at least one granularity.

[0053] In some embodiments, generating the metadata for the target data via the metadata generation component 113 between the on-chip memory 112 and the direct memory access component 114 includes: dividing, by the computing unit 110, the target data into multiple data slices using the metadata generation component 113; and determining the metadata for the target data based on the multiple data slices.

[0054] In the process of writing the target data in the on-chip memory 112 to the off-chip memory 120, the computing unit 110 divides the target data using the metadata generation component 113 in the process of flowing through the metadata generation component 113, to obtain multiple data slices of the same data size, and then performs parallel statistics on the multiple data slices to generate local metadata corresponding to each data slice, and can also generate whole-dimension metadata corresponding to the target data, thereby obtaining metadata associated with the target data.

[0055] In this way, the scheme can not only generate corresponding whole-dimension metadata for the target data, but also generate local metadata corresponding to the data slices after division, so as to synchronously obtain metadata corresponding to different data granularities of the target data, effectively enrich the data content of the metadata, and enrich the processing types that can be performed on the target data based on the metadata, and further improve the efficiency of subsequent processing using the metadata.

[0056] In some embodiments, the target data includes a plurality of unit data corresponding to a preset data dimension, and dividing the target data into a plurality of data slices includes: the computing unit 110 divides the target data into a plurality of data slices based on the preset data dimension, each data slice corresponding to a predetermined number of groups of unit data.

[0057] As an example, such a preset data dimension can include the lowest data dimension corresponding to the target data. For example, the target data includes 128 data matrices of the same size, and the lowest data dimension can be that each data matrix is independently taken as a data slice, that is, the number of data slices obtained by division can be 128.

[0058] In some scenarios, the preset data dimension can also include any dimension higher than the lowest data dimension and lower than the highest data dimension (whole-dimension data). Continuing with the example that the target data includes 128 data matrices of the same size, the number of data slices corresponding to the preset data dimension can also be 64, 32, 16, 8, 4, 2, etc.

[0059] In some implementations, the computing unit 110 can divide the target data corresponding to at least one data granularity (e.g., one data as a group, two data as a group, or more data as a group, etc.) according to the preset data dimension, to obtain a plurality of data slices corresponding to at least one data granularity. The plurality of data slices corresponding to the same data granularity have the same number of data in the corresponding group of unit data.

[0060] Taking seeking the maximum value or the minimum value as an example, the metadata generation component 113 can construct a corresponding comparator tree by arranging a plurality of levels of comparators. The data dimension and data granularity corresponding to the plurality of comparators in each level are the same, that is, the number of data in the group of unit data corresponding to the plurality of comparators is the same, and the data granularity of the plurality of metadata (e.g., maximum value or minimum value) output by the plurality of comparators is the same.

[0061] The metadata generation component 113 can utilize the structure of the comparator tree to output at least one maximum value or minimum value at each level, thereby obtaining metadata of multiple different levels and the whole-dimension metadata corresponding to the target data, effectively enriching the data content of the metadata and the processing type and processing granularity that can be performed on the target data based on the metadata, further improving the efficiency of subsequent processing using the metadata, and effectively expanding the applicable scenarios of the metadata.

[0062] In this way, the embodiments of the present disclosure can divide the target data based on the preset data dimension, thereby effectively ensuring that multiple data slices of the same data dimension correspond to the same number of data, and further ensuring the accuracy and effectiveness of the metadata of the same data dimension.

[0063] In some implementations, the computing unit 110 can mark the division markers corresponding to the multiple data slices after dividing the target data into multiple data slices. For example, such division markers can include grouping descriptors and boundary markers, etc.

[0064] As an example, the computing unit 110 can write the division markers in association with the metadata when writing the metadata into the on-chip memory 112 using the metadata generation component 113, to construct a corresponding data index.

[0065] In some embodiments, based on the multiple data slices, determining the metadata for the target data includes that the computing unit 110 generates metadata corresponding to multiple granularities based on the multiple data slices, the granularity indicating the number of data slices used to generate the statistical value.

[0066] Continuing with the example of the comparator tree, the computing unit 110 can utilize the comparator tree in the metadata generation component 113 to first divide the target data at the minimum granularity, and then accumulate layer by layer, which can achieve the division of the target data at multiple different granularities, i.e., obtaining multiple data slices corresponding to multiple different granularities respectively. The metadata generation component 113 can obtain the metadata corresponding to each granularity synchronously in the process of utilizing the comparator tree to achieve the division at multiple different granularities, thereby achieving the generation of metadata corresponding to multiple different granularities for the target data, and further improving the efficiency of the generation of the metadata.

[0067] In this way, the embodiments of the present disclosure can generate metadata corresponding to multiple different granularities for the target data, effectively enriching the data content of the metadata and the processing type that can be performed on the target data based on the metadata, further improving the efficiency of subsequent processing using the metadata.

[0068] Figure 3 A schematic structural block diagram of the metadata generation component 113 according to some embodiments of the present disclosure is shown. The following is comprehensively referred to Figure 1 andFigure 3 The generation process of the metadata is exemplarily described.

[0069] As shown in Figure 3 The metadata generation component 113 includes a multiplexer 310. As an example, the target data from the on-chip memory 112 is written to the off-chip memory 120 via the multiplexer 310 in a multi-lane parallel input manner.

[0070] In the scheme of the present disclosure, the metadata generation component 113 can not only write the target data to the off-chip memory 120 via the multiplexer 310, but also generate the metadata corresponding to the target data. For example, the metadata generation component 113 can also generate the metadata corresponding to at least one granularity of the target data in the streaming process of the target data, and such metadata can include at least one of the following statistical values: maximum value, minimum value, cumulative value, mean value, variance, etc.

[0071] In some scenarios, for example, the data size of the target data is too large or the available storage space of the on-chip memory 112 is small, the metadata generation component 113 can also write the generated metadata to the off-chip memory 120.

[0072] As an example, the metadata generation component 113 can also include a packer 320. The metadata generation component 113 sends the target data from the on-chip memory 112 to the packer 320 via the multiplexer 310, and also sends the generated metadata to the packer 320, so that the packer 320 packs the metadata and the target data together to write to the off-chip memory 120. For example, the packer 320 can write the packed data to the off-chip memory 120 via the direct memory access component 114.

[0073] In some scenarios, taking the metadata including the maximum value or the minimum value as an example, the metadata generation component 113 can include a comparator tree 330 and a first register group 331. The target data from the on-chip memory 112 is input to the comparator tree 330 in a multi-lane parallel input manner, and a plurality of maximum values corresponding to different granularities of the target data are output. The comparator tree 330 can write the comparison results of each layer of comparators (i.e., the plurality of maximum values output by the layer) to the first register group 331, and can also write to the on-chip memory 112.

[0074] In some scenarios, the comparator tree 330 can only compare a fixed number of input data at a time. For example, if the comparator tree 330 supports a maximum of 128-way input, it can only count the maximum or minimum value of 128-way input data at a time. If the target data exceeds 128-way, the comparator tree 330 needs to perform multiple counting processes. In this case, the comparator tree 330 can store the execution result corresponding to each 128-way input in the first register group 331, and then perform the next 128-way input. By analogy, the final statistical result of each 128-way input is taken as the input of the comparator tree 330 again to obtain the final statistical result of the target data corresponding to the metadata of different granularities.

[0075] The present disclosure aims to protect the deployment and application of the metadata generation component 113 in the computing unit 110, and does not aim to emphasize the specific implementation structure of the comparator tree 330. In actual application, the specific implementation structure of the comparator tree 330 can refer to related technologies, which will not be repeated here.

[0076] Taking the pre-padding operation process in the Transformer architecture model as an example, the Softmax operator mainly performs the processing process of converting a set of real numbers (usually called logits) into a probability distribution, including: first performing exponential operation on the input data, and then normalizing the result to make the sum equal to 1, thereby forming a valid probability distribution. In actual application, in order to ensure numerical stability, the maximum value is usually subtracted from the input data before the exponential operation and normalization.

[0077] According to the scheme of the present disclosure, in the scenario where the Softmax operator needs to process the target data, the computing unit 110 can use the metadata generation component 113 to count the maximum value in the target data as at least part of the data content of the metadata corresponding to the target data during the process of writing the target data to the off-chip memory 120, and write it back to the on-chip memory 112. Then the Softmax operator can obtain the corresponding maximum value from the on-chip memory 112 without the need to obtain the complete target data and perform the process of counting the maximum value, thereby effectively reducing the read consumption of the target data and improving the processing efficiency of the Softmax operator.

[0078] The following takes the data structure of the target data corresponding to the Softmax operator as an example to illustrate the process of counting the maximum value in the target data by the metadata generation component 113.

[0079] The data structure of the Softmax operator input and output is [batch_size, num_head, q_length, k_length], wherein batch_size represents the batch size of the data, num_head represents the number of attention heads, q_length represents the query sequence length, and k_length represents the key sequence length.

[0080] The metadata generation component 113 takes the last dimension [k_length] of the input data (i.e., target data) [batch_size, num_head, q_length, k_length] as the maximum value, that is, the [k_length] corresponding to each [batch_size, num_head, q_length] is taken as the minimum input unit of the comparator tree 330 (i.e., the input data of the first layer of the comparator tree 330). The maximum values corresponding to the minimum data granularity are output by the first layer of the comparator, and are stored in the first register group 331 and used as the input data of the second layer of the comparator; the maximum values corresponding to the second granularity are counted by the second layer of the comparator, and are stored in the first register group 331 and used as the input data of the third layer of the comparator; and so on. The maximum values corresponding to different data granularities of the target data are counted by the comparator tree 330 layer by layer until the maximum value corresponding to the whole dimension of the target data is finally output, which is stored in the first register group 331 and can also be written into the on-chip memory 112. In this process, the maximum values corresponding to the same data granularity in the comparator tree 330 can be counted in parallel.

[0081] After the metadata generation component 113 counts the maximum value of the target data, the Softmax operator can directly perform data processing based on the maximum value. For example, the Softmax operator subtracts the maximum value from all values of [k_length], and then calculates the output according to the standard formula. As an example, the calculation formula of the Softmax operator can be expressed as:

[0082] ;

[0083]

[0084] For the scenario in which the metadata includes the minimum value, the comparator tree 330 and the first register group 331 described above can also be used. The process of generating the minimum value by the metadata generation component 113 can refer to the process of generating the maximum value described above, which will not be described here.

[0085] In some embodiments of the present disclosure, all the maximum values and / or minimum values written in the first register group 331 can also be sent to the packer 320 during the process of writing into the on-chip memory 112, and the target data and the multiple maximum values and / or minimum values corresponding to different granularities are jointly packed by the packer 320 and sent to the direct memory access component 114 for writing into the off-chip memory 120.

[0086] For example, in response to the storage space of the on-chip memory 112 being less than a first threshold, or in response to the data size of the target data or the metadata (e.g., the multiple maximum values and / or minimum values) being greater than a second threshold, the metadata generation component 113 can no longer write the generated metadata into the on-chip memory 112, but write the metadata (e.g., the multiple maximum values and / or minimum values) and the target data jointly via the packer 320 into the off-chip memory 120 to save the storage space of the on-chip memory 112.

[0087] In some embodiments, taking the metadata including the accumulated values as an example, the metadata generation component 113 can include an adder tree 340 and a second register group 341. The target data from the on-chip memory 112 is input into the adder tree 340 in a multi-parallel manner to obtain multiple accumulated values corresponding to different granularities of the target data. The adder tree 340 writes the multiple accumulated values output by each layer of adders into the second register group 341 and can also write into the on-chip memory 112.

[0088] In some scenarios, the comparator tree 330 can only compare a fixed number of inputs at a time. Taking the comparator tree 330 supporting a maximum of 128 inputs as an example, the comparator tree 330 can only count the accumulated values of 128 inputs at a time. If the target data exceeds 128 inputs, the adder tree 340 needs to perform multiple counting processes. In this case, the adder tree 340 can store the execution results corresponding to every 128 inputs in the second register group 341, and then perform the next set of 128 inputs. In this way, the execution results corresponding to each set of 128 inputs are input into the adder tree 340 again to obtain the final counting results and generate the metadata corresponding to different granularities of the target data.

[0089] The present disclosure aims to protect the deployment and application of the metadata generation component 113 in the computing unit 110, and does not aim to emphasize the specific implementation structure of the adder tree 340. In actual applications, the specific implementation structure of the adder tree 340 can refer to related technologies, which are not described here in detail.

[0090] In some scenarios, the metadata generation component 113 can further include a first multiplier 342, for example, to include the square sum in the accumulated value. The first multiplier 342 can be configured to calculate the square value of any data in the target data. The target data from the on-chip memory 112 is input in a multi-lane parallel manner, and the square value of each data in the target data is calculated one by one via the first multiplier 342, and then the accumulated calculation of the square values corresponding to the target data at different granularities is performed by using the adder tree 340 to determine the square sums of the target data at different granularities. The adder tree 340 can further write the square sums at each granularity to the second register group 341 and the on-chip memory 112.

[0091] Continuing with the example of the pre-filling operation in the Transformer architecture model, the RMSNorm operator can normalize the features of the target data by using the root mean square of the target data. The data structure of the input and output of the RMSNorm operator is [batch_size, seqlen, hidden_dim], where batch_size represents the batch size, seqlen represents the sequence length, and hidden_dim represents the hidden dimension.

[0092] Taking the RMSNorm operator as an example, the process of calculating the root mean square of the last dimension [hidden_dim] of the target data [batch_size, seqlen, hidden_dim] can be expressed as:

[0093] ;

[0094] wherein is a preset value to prevent data instability caused by division by zero.

[0095] As an example, the final output result of the RMSNorm operator can be expressed as wherein is a learned scaling parameter.

[0096] As can be seen, in the process of calculating the root mean square of the target data by the RMSNorm operator, the square sum of the target data needs to be calculated first, then the mean value is calculated, and then the root mean square value is calculated. The square sum of the target data can be generated by the metadata generation component 113 to reduce the data acquisition and statistical processing process of the RMSNorm operator and improve the data processing efficiency.

[0097] As an example, the metadata generation component 113 can utilize the first multiplier 342 to square the last dimension [hidden_dim] of the input data (i.e., the target data) [batch_size, seqlen, hidden_dim], that is, the first multiplier 342 calculates the corresponding square value for each [hidden_dim], and then takes the calculated square value as the input of the adder tree 340. That is, the first multiplier 342 takes the calculated square value as the minimum input unit of the adder tree 340 (i.e., the input data of the first layer adder of the adder tree 340). The multiple square sums corresponding to the minimum data granularity are output by the first layer adder and stored in the second register group 341 and taken as the input data of the second layer adder; the multiple square sums corresponding to the second granularity are counted by the second layer adder and stored in the second register group 341 and taken as the input data of the third layer adder; and so on and so forth, the adder tree 340 is utilized to count the multiple square sums corresponding to different data granularities of the target data layer by layer until the square sum corresponding to the whole dimension of the target data is finally output and stored in the second register group 341 and can also be written into the on-chip memory 112. In this process, the multiple square sums corresponding to the same data granularity in the adder tree 340 can be counted in parallel.

[0098] In some embodiments, all the accumulated values (e.g., accumulated sums, square sums, etc.) written into the second register group 341 can also be sent to the packer 320 in the process of being written into the on-chip memory 112, and the target data and the multiple accumulated values corresponding thereto of different granularities are collectively packed by the packer 320 and sent to the direct memory access component 114 to be written into the off-chip memory 120.

[0099] As an example, in response to the storage space of the on-chip memory 112 being less than a first threshold, or in response to the data scale of the target data or the metadata (e.g., the multiple accumulated values) being greater than a second threshold, the metadata generation component 113 can no longer write the generated metadata (e.g., the multiple accumulated values) into the on-chip memory 112, but write the metadata (e.g., the multiple accumulated values) and the target data collectively via the packer 320 into the off-chip memory 120 to save the storage space of the on-chip memory 112.

[0100] In some scenarios of the present disclosure, the subsequent processing process for the target data can also include a quantization or dequantization process.

[0101] For example, in a scenario where the candidate processing procedure for the target data includes a quantization procedure, the quantization procedure can include a matrix multiplication or convolution operation procedure in a model. Before quantization, the data type of the matrix multiplication or convolution operation can be FP16 or BFP16. The quantization procedure can convert the target data into low-precision data (e.g., INT8, INT4, or FP8, generally, a data type of less than or equal to 8 bits can be referred to as a low-precision data format), then perform the matrix multiplication or convolution operation using the converted low-precision data, and convert the operation result into a high-precision data format that is the same as the target data. In this process, both the conversion of the target data into low-precision data and the conversion of the operation result of the low-precision data into a high-precision data format need to use a scaling coefficient, and such a scaling coefficient is associated with the data granularity of the conversion of the target data into low-precision data.

[0102] As can be seen, for a scenario where quantization needs to be performed on target data, the metadata generation component 113 in the present solution can generate corresponding scaling coefficients in the process of writing the target data from the on-chip memory 112 to the off-chip memory 120. As an example, for target data, the metadata generation component 113 can calculate a scaling coefficient (also referred to as a quantization coefficient) corresponding to a maximum value of the entire tensor range of the target data, or calculate a scaling coefficient corresponding to a maximum value of every 128 / 64 / 32 / 16 data elements.

[0103] Thus, the present solution can effectively avoid repeated acquisition of target data for calculating a maximum value and a scaling coefficient in a quantization procedure, thereby saving data bandwidth and reducing end-to-end delay caused by excessive occupation of data bandwidth.

[0104] The metadata generation component 113 can use the comparator tree 330 to generate multiple maximum values corresponding to different granularities for the target data, and then input at least one maximum value associated with the scaling granularity of the target data to the scaling coefficient solver 351 to calculate at least one corresponding scaling coefficient.

[0105] Reference Figure 3As shown, the metadata generation component 113 can further include a scaling factor solver 351 and a second multiplier 352. The scaling factor solver 351 is configured to calculate the scaling factor corresponding to the target data and send the scaling factor to the second multiplier 352 and send the reciprocal of the scaling factor to the packer 320. The target data from the on-chip memory 112 is input to the second multiplier 352 in a multi-lane parallel manner, and then the second multiplier 352 performs multiplication operation on each data element in the target data using the scaling factor to convert the target data into a low-precision data format, and sends the calculation result (the low-precision data corresponding to the target data) to the packer 320 via the multiplexer 310. The target data, the low-precision data corresponding to the target data, and the reciprocal of the scaling factor are packed by the packer 320 and sent to the direct memory access component 114 for writing to the off-chip memory 120.

[0106] In this way, the execution subject of the matrix multiplication or convolution operation does not need to obtain the target data and calculate the corresponding scaling factor and perform data conversion when performing quantization on the target data, but can directly obtain the low-precision data corresponding to the target data and the reciprocal of the scaling factor from the off-chip memory 120 to perform the matrix multiplication or convolution operation, and convert the operation result into a data format with the same precision as the target data using the reciprocal of the scaling factor.

[0107] In the present solution, the scaling factor and the reciprocal of the scaling factor appear in pairs. The reciprocal of the scaling factor can also be sent by the scaling factor solver 351 to the second multiplier 352 to perform the first data conversion (e.g., which can be used in the dequantization process), and the scaling factor can be sent to the packer 320 to facilitate subsequent processing and then perform the second data conversion using the scaling factor to restore the data format to the same as the target data.

[0108] In some implementations, the scaling factor solver 351 can also calculate the scaling factor based on the variance of the target data first, and then calculate the scaling factor based on the variance. As an example, the adder tree 340 can obtain multiple square sums corresponding to different granularities of the target data after performing accumulation calculation of different granularities based on the square values output by the first multiplier 342. Then the scaling factor solver 351 can obtain at least one square sum of a corresponding granularity from the output of the adder tree 340 based on the scaling granularity of the target data for calculating the scaling factor and the reciprocal of the scaling factor.

[0109] In some implementations, if the storage space of the on-chip memory 112 is large enough, the second multiplier 352 can write the data converted based on the scaling factor (or the reciprocal of the scaling factor) and the reciprocal of the scaling factor (or the scaling factor) output by the scaling factor solver 351 into the on-chip memory 112 to facilitate other execution subjects to quickly obtain the corresponding data from the on-chip memory 112.

[0110] Referring back to Figure 2 As shown in block 230, the computing unit 110 writes the target data to the off-chip memory 120 via the direct memory access component 114.

[0111] By way of example, with reference to Figure 1 and Figure 3 As shown, the computing unit 110 can send the target data in the on-chip memory 112 to the direct memory access component 114 via the multiplexer 310 in the metadata generation component 113, and write the target data to the off-chip memory 120 by the direct memory access component 114.

[0112] In some embodiments, the metadata includes at least one quantization parameter for quantizing the target data, and writing the target data to the off-chip memory via the direct memory access component includes: the computing unit 110 performing a quantization process on the target data based on the at least one quantization parameter; and writing the quantized target data to the off-chip memory via the direct memory access component 114.

[0113] By way of example, the at least one quantization parameter can include a scaling factor or a scaling factor inverse corresponding to the target data.

[0114] In some scenarios, the computing unit 110 can utilize the metadata generation component 113 to perform a quantization process on the target data based on the at least one quantization parameter. With reference to Figure 3 As shown, the computing unit 110 can utilize the second multiplier 352 in the metadata generation component 113 to perform a quantization process on the target data based on the scaling factor to obtain low-precision data corresponding to the target data, or based on the scaling factor inverse to obtain high-precision data corresponding to the target data.

[0115] By way of example, in the process of writing the quantized target data to the off-chip memory 120 via the direct memory access component 114, the parameter inverse corresponding to the at least one quantization parameter can also be written to the off-chip memory 120.

[0116] Taking the scaling factor resolver 351 delivering the scaling factor to the second multiplier 352 as an example, the packer 320 in the metadata generation component 113 can pack the scaling factor inverse from the scaling factor resolver 351 and the target data quantization result output by the second multiplier 352, and write them to the off-chip memory 120 via the direct memory access component 114.

[0117] With the scaling coefficient solver 351 delivering the scaling coefficient inverse to the second multiplier 352 as an example, the packer 320 in the metadata generation component 113 can pack the scaling coefficient from the scaling coefficient solver 351 and the target data quantization result output by the second multiplier 352, and write into the off-chip memory 120 via the direct memory access component 114.

[0118] In this way, the embodiments of the present disclosure can perform quantization processing on the target data, and write the quantized target data into the off-chip memory, thereby effectively reducing the data reading and calculation consumption in the subsequent target data quantization process, and improving the subsequent data processing efficiency.

[0119] Returning to continue referring to Figure 2 As shown, at block 240, the computing unit 110 obtains the metadata from the on-chip memory 112 in response to receiving a second instruction to utilize the computing unit 110 to perform a target processing on the target data, the target processing being related to a target statistical value in one or more statistical values.

[0120] In the present solution, such a target processing can include but is not limited to: determining a probability distribution, normalization, matrix multiplication or convolution operation, etc. Such a target statistical value can include but is not limited to: maximum value, minimum value, sum of squares, mean value, variance, low-precision data and corresponding scaling coefficient inverse, high-precision data and corresponding scaling coefficient, etc.

[0121] As an example, in response to the second instruction indicating that the target processing is to determine a probability distribution, the computing unit 110 can obtain at least one maximum value corresponding to at least one granularity of the target data from the on-chip memory, such at least one maximum value being generated by the metadata generation component 113 (e.g. the comparator tree 330) and written into. In response to the second instruction indicating that the target processing is normalization, the computing unit 110 can obtain at least one sum of squares corresponding to at least one granularity of the target data from the on-chip memory, such at least one sum of squares being generated by the metadata generation component 113 (e.g. the adder tree 340) and written into. In response to the second instruction indicating that the target processing is matrix multiplication or convolution operation, the computing unit 110 can obtain the quantization result corresponding to the target data and the scaling coefficient or the scaling coefficient inverse from the on-chip memory.

[0122] In some embodiments, the on-chip memory also maintains the metadata and corresponding data index, and obtaining the metadata from the on-chip memory includes: the computing unit 110 determines at least one data index corresponding to the target data; and obtains the metadata corresponding to the at least one data index from the on-chip memory.

[0123] In the scheme, the metadata generation component 113 can synchronize the data index corresponding to the metadata to the on-chip memory 112 in the process of writing the metadata to the on-chip memory 112.

[0124] As an example, such a data index is determined in the process of generating the metadata. For example, the data index can be determined by dividing the target data to obtain a plurality of data slices.

[0125] In this way, the embodiments of the present disclosure can quickly obtain the corresponding metadata based on the data index by maintaining the metadata and the corresponding data index by using the on-chip memory, and improve the metadata search and acquisition efficiency.

[0126] In some embodiments, at least one data index indicates a batch identification and a sequence identification corresponding to the target data.

[0127] As an example, such a batch identification can indicate the generation batch of the target data corresponding to the metadata.

[0128] As an example, such a sequence identification can indicate sequence information corresponding to the target data, or can indicate the sequence position of the data slice corresponding to the metadata in the target data.

[0129] In this way, the embodiments of the present disclosure can quickly locate the metadata information corresponding to the target data from the data index by the batch identification and the sequence identification, thereby quickly searching and acquiring the corresponding metadata, effectively improving the metadata search and acquisition efficiency, and ensuring the accuracy of the acquired metadata.

[0130] In block 250, the computing unit 110 performs target processing on the target data based on the target statistical value in the metadata.

[0131] In this way, the embodiments of the present disclosure set the metadata generation component between the on-chip memory and the direct memory access component, for generating the metadata for the target data in the process of sending the target data in the on-chip memory to the direct memory access component, and returning the metadata to the on-chip memory; then the computing unit can perform target processing on the target data based on the target statistical value in the metadata, thereby reducing the secondary acquisition and calculation process of the target data, saving data processing resources, and improving data processing efficiency.

[0132] The scheme of the present disclosure utilizes the metadata generation component set between the on-chip memory and the direct memory access component, and migrates the metadata generation process from the downstream operator side back to the data write-back process of writing the data in the on-chip memory to the off-chip memory, which can form a unified and decoupled general statistical and quantization service from specific operators, thereby effectively reducing redundant data read-back and repeated calculation.

[0133] In some embodiments, the process 200 can further include that the computing unit 110 writes the metadata into the off-chip memory 120 via the direct memory access component 114.

[0134] As an example, in response to determining that the storage space of the on-chip memory 112 is less than the first threshold, or in response to determining that the data size of the target data or the metadata is greater than the second threshold, the computing unit 110 can utilize the packer 320 in the metadata generation component 113 to pack the metadata together with the target data, and write the packed data into the off-chip memory 120 via the direct memory access component 114. In this way, the present scheme not only enables effective storage of the metadata, but also guarantees the association between the metadata and the target data.

[0135] As an example, in the process of writing the metadata into the off-chip memory 120, the direct memory access component 114 can not only associate the metadata with the corresponding target data, but also construct a data index corresponding to the metadata by associating the partition markers corresponding to the plurality of data slices associated with the metadata, and write the data index into the off-chip memory 120. For example, the computing unit 110 can utilize the metadata generation component 113 to construct the corresponding data index and send it to the packer 320, which packs the data index, the metadata, and the target data together and writes them into the off-chip memory 120 via the direct memory access component 114.

[0136] In this way, the embodiments of the present disclosure can write the metadata into the off-chip memory, effectively enriching the storage path and storage method of the metadata, thereby ensuring that the metadata is effectively stored and ensuring the effectiveness of the metadata.

[0137] The following is a comprehensive reference Figure 1 and Figure 3 The data structure generated by the model in the inference process is an intermediate activation tensor of [bs, seqlen, hidden_dim], which is taken as an example to describe the implementation process provided by the present disclosure for data processing. In the target data, bs represents the batch size, seqlen represents the sequence length, and hidden_dim represents the hidden dimension.

[0138] As an example, such a compute unit 110 can be a linear projection of a certain feed-forward layer or attention output of the model. The processor core 111 in the compute unit 110 stores the target data in the on-chip memory 112 after generating the target data. In response to receiving a first instruction to write the target data to the off-chip memory 120, the compute unit 110 can read the target data out of the on-chip memory 112 and stream the target data to the direct memory access component 114 via the metadata generation component 113. As an example, the compute unit 110 can write the target data in rows to the off-chip memory 120, and such a row can be defined as a contiguous slice in the hidden_dim dimension.

[0139] In this process, the metadata generation component 113 can directly send the target data to the direct memory access component 114 using the multiplexer 310. Meanwhile, the metadata generation component 113 can generate metadata corresponding to at least one data granularity of the target data using at least one of the comparator tree 330, the first multiplier 342 and the adder tree 340, the scaling coefficient solver 351, and the second multiplier 352. The metadata generation component 113 can write the metadata back to the on-chip memory 112, or can send the metadata to the direct memory access component 114 together with the target data via the packer 320 for writing to the off-chip memory 120.

[0140] In some implementations, the metadata generation component 113 can divide the target data into a plurality of data slices according to the number of allowed input paths of the comparator tree 330 or the adder tree 340 to obtain the plurality of data slices; or can divide the target data into a plurality of data slices according to the number of configured layers of the comparator tree 330 or the adder tree 340 to obtain the plurality of data slices.

[0141] It should be noted that the metadata generation component 113 can determine the group descriptor and the boundary marker corresponding to each data slice in the target data in the process of dividing the target data into a plurality of data slices, and then the metadata generation component 113 can construct a data index corresponding to each data slice based on the group descriptor and the boundary marker, in combination with the batch marker and the sequence marker of the target data, to obtain the data index corresponding to the metadata and the target data.

[0142] In some implementations, in the default case, the statistics axes are outer-layered by each pair of indices of [bs, seqlen] and inner-layered by hidden_dim, so that the start and end of each row segment can be indicated by the length field or EOS marker carried by the metadata generation component 113. For sequences with padding, the metadata generation component 113 can carry a mask bit, which the metadata generation component 113 can ignore the masked elements or fill the padding with values that are invalid for subsequent budgeting (e.g., fill with -inf for max or softmax) in the process of generating metadata. For variable-length sequences, the metadata generation component 113 can define the grouping termination of data segments by length tags or EOS tags in the form of TLV.

[0143] In the full-dimension statistics mode for target data, the comparator tree 330 online maintains the maximum and minimum values on the full hidden_dim dimension, the adder tree 340 online accumulates the sum of the row, and the mean and variance accumulations in the form of Welford can be maintained in parallel; in the statistics mode for dividing the target data into multiple data segments, the hidden_dim dimension is divided into multiple data segments of 32 or 64 or 128 in size by the hardware configuration of the comparator tree 330 or the adder tree 340, and the comparator tree 330 and the adder tree 340 can set programmable “taps” on the corresponding level to lead out the intermediate maximum, minimum, and partial sum of each data segment on the pipeline, while continuing to aggregate to the upper layer to form the full-dimension statistics. Through this “intermediate node visible” structure design, the same statistics pipeline can output both the full-dimension statistics result at the group tail and the block statistics result at the boundary corresponding to the data segment, and the two types of results can be written to the on-chip memory 112 or the off-chip memory 120 at the same time and distinguished by different GroupID or labels.

[0144] To ensure numerical stability, the comparator tree 330 in the present scheme for statistics of minimum / maximum values can adopt NaN robust rules (such as ignoring NaN or propagating according to IEEE754); the adder tree 340 in the process of statistics accumulation and mean / variance accumulation, etc. can adopt high-precision data paths (such as internal FP32 accumulation), and support Kahan / Neumaier compensation to reduce rounding errors. For scenarios that require more rich distribution information, the metadata generation component 113 can enable programmable histogram functions to perform linear or logarithmic scale binning on the data domain, and output the count of each bin at the group tail for subsequent calculation of approximate quantile.

[0145] In the scenario of quantization, the metadata generation component 113 can configure a second multiplier 352 with a programmable fixed-point packer 320 per channel (or per data slice). When the online statistics form min / max (or based on mean / std, quantile threshold) at the end of a certain group (or data slice), the metadata generation component 113 (e.g., with the scaling factor solver 351) can immediately derive the quantization parameters (e.g., scaling factors for FP8) for that group.

[0146] In the “on-the-fly quantization” mode, the target data is scaled by the second multiplier 352 per channel and encoded into INT8 / INT4 / FP8, etc. target format by the packer 320 before being written to the data area in compressed format, while the corresponding quantization parameters and statistics metadata are written to the metadata area in parallel.

[0147] In the “metadata only” mode, the metadata written back to the off-chip memory 120 is in floating-point data structure, but the group quantization parameters and statistics metadata are already available for the subsequent operators to directly read for on-demand quantization or normalization.

[0148] In both modes, the metadata generation component 113 can generate an “completion notification” at the end of a group and ensure the sequential relationship between “target data write completion” and “metadata visibility” through the doorbell or fence mechanism, so as to guarantee that the execution subject of the subsequent processing can obtain the matching metadata in a deterministic timing when reading the target data. To adapt to the high-concurrency and multi-queue out-of-order write SoC form, the metadata generation component 113 can also allocate a corresponding identifier for each data slice and record the start and end addresses or handles of the data area in the metadata, so that the downstream operators can still obtain and use the metadata in an unambiguous manner in a cross-channel, multi-core environment.

[0149] In one numerical configuration example, the calculation unit 110 injects target data to the metadata generation component 113 with a parallelism of 128 ways per shot. The depth of the comparator tree 330 is about log2(128)=7 levels, and the intermediate nodes are introduced at the 4th, 5th, and 6th levels to align the 32 / 64 / 128 granularity block statistics (here, 32 granularity means that a statistical quantity is output for every 32 inputs); the adder tree 340 adopts a pipeline structure with the same level as the comparator tree 330 and completes the group sum in FP32 at the tree root while bypassing the Welford cumulative pair to form the mean and variance. The group descriptor sets the statistics axis to hidden_dim and sets the group length to hidden_dim or its sub-block length (32 / 64 / 128), and can optionally enable histogram to derive distribution information.

[0150] In real-time quantization mode, the min / max obtained by dividing the data slices is used to generate quantization parameters at the end of the corresponding data slices. The second multiplier 352 scales the continuous data according to the data slices, and the packer 320 packs it into INT8 / INT4 (packing in pairs or quads to improve write channel efficiency), so that the data written back to the off-chip memory 120 is in compressed format. If FP8 is enabled, the scaling factor is generated at the granularity of the data slices and the floating-point number is re-encoded into FP8 E4M3 / E5M2.

[0151] This embodiment can form a data stream for each row (i.e., fixed bs and seqlen, along hidden_dim) on [bs, seqlen, hidden_dim]. The end of the row triggers integer-dimensional statistics and metadata writing, while the sub-block boundaries of every 32 / 64 / 128 channels form block statistics and block quantization parameters.

[0152] This scheme allows Softmax to directly read the maximum value of each row for numerically stable subtraction and normalization, RMSNorm or other Norms to directly read the mean or square mean for normalization, and quantization operators to directly read scaling factors of different granularities, significantly reducing repeated readbacks of the original tensor.

[0153] Example device:

[0154] like Figure 4 As shown, electronic device 400 is in the form of a general-purpose electronic device. Components of electronic device 400 may include, but are not limited to, at least one computing unit 110, memory 420, storage device 430, one or more communication units 440, one or more input devices 450, and one or more output devices 460. The computing unit 110 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 420. In a multiprocessor system, multiple computing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 400.

[0155] Electronic device 400 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 400, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 420 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 430 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 400.

[0156] The electronic device 400 can further include additional detachable / non-detachable, volatile / non-volatile storage media. Although not shown in FIG. 4, a disk drive for reading from or writing to a detachable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a detachable, non-volatile optical disk can be provided. In these cases, each drive can be connected to the bus (not shown) by one or more data media interfaces. The memory 420 can include a computer program product 425 having one or more program modules configured to carry out the various methods or acts of the various embodiments of the present disclosure. Figure 4

[0157] The communication unit 440 enables communication with other electronic devices through communication media. Additionally, the functionality of the components of the electronic device 400 can be implemented in a single computing cluster or a plurality of computer machines capable of communicating with one another through a communication connection. As such, the electronic device 400 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0158] The input device 450 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 460 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 400 can also communicate with one or more external devices (not shown), such as a storage device, a display device, etc., one or more devices that enable a user to interact with the electronic device 400, or any devices (e.g., a network card, a modem, etc.) that enable the electronic device 400 to communicate with one or more other electronic devices, through the communication unit 440, as needed. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0159] According to an example implementation of the present disclosure, a computer readable storage medium having computer executable instructions stored thereon is provided, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0160] Various aspects of the disclosure are now described with reference to the drawings. In general, the drawings described below are diagrammatic and schematic representations of actual or conceptual structures and processes, and are not limiting of the scope of the present disclosure. In the drawings, the same reference numerals are used to represent similar or like items. FIG. 1 illustrates an example of a wireless communication system 100, in accordance with aspects of the present disclosure. The wireless communication system 100 includes a number of UEs 120a, 120b, 120c, a RAN 110, and a core network 140. The RAN 110 can include one or more base stations 110a, 110b, 110c, 110d, 110e, 110f, each comprising one or more transceivers for communicating with the UEs 120a, 120b, 120c. The core network 140 can comprise a plurality of network entities for communicating with the base stations 110a, 110b, 110c, 110d, 110e, 110f.​

[0161] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage cycles that change state. The instructions can be executed by one or more processors of a computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process such that the instructions which execute via the one or more processors of the computer or other programmable data processing devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0162] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0163] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0164] The implementations of the disclosure have been described above with the intent to be illustrative rather than limiting. Although being shown and described in connection with certain implementations, it will be appreciated that various modifications and adaptations can occur to those skilled in the art. Therefore, the foregoing description is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. It is intended that the disclosure cover all modifications and adaptations of the foregoing implementations and equivalents thereof. The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting.

Claims

1. A method for data processing, characterized in that, The method comprises: receiving a first instruction to write target data in an on-chip memory of a computing unit to an off-chip memory; generating, via a metadata generation component disposed between the on-chip memory and a direct memory access component, metadata for the target data, the metadata describing one or more statistical values corresponding to at least part of the target data, and storing the metadata in the on-chip memory; writing, via the direct memory access component, the target data to the off-chip memory; in response to receiving a second instruction to execute, with the computing unit, a target processing for the target data, retrieving the metadata from the on-chip memory, the target processing being related to a target statistical value of the one or more statistical values, the target processing comprising at least one of: determining a probability distribution, normalizing, matrix multiplication, convolution operation, the target statistical value comprising at least one of: maximum value, minimum value, sum of squares, mean, variance, low-precision data and corresponding scaling coefficient inverse, high-precision data and corresponding scaling coefficient; and executing, with the computing unit, the target processing for the target data based on the target statistical value in the metadata.

2. The method of claim 1, wherein, The type of the metadata is determined based on at least one subsequent processing to be performed for the target data.

3. The method of claim 1, wherein, The generating, via the metadata generation component between the on-chip memory and the direct memory access component, the metadata for the target data comprises: dividing the target data into a plurality of data slices; and determining the metadata for the target data based on the plurality of data slices.

4. The method of claim 3, wherein, The target data comprises a plurality of unit data corresponding to a preset data dimension, and the dividing the target data into a plurality of data slices comprises: dividing the target data into the plurality of data slices based on the preset data dimension, each data slice corresponding to a predetermined number of groups of unit data.

5. The method of claim 3, wherein, The determining the metadata for the target data based on the plurality of data slices comprises: generating the metadata corresponding to a plurality of granularities based on the plurality of data slices, the granularity indicating a number of data slices used to generate a statistical value.

6. The method of claim 1, wherein, The on-chip memory further maintains the metadata and corresponding data indices, and the retrieving the metadata from the on-chip memory comprises: determining at least one data index corresponding to the target data; and retrieving the metadata corresponding to the at least one data index from the on-chip memory.

7. The method of claim 6, wherein, The at least one data index indicates a batch identification and a sequence identification corresponding to the target data.

8. The method of claim 1, wherein, The method further comprises: writing, via the direct memory access component, the metadata to the off-chip memory.

9. The method of claim 1, wherein, The metadata indicates at least one statistical value of: maximum value, minimum value, accumulated value, mean, variance, sum of squares.

10. The method of claim 1, wherein, The metadata comprises at least one quantization parameter used to quantize the target data, and the writing, via the direct memory access component, the target data to the off-chip memory comprises: performing a quantization process on the target data based on the at least one quantization parameter; and writing, via the direct memory access component, the quantized target data to the off-chip memory.

11. The method of claim 1, wherein, The target data is intermediate data generated by a language model in a pre-population stage.

12. An electronic device, comprising: The electronic device comprises: at least one computing unit; and at least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit, the instructions, when executed by the at least one computing unit, cause the electronic device to perform the method according to any one of claims 1-11.

13. A computer-readable storage medium having stored thereon computer- executable instructions, wherein, The computer-executable instructions are executable by a computing unit to implement the method according to any one of claims 1-11.

14. A computer program product, the computer program product being tangibly stored in a computer storage medium and comprising computer-executable instructions, the computer program product being characterized in that, The computer-executable instructions, when executed by a device, cause the device to perform the method according to any one of claims 1-11.

Images