Data processing method, device and equipment for large language model

Abstract

The present disclosure provides a data processing method, device and equipment for a large language model, which can be applied to the technical field of data processing. The method comprises: in response to a storage instruction, reading an instruction sub-tensor corresponding to the storage instruction from a plurality of data lookup tables, the plurality of sub-tensors being stored in a memory controller based on a floating-point number rule, and an instruction index corresponding to the instruction sub-tensor matching a preset index condition corresponding to the storage instruction; updating initial index bias data of the instruction sub-tensor to obtain index bias data; fusing the instruction sub-tensor and a plurality of sub-tensors in tensor data using an encoder to obtain a target tensor; and sending the target tensor and the index bias data to a target cache unit.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically to a data processing method, apparatus, and device for large language models. Background Technology

[0002] Deploying large language models places extremely high demands on memory capacity and bandwidth. High-bandwidth memory, due to its superior bandwidth and capacity characteristics, has become a key solution for deploying large language models.

[0003] However, the above deployment scheme suffers from random fluctuations in the charge retention time of the storage cells, which can lead to bit flip errors in the stored data.

[0004] To avoid the aforementioned bit-flip errors, related technologies utilize error detection techniques such as error correction codes to identify bit-flip errors. However, this method not only consumes excessive energy but also struggles to directly prevent bit-flip errors, resulting in poor data storage stability. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a data processing method, apparatus and device for large language models.

[0006] According to a first aspect of this disclosure, a data processing method for a large language model is provided, applied to a processor, the processor being communicatively connected to a memory controller, and multiple sub-tensors in tensor data of the large language model being stored in multiple data lookup tables of the memory controller. The method includes: in response to a store instruction, reading an instruction sub-tensor corresponding to the store instruction from the multiple data lookup tables; storing the multiple sub-tensors in the memory controller based on floating-point rules; matching the instruction exponent corresponding to the instruction sub-tensor with a preset exponent condition corresponding to the store instruction; updating the initial exponent bias data of the instruction sub-tensor to obtain exponent bias data; fusing the instruction sub-tensor and the multiple sub-tensors in the tensor data using an encoder to obtain a target tensor; and sending the target tensor and the exponent bias data to a target cache unit.

[0007] According to embodiments of this disclosure, an encoder is used to fuse instruction subtensors and multiple subtensors in tensor data to obtain a target tensor. This includes: obtaining tensor indices corresponding to each of the multiple subtensors, wherein the multiple tensor indices include an instruction index corresponding to the instruction subtensor and a base tensor index corresponding to the base subtensor, and the tensor indices are obtained by performing exponentiation on the subtensors based on preset constants; fusing the instruction index and the base tensor index to obtain a target index corresponding to the base subtensor; determining the target subtensor corresponding to the base subtensor based on the target index; and determining the target tensor based on the multiple target subtensors and the instruction subtensor.

[0008] According to embodiments of this disclosure, fusing the instruction index and the basic tensor index to obtain a target index corresponding to the basic subtensor includes: encoding a preset value to obtain a preset index; determining an adjustment coefficient based on the offset between the instruction index and the preset index; and fusing the adjustment coefficient and the basic tensor index to obtain a target index corresponding to the basic subtensor.

[0009] According to embodiments of this disclosure, updating the initial exponential bias data of the instruction subtensor to obtain exponential bias data includes: fusing the adjustment coefficient and the initial exponential bias data to obtain exponential bias data.

[0010] According to embodiments of this disclosure, multiple sub-tensors are assigned data identifiers, and the multiple sub-tensors are stored in data lookup tables corresponding to the data identifiers. The process of reading the instruction sub-tensor corresponding to the storage instruction from the multiple data lookup tables in response to a storage instruction includes: determining a storage identifier based on a preset index condition corresponding to the storage instruction; performing an identifier index on the data identifiers corresponding to each of the multiple data lookup tables based on the storage identifier to determine a target data identifier corresponding to the storage identifier; and reading the instruction sub-tensor from the data lookup table corresponding to the target data identifier. The data identifier corresponding to the instruction sub-tensor is obtained by performing the following operations using a memory controller: sorting the tensor indices corresponding to each of the multiple sub-tensors to obtain a sorting result; and determining the data identifier corresponding to each of the multiple sub-tensors based on the sorting result. The data identifier of the sub-data represents the sorting position of the instruction index corresponding to the instruction sub-tensor in the sorting result.

[0011] According to embodiments of this disclosure, the method further includes: in response to a read instruction, reading target tensor and exponent bias data from a target cache unit; decoding the target tensor using a decoder to obtain a decoded tensor that matches the tensor data; and sending the decoded tensor and exponent bias data to a memory controller.

[0012] According to embodiments of this disclosure, decoding a target tensor using a decoder to obtain a decoded tensor includes: fusing an instruction sub-tensor and a target sub-tensor to obtain a decoded sub-tensor corresponding to a base sub-tensor; and determining the decoded tensor based on multiple decoded sub-tensors and instruction sub-tensors.

[0013] According to embodiments of this disclosure, fusing instruction subtensors and target subtensors to obtain a decoded subtensor corresponding to the base subtensor includes: determining a decoding exponent corresponding to the base subtensor based on the difference between the adjustment coefficient and the target exponent; and determining the decoded subtensor corresponding to the base subtensor based on the decoding exponent corresponding to the base subtensor.

[0014] A second aspect of this disclosure provides a data processing apparatus for a large language model, comprising: a memory controller for storing a plurality of data lookup tables for storing a plurality of sub-tensors in tensor data of the large language model; and a processor for executing a data processing method for the large language model.

[0015] The target cache unit is used to receive the target tensor and exponential bias data sent by the processor.

[0016] A third aspect of this disclosure provides an electronic device including a data processing apparatus for a large language model.

[0017] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0018] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0019] According to embodiments of this disclosure, by updating the initial exponent bias data of the instruction sub-tensor to obtain exponent bias data, the exponent bias data can be dynamically adjusted for different instruction sub-tensors. Furthermore, by fusing multiple sub-tensors in the instruction sub-tensor and tensor data to obtain the target tensor, the exponent flipping method of the tensor data can be adjusted, reducing the dependence on error correction codes, improving the stability of data storage, and reducing power consumption. Attached Figure Description

[0020] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 This schematically illustrates a structural block diagram of an apparatus for a data processing method for a large language model according to an embodiment of the present disclosure;

[0022] Figure 2 A flowchart illustrating a data processing method for a large language model according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 3 A schematic diagram illustrating the effect before encoding according to an embodiment of the present disclosure is shown;

[0024] Figure 4 A schematic diagram illustrating the effect of encoding according to embodiments of the present disclosure is shown; and

[0025] Figure 5A block diagram schematically illustrates an electronic device suitable for implementing a data processing method for large language models according to embodiments of the present disclosure. Detailed Implementation

[0026] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] With the outstanding performance of Large Language Models (LLMs) in natural language processing and computer vision tasks, their deployment places extremely high demands on memory capacity and bandwidth. High-bandwidth memory (HBM), due to its superior bandwidth and capacity characteristics, has become a key solution for LLM deployment.

[0031] Dynamic Random Access Memory (DRAM) faces serious runtime error problems, with Variable Retention Time (VRT) being the primary source. VRT causes random fluctuations in the charge retention time of memory cells, potentially causing cells with sufficient retention time to occasionally discharge before the refresh cycle, resulting in bit flip errors. These errors are intermittent and unpredictable, representing potential inherent faults that cannot be detected during manufacturing testing.

[0032] To ensure data reliability, HBM employs SECDED (72,64) Hamming code as its error correction code (ECC). This technique divides the 32-byte data block into four independent segments, each of which applies ECC independently, adding a total of 32 extra parity bits, equivalent to a 12.5% ​​increase in data volume.

[0033] In summary, the relevant technologies have the following technical problems:

[0034] Large area overhead: SECDED(72,64) requires additional hardware circuitry to implement encoding and decoding functions, occupying valuable chip area.

[0035] High power consumption: ECC encoding / decoding computation and redundant memory access introduce significant dynamic power consumption.

[0036] High bandwidth pressure: The additional 12.5% ​​of parity bits consume memory capacity and bandwidth resources, placing an extra burden on memory-intensive LLM inference.

[0037] Failure to utilize the inherent fault tolerance of LLM: Modern LLMs have a certain fault tolerance due to over-parameterization, but traditional ECC does not take this characteristic into account and provides indiscriminate protection for all data bits, resulting in a waste of resources.

[0038] Errors in the exponent bit are extremely dangerous: In floating-point representation, errors in the exponent bit (especially the highest bit) can cause drastic changes in the numerical value. In particular, a 0 to 1 flip in the highest bit exponent bit will magnify the value by 2¹. 6 This significantly impacts the accuracy of model inference.

[0039] The embodiments of this disclosure provide a data processing method for large language models. By updating the initial exponent bias data of the instruction sub-tensor, exponent bias data is obtained. The method can dynamically adjust the exponent bias data for different instruction sub-tensors. Furthermore, the method can fuse the instruction sub-tensor and multiple sub-tensors in the tensor data to obtain the target tensor. The method can adjust the exponent flipping mode of the tensor data, reduce the dependence on error correction codes, improve the stability of data storage, and reduce power consumption.

[0040] Figure 1 The diagram illustrates a structural block diagram of a data processing method apparatus for a large language model according to an embodiment of the present disclosure.

[0041] like Figure 1 As shown, the application scenario according to this embodiment may include a processor, a memory controller, and a target cache unit.

[0042] The processor includes an encoder and a decoder, both of which are used to execute data processing methods for large language models.

[0043] The memory controller includes multiple data lookup tables for storing tensor data and sending tensor data to the processor.

[0044] The target cache unit is used to receive the target tensor and exponential bias data sent by the processor.

[0045] The following will be based on Figure 1 The described scenario provides a detailed description of the data processing method for a large language model according to the disclosed embodiments.

[0046] Figure 2 A flowchart illustrating a data processing method for a large language model according to an embodiment of the present disclosure is shown schematically.

[0047] like Figure 2 As shown, the data processing method for a large language model in this embodiment is applied to a processor, which is communicatively connected to a memory controller, and includes operations S210 to S240.

[0048] In operation S210, in response to a store instruction, an instruction sub-tensor corresponding to the store instruction is read from multiple data lookup tables. The multiple sub-tensors are stored in the memory controller based on floating-point rules, and the instruction exponent corresponding to the instruction sub-tensor is matched with the preset exponent condition corresponding to the store instruction.

[0049] In operation S220, the initial exponential bias data of the instruction subtensor is updated to obtain the exponential bias data.

[0050] In operation S230, the encoder is used to fuse the instruction sub-tensor and multiple sub-tensors in the tensor data to obtain the target tensor.

[0051] In operation S240, the target tensor and exponential bias data are sent to the target buffer unit.

[0052] The processor communicates with the memory controller, allowing it to read tensor data from the memory controller's lookup table. This lookup table corresponds to the virtual memory address that is the physical memory address in the memory controller.

[0053] In some embodiments, the memory controller may also be integrated inside the processor and is a component of the processor.

[0054] Multiple sub-tensors in the tensor data used for large language models are stored in multiple data lookup tables of the memory controller. The tensor data can be model parameters of the large language model, or it can be the input and output data of the large language model, etc.

[0055] In response to a store instruction, the processor determines the tensor data that needs to be stored, as indicated by the store instruction, and then indexes multiple data lookup tables to read the tensor data indicated by the store instruction from the memory controller based on the data lookup tables.

[0056] Tensor data includes multiple sub-tensors, each corresponding to different data used in large language models. These sub-tensors can be model parameters of large language models (such as network layer weights or network layer biases).

[0057] Multiple subtensors are stored in the memory controller based on floating-point rules. Specifically, multiple subtensors are converted to floating-point numbers based on floating-point rules and stored in the memory controller in floating-point format.

[0058] The stored instruction includes a preset exponent condition. Based on this preset exponent condition, the processor selects the instruction sub-tensor from a plurality of sub-tensors that matches the preset exponent condition corresponding to the stored instruction. The preset exponent condition can be either a tensor whose instruction exponent is greater than the exponents of other sub-tensors in the tensor data, or a sub-tensor whose tensor exponent falls within a preset exponent range.

[0059] In some embodiments, for tensor exponents larger than other subtensors in the tensor data, i.e., the first exponent bit (the highest bit), the frequency of 1 in the exponent bit is close to 0%. This is because the initialization and regularization techniques used during LLM training limit the numerical range of the parameters.

[0060] During the reading of tensor data and the determination of instruction sub-tensors within the tensor data, the initial exponent bias data of the instruction sub-tensor is simultaneously obtained. The initial exponent bias data is used to perform floating-point conversion on the instruction sub-tensor based on the aforementioned floating-point rules.

[0061] The initial exponent bias data of the instruction subtensor is updated by adjusting the value of the initial exponent bias data to obtain the exponent bias data, so that the exponent bias data matches the instruction exponent of the instruction subtensor.

[0062] The encoder is integrated inside the processor and is one of the components of the processor.

[0063] The encoder is used to fuse the instruction subtensor and multiple subtensors in the tensor data respectively. This can be done by encoding the multiple subtensors using the instruction index of the instruction subtensor to obtain the target tensor. The target tensor includes the instruction subtensor and the adjusted multiple subtensors. The tensor index of the adjusted multiple subtensors changes compared to the original multiple subtensors.

[0064] After obtaining the target tensor and exponential bias data, the target tensor and exponential bias data are sent to the target cache unit, which can be a dynamic random access memory (DRAM).

[0065] After the target cache unit receives the target tensor and exponential bias data, it determines that the storage operation of the tensor data indicated by the storage instruction has been completed.

[0066] According to embodiments of this disclosure, by updating the initial exponent bias data of the instruction sub-tensor to obtain exponent bias data, the exponent bias data can be dynamically adjusted for different instruction sub-tensors. Furthermore, by fusing multiple sub-tensors in the instruction sub-tensor and tensor data to obtain the target tensor, the exponent flipping method of the tensor data can be adjusted, reducing the dependence on error correction codes, improving the stability of data storage, and reducing power consumption.

[0067] According to embodiments of this disclosure, an encoder is used to fuse instruction subtensors and multiple subtensors in tensor data to obtain a target tensor. This includes: obtaining tensor indices corresponding to each of the multiple subtensors, wherein the multiple tensor indices include an instruction index corresponding to the instruction subtensor and a base tensor index corresponding to the base subtensor, and the tensor indices are obtained by performing exponentiation on the subtensors based on preset constants; fusing the instruction index and the base tensor index to obtain a target index corresponding to the base subtensor; determining the target subtensor corresponding to the base subtensor based on the target index; and determining the target tensor based on the multiple target subtensors and the instruction subtensor.

[0068] Tensor exponents are obtained by performing exponentiation on subtensors based on a preset constant, where the preset constant can be the number of times the base is multiplied by itself. In some embodiments, zero-degree exponentiation or fractional arithmetic can also be performed to obtain the tensor exponent.

[0069] Obtain the instruction exponent corresponding to the instruction subtensor. The instruction exponent can be expressed as: And obtaining the fundamental tensor exponent corresponding to the fundamental subtensor can be expressed as: .

[0070] The fusion of instruction exponent and basic tensor exponent can be achieved by performing fusion operations such as adding the instruction exponent and basic tensor exponent, so as to adjust the basic tensor exponent based on the instruction exponent and obtain the target exponent corresponding to the basic subtensor.

[0071] The target subtensor corresponding to the basic subtensor can be determined by performing an inverse exponentiation operation on the target exponent to obtain the target subtensor corresponding to the target exponent, which is the target subtensor corresponding to the basic subtensor.

[0072] After determining the target sub-tensor corresponding to each of the multiple basic sub-tensors, it is used together with the instruction sub-tensor as the target tensor.

[0073] According to embodiments of this disclosure, by fusing the instruction exponent and the basic tensor exponent, the exponent flipping method of the basic tensor exponent can be changed. The changed exponent flipping method has a smaller impact on the target tensor, thus ensuring the stability of the target tensor during storage.

[0074] According to embodiments of this disclosure, fusing the instruction index and the basic tensor index to obtain a target index corresponding to the basic subtensor includes: encoding a preset value to obtain a preset index; determining an adjustment coefficient based on the offset between the instruction index and the preset index; and fusing the adjustment coefficient and the basic tensor index to obtain a target index corresponding to the basic subtensor.

[0075] Encoding the preset value yields the preset index, which can be represented as: .

[0076] The adjustment coefficient is determined based on the offset between the command index and the preset index. This can be achieved by subtracting the command index from the preset index, resulting in the difference between them, which can be expressed as... .

[0077] By fusing the adjustment coefficient and the fundamental tensor exponent, the target exponent corresponding to the fundamental subtensor can be obtained. This can be achieved by adding the adjustment coefficient and the fundamental tensor exponent together, resulting in the sum of the adjustment coefficient and the fundamental tensor exponent, which can be expressed as the target exponent. .

[0078] After encoding, the exponent bit pattern corresponding to the instruction exponent in the tensor data becomes... That is, all exponent bits are 1.

[0079] The fact that the first exponent bit is almost always 0 means that when this bit flips from 0 to 1, the value will be magnified by 2^16 = 65536 times, causing serious consequences; while a flip from 1 to 0 (which rarely happens in practice because this bit is almost 0) will only make the value approach zero, with a smaller impact.

[0080] According to embodiments of this disclosure, by fusing adjustment coefficients and the fundamental tensor exponent, a target exponent corresponding to the fundamental subtensor is obtained. The inherent fault tolerance of large language models is utilized to reduce reliance on error correction codes (ECCs), transforming exponent bit errors from a 0→1 flip that significantly impacts the fundamental subtensor to a 1→0 flip that has a smaller impact.

[0081] According to embodiments of this disclosure, updating the initial exponential bias data of the instruction subtensor to obtain exponential bias data includes: fusing the adjustment coefficient and the initial exponential bias data to obtain exponential bias data.

[0082] By fusing the adjusted coefficients and the initial exponential bias data, the exponential bias data can be obtained. Adding them together, we get the sum of the adjustment coefficient and the initial exponential bias data, which can be expressed as exponential bias data. .

[0083] In some embodiments = ,and then .

[0084] According to embodiments of this disclosure, by fusing adjustment coefficients and initial exponential bias data, it is possible to maximize the use of unused exponential space and achieve optimal error resistance for different tensors.

[0085] According to embodiments of this disclosure, multiple sub-tensors are assigned data identifiers, and the multiple sub-tensors are stored in data lookup tables corresponding to the data identifiers. The process of reading the instruction sub-tensor corresponding to the storage instruction from the multiple data lookup tables in response to a storage instruction includes: determining a storage identifier based on a preset index condition corresponding to the storage instruction; performing an identifier index on the data identifiers corresponding to each of the multiple data lookup tables based on the storage identifier to determine a target data identifier corresponding to the storage identifier; and reading the instruction sub-tensor from the data lookup table corresponding to the target data identifier. The data identifier corresponding to the instruction sub-tensor is obtained by performing the following operations using a memory controller: sorting the tensor indices corresponding to each of the multiple sub-tensors to obtain a sorting result; and determining the data identifier corresponding to each of the multiple sub-tensors based on the sorting result. The data identifier of the sub-data represents the sorting position of the instruction index corresponding to the instruction sub-tensor in the sorting result.

[0086] The data identifier is matched with the tensor exponent of the subtensor. Specifically, after calculating the tensor exponents of multiple subtensors, the tensor exponents of the multiple subtensors are sorted, which can be in descending order, to obtain the sorted results of the tensor exponents of the multiple subtensors.

[0087] After obtaining the sorting results, the sorting position of the tensor exponent of each of the multiple sub-tensors in the sorting results is determined. Based on the sorting position of each of the multiple sub-tensors, data identifiers are added to the multiple sub-tensors, so that the sub-tensors can be identified from the data lookup table based on the data identifiers, and the sub-tensors corresponding to the tensor exponents indicated by the preset exponent conditions can be selected.

[0088] Based on the preset index conditions corresponding to the storage instruction, the range of tensor index indicated by the preset index conditions is determined, and the storage identifier is determined based on the range of tensor index.

[0089] An identifier index is performed on the data identifiers corresponding to each of the multiple data lookup tables. From the data identifiers corresponding to each of the multiple data lookup tables, the data identifier that matches the storage identifier is determined and used as the target data identifier.

[0090] After determining the target data identifier, the target data lookup table corresponding to the target data identifier is selected from multiple data lookup tables, and a sub-tensor is read from the target data lookup table and used as the instruction sub-tensor.

[0091] According to embodiments of this disclosure, by adding data identifiers to multiple sub-tensors based on their respective tensor exponents, instruction sub-tensors that match the preset exponent conditions corresponding to the storage instruction can be quickly read in response to a storage instruction, thereby improving storage efficiency.

[0092] According to embodiments of this disclosure, the method further includes: in response to a read instruction, reading target tensor and exponent bias data from a target cache unit; decoding the target tensor using a decoder to obtain a decoded tensor that matches the tensor data; and sending the decoded tensor and exponent bias data to a memory controller.

[0093] In response to a read instruction, the processor reads the target tensor and exponential bias data corresponding to the read instruction from the target cache unit.

[0094] After the processor reads the target tensor and exponential bias data, it uses a decoder to decode the target tensor. This can be done by restoring the target tensor to its pre-encoding state to obtain a decoded tensor that matches the tensor data.

[0095] After obtaining the decoded tensor that matches the tensor data, the decoded tensor and exponent bias data are sent to the memory controller. After the memory controller receives the decoded tensor and exponent bias data, it restores multiple sub-tensors in the decoded tensor to floating-point format based on the decoded tensor and exponent bias data.

[0096] Among them, the floating-point number of the subtensor Fully represented as , 0 <= < 1, where, This is the sign bit (1 bit), where 0 represents a positive number and 1 represents a negative number; The exponent bits (5 bits) of the decoded tensor indicate the size of the binary exponent; The last digit (10 digits) represents the decimal part. This is exponentially biased data.

[0097] According to embodiments of this disclosure, by reading target tensor and exponential bias data from the target cache unit and decoding the target tensor using a decoder, the tensor data stored in the target cache unit can be restored, thereby avoiding data errors when sending decoded tensor and exponential bias data to the memory controller.

[0098] According to embodiments of this disclosure, decoding a target tensor using a decoder to obtain a decoded tensor includes: fusing an instruction sub-tensor and a target sub-tensor to obtain a decoded sub-tensor corresponding to a base sub-tensor; and determining the decoded tensor based on multiple decoded sub-tensors and instruction sub-tensors.

[0099] The fusion of instruction subtensors and target subtensors can be achieved by fusing the instruction exponent of the instruction subtensor and the target exponent of the target subtensor, thereby adjusting the target exponent based on the instruction exponent and restoring the target exponent to its state before encoding, thus obtaining the decoded subtensor corresponding to the base subtensor.

[0100] After obtaining multiple decoded sub-tensors, the multiple decoded sub-tensors and the instruction sub-tensor are used together as the decoded tensor.

[0101] According to embodiments of this disclosure, by fusing the instruction sub-tensor and the target sub-tensor, the target sub-tensor can be decoded based on the instruction sub-tensor, and the state of the target sub-tensor before encoding can be accurately restored.

[0102] According to embodiments of this disclosure, fusing instruction subtensors and target subtensors to obtain a decoded subtensor corresponding to the base subtensor includes: determining a decoding exponent corresponding to the base subtensor based on the difference between the adjustment coefficient and the target exponent; and determining the decoded subtensor corresponding to the base subtensor based on the decoding exponent corresponding to the base subtensor.

[0103] The decoding exponent corresponding to the basic subtensor is determined based on the difference between the adjustment coefficient and the target exponent. This can be achieved by calculating the difference between the adjustment coefficient and the target exponent, which can then be expressed as the decoding exponent corresponding to the basic subtensor. .

[0104] After obtaining the decoding exponent corresponding to the basic subtensor, perform inverse exponentiation based on the decoding exponent to obtain the decoding subtensor corresponding to the basic subtensor.

[0105] A second aspect of this disclosure provides a data processing apparatus for a large language model, comprising: a memory controller for storing multiple data lookup tables for storing multiple sub-tensors in tensor data of the large language model; a processor for executing a data processing method for the large language model; and a target cache unit for receiving target tensors and exponential bias data sent by the processor.

[0106] The processor, used in the above operations S210 to S240, specifically: in response to a store instruction, reads the instruction sub-tensor corresponding to the store instruction from multiple data lookup tables; the multiple sub-tensors are stored in the memory controller based on floating-point rules; the instruction exponent corresponding to the instruction sub-tensor is matched with the preset exponent condition corresponding to the store instruction; the initial exponent bias data of the instruction sub-tensor is updated to obtain exponent bias data; the encoder is used to fuse the instruction sub-tensor and the multiple sub-tensors in the tensor data respectively to obtain the target tensor; and the target tensor and exponent bias data are sent to the target cache unit.

[0107] The processor includes a decoder and an encoder.

[0108] The encoder is used to fuse the instruction exponent and the basic tensor exponent to obtain the target exponent corresponding to the basic subtensor; based on the target exponent corresponding to the basic subtensor, the target subtensor corresponding to the basic subtensor is determined; based on multiple target subtensors and instruction subtensors, the target tensor is determined.

[0109] When the processor responds to a store instruction, it reads the target tensor and exponent bias data from the target cache unit; decodes the target tensor using a decoder to obtain a decoded tensor that matches the tensor data; and sends the decoded tensor and exponent bias data to the memory controller.

[0110] The decoder is used to fuse the instruction subtensor and the target subtensor to obtain the decoded subtensor corresponding to the base subtensor; and to determine the decoded tensor based on multiple decoded subtensors and instruction subtensors.

[0111] The data processing device for large language models disclosed herein has been experimentally verified based on multiple large language models, the configurations of which are shown in Table 1.

[0112]

[0113] Figure 3 A schematic diagram illustrating the effect before encoding according to an embodiment of the present disclosure is shown; Figure 4 A schematic diagram illustrating the effect of encoding according to an embodiment of the present disclosure is shown.

[0114] Before and after applying exponential recoding to the tensor data of large language models of types A, B, C, and D, the probability of the five exponent bits being 1 is as follows: Figure 3 and Figure 4 As shown, blue represents large language models of type A, pink represents large language models of type B, yellow represents large language models of type C, and green represents large language models of type D.

[0115] Figure 3 Before encoding, the frequency of the first exponent bit being 1 (bit position 14) is 0. After encoding, the frequency of the first exponent bit being 1 is 100, ensuring the effectiveness of the exponent encoding.

[0116] The perplexity of a large language model of type B1 was evaluated, and the experimental results are shown in Table 2.

[0117]

[0118] At the error injection level is When the error of the first exponent E[4] is not protected, the perplexity is infinite (inf). However, after applying the data processing device for large language models disclosed herein, the perplexity is maintained at 29.94, which achieves a very good protection effect.

[0119] Error injection experiments were conducted on a large language model of type A to evaluate the accuracy of sentiment classification. The results are shown in Table 3.

[0120]

[0121] Where L represents Location, W represents Weight, and A represents Activation.

[0122] Of the three models, "activation" is more sensitive to errors than "weights". When applying the data processing apparatus for large language models disclosed herein, the error injection level is lower than... The model can maintain its inference performance at all times.

[0123] Error injection experiments were conducted on a large language model of type B, and the results are shown in Table 4.

[0124]

[0125] At the error injection level is When the error occurs only in the "weights", all four large language models of type B can maintain normal performance; however, with the same amount of error injection, if the error occurs only in the "activations", only two of the larger models can maintain normal performance.

[0126] Error injection experiments were conducted on a large language model of type C, and the results are shown in Table 5.

[0127]

[0128] At the error injection level is When the error occurs only in the "weights", the large language model of type C3 maintains normal performance; however, with the same amount of error injection, when the error occurs only in the "activations", all three models have a perplexity increase of about 1.

[0129] Error injection experiments were conducted on a large language model of type D, and the results are shown in Table 6.

[0130]

[0131] At the error injection level is When errors occur only in "weights", large language models of type D2 maintain normal performance; however, with the same amount of error injection, when errors occur only in "activations", the perplexity of both models increases more.

[0132] A quantization experiment was conducted on a large language model of type A, and the results are shown in Table 7.

[0133]

[0134] When the error injection level is lower The model can maintain its inference performance at all times.

[0135] A quantization experiment was conducted on a large language model of type B, and the results are shown in Table 8.

[0136]

[0137] When the error injection level is lower than The model can maintain its inference performance at all times.

[0138] A quantization experiment was conducted on a large language model of type D, and the results are shown in Table 9.

[0139]

[0140] If the error only occurs in the "weight", its error injection amount is Even at this time, it can still maintain normal performance.

[0141] The data processing apparatus for large language models disclosed herein is compared with a comparative example, and the results are shown in Table 10.

[0142]

[0143] The data processing apparatus for large language models disclosed herein achieves 1.2 times lower area overhead and 2.7 times higher energy efficiency.

[0144] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method for large language models according to embodiments of the present disclosure.

[0145] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0146] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0147] According to embodiments of this disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0148] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0149] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0150] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the data processing method for large language models provided in embodiments of this disclosure.

[0151] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0152] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0153] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0156] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0157] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data processing method for large language models, characterized in that, Applied to a processor, which is communicatively connected to a memory controller, the method stores multiple sub-tensors in the tensor data of a large language model in multiple data lookup tables of the memory controller; the method includes: In response to a storage instruction, an instruction sub-tensor corresponding to the storage instruction is read from a plurality of data lookup tables. The plurality of sub-tensors are stored in the memory controller based on floating-point rules, and the instruction exponent corresponding to the instruction sub-tensor is matched with a preset exponent condition corresponding to the storage instruction. The initial exponential bias data of the instruction subtensor is updated to obtain the exponential bias data; The target tensor is obtained by fusing the instruction sub-tensor and multiple sub-tensors in the tensor data using an encoder. The target tensor and the exponential bias data are sent to the target cache unit.

2. The method according to claim 1, characterized in that, The step of fusing the instruction sub-tensor and multiple sub-tensors in the tensor data using an encoder to obtain the target tensor includes: Obtain the tensor exponent corresponding to each of the plurality of sub-tensors, wherein the plurality of tensor exponents include an instruction exponent corresponding to the instruction sub-tensor and a base tensor exponent corresponding to the base sub-tensor, and the tensor exponent is obtained by performing an exponentiation operation on the sub-tensor based on a preset constant; By fusing the instruction index and the basic tensor index, a target index corresponding to the basic subtensor is obtained; Based on the target index corresponding to the basic subtensor, determine the target subtensor corresponding to the basic subtensor; The target tensor is determined based on the plurality of target sub-tensors and the instruction sub-tensor.

3. The method according to claim 2, characterized in that, The process of fusing the instruction index and the basic tensor index to obtain the target index corresponding to the basic subtensor includes: Encode the preset value to obtain the preset index; The adjustment coefficient is determined based on the offset between the command index and the preset index; By combining the adjustment coefficient and the basic tensor exponent, a target exponent corresponding to the basic subtensor is obtained.

4. The method according to claim 3, characterized in that, The step of updating the initial exponential bias data of the instruction subtensor to obtain exponential bias data includes: The index bias data is obtained by combining the adjustment coefficient and the initial index bias data.

5. The method according to claim 4, characterized in that, The multiple sub-tensors are assigned data identifiers, and the multiple sub-tensors are stored in a data lookup table corresponding to the data identifiers; The step of reading the instruction sub-tensor corresponding to the storage instruction from a plurality of data lookup tables in response to a storage instruction includes: The storage identifier is determined based on the preset index conditions corresponding to the storage instruction; Based on the storage identifier, an identifier index is performed on the data identifiers corresponding to each of the multiple data lookup tables to determine the target data identifier corresponding to the storage identifier; Read the instruction sub-tensor from the data lookup table corresponding to the target data identifier; The data identifier corresponding to the instruction sub-tensor is obtained by performing the following operation using the memory controller: The tensor exponents corresponding to each of the multiple sub-tensors are sorted to obtain the sorting result; Based on the sorting result, a data identifier corresponding to each of the multiple sub-tensors is determined. The data identifier of the sub-data represents the sorting position of the instruction index corresponding to the instruction sub-tensor in the sorting result.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: In response to a read command, the target tensor and the exponential bias data are read from the target cache unit; The target tensor is decoded using a decoder to obtain a decoded tensor that matches the tensor data; The decoded tensor and the exponential bias data are sent to the memory controller.

7. The method according to claim 6, characterized in that, The step of decoding the target tensor using a decoder to obtain a decoded tensor includes: The instruction sub-tensor and the target sub-tensor are fused to obtain a decoded sub-tensor corresponding to the base sub-tensor; The decoding tensor is determined based on the plurality of decoding sub-tensors and the instruction sub-tensor.

8. The method according to claim 7, characterized in that, The process of fusing the instruction sub-tensor and the target sub-tensor to obtain the decoded sub-tensor corresponding to the base sub-tensor includes: The decoding exponent corresponding to the basic subtensor is determined based on the difference between the adjustment coefficient and the target exponent; The decoded subtensor corresponding to the basic subtensor is determined based on the decoding exponent corresponding to the basic subtensor.

9. A data processing apparatus for a large language model, comprising: A memory controller is used to store multiple data lookup tables, which are used to store multiple sub-tensors in the tensor data of a large language model; A processor for performing the method according to any one of claims 1 to 8. A target cache unit is used to receive the target tensor and the exponential bias data sent by the processor.

10. An electronic device comprising the data processing apparatus for a large language model as described in claim 9.

Images