Data processing method and apparatus, chip, server, and system

Abstract

The present application relates to the technical field of chips, and discloses a data processing method and apparatus, a chip, a server, and a system. The method relates to a means of processing at least one data set, the at least one data set comprising multiple types of data, and different types of data corresponding to different precisions. Data in the data set is classified, and different types of data correspond to different precisions; therefore, when the data set is stored, it is not necessary to perform compression according to a single precision, thereby increasing the compression rate, and saving chip resources.

Description

Data processing methods, devices, chips, servers and systems

[0001] This application claims priority to Chinese Patent Application No. 202410947560.1, filed on July 15, 2024, entitled "Data Processing Method, Apparatus, Chip, Server and System", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of chip technology, and in particular to a data processing method, apparatus, chip, server and system. Background Technology

[0003] With the rapid development of artificial intelligence (AI) and deep learning technologies, the scale of AI models is constantly increasing, and the number of parameters is also growing exponentially. The memory capacity and bandwidth of chips have become the main bottlenecks in the model inference process.

[0004] In related technologies, taking large language model (LLM) as an example, during the training phase, the weight matrix of LLM is usually expressed in high precision (such as FP16). In order to reduce the memory usage of the chip, at the end of training, a quantization algorithm can be used to express the weight matrix in low precision (such as FP8) and store it in the chip memory. During the inference phase, an inverse quantization algorithm is used to restore the low precision weight matrix to the high precision weight matrix to ensure the accuracy of the inference results.

[0005] However, in the above methods, since the quantized model usually needs to maintain a similar level of accuracy to the original model (allowing for a small amount of loss), it is difficult to achieve a high compression rate under this constraint. Summary of the Invention

[0006] This application provides a data processing method, apparatus, chip, server, and system that can improve the compression rate during data storage and save chip resources.

[0007] Firstly, this application provides a data processing method, the method comprising:

[0008] Obtain at least one dataset, which includes multiple data types, with different data types corresponding to different precision.

[0009] Process at least one data set.

[0010] The above method can be applied to model compression storage and model inference scenarios. This application does not limit the type of model or the application field. This method relates to a way of processing at least one dataset, wherein the at least one dataset includes multiple types of data, with different types of data corresponding to different precisions. Because the data in the dataset is classified, different types of data correspond to different precisions. Therefore, when storing the dataset, it is not necessary to compress it according to a uniform precision, thereby improving the compression ratio and saving chip resources.

[0011] In some embodiments, a type is used to indicate the importance of data. Multiple data types include first data of a first type and second data of a second type, wherein the importance indicated by the first type is lower than that indicated by the second type, and the precision of the first data is lower than that of the second data. In this application, importance refers to the impact of data on the execution result of an operator, or it can be the impact on the model's precision. Illustratively, the greater the impact of a piece of data on the execution result of an operator, the higher the importance of that data. Correspondingly, using higher precision to express that data can ensure the accuracy of the model.

[0012] In some embodiments, the first data and the second data are different data in the same data set; or, the first data and the second data belong to different data sets. That is, this application does not limit the granularity of data partitioning for mixed-precision storage. For example, taking a matrix as an example, different matrices can be stored with different precisions, or different matrix elements in the same matrix can be stored with different precisions, thereby improving the compression rate in a targeted manner.

[0013] In some embodiments, the bit width of the first data and the bit width of the second data are different; or, the bit width of the first data and the bit width of the second data are the same, but the numerical allocation ratio of the first data is different from that of the second data.

[0014] In some embodiments, the proportion of the first type of data in at least one dataset is greater than the proportion of the second type of data in at least one dataset. That is, the proportion of low-precision data is greater than the proportion of high-precision data; for example, low-precision data accounts for 80%, and high-precision data accounts for 20%. Thus, although highly important data is represented using high precision, its proportion is small and does not affect the overall compression ratio. Meanwhile, using low precision to represent most of the less important data effectively improves the compression ratio. Furthermore, because highly important data is represented using high precision, the accuracy of the overall model inference can be ensured.

[0015] In some embodiments, the method further includes any of the following:

[0016] The third data is precision-converted according to the first scaling factor corresponding to the first type to obtain the first data, and the fourth data is precision-converted according to the second scaling factor corresponding to the second type to obtain the second data; or,

[0017] The precision of the third data is converted according to the first bit length corresponding to the first type to obtain the first data. The precision of the fourth data is converted according to the second bit length corresponding to the second type to obtain the second data. The first bit length is less than the second bit length.

[0018] The above methods provide a variety of ways to perform precision conversion on different types of data. In practical applications, these methods can be set according to requirements, and this application does not impose any limitations on them.

[0019] In some embodiments, processing at least one data set includes:

[0020] Perform precision transformation on at least one dataset to obtain a target dataset, where the precision of the target dataset is the target precision.

[0021] In some embodiments, at least one dataset is subjected to precision transformation to obtain a target dataset, including any of the following:

[0022] Convert the precision of the first and second data sets to the target precision to obtain the target data set, where the precision of the first and second data sets differs; or,

[0023] The precision of each data point in the third dataset is converted to the target precision to obtain the target dataset. The third dataset includes multiple types of data, with different types of data corresponding to different precisions.

[0024] Since at least one dataset includes multiple types of data, and different types of data correspond to different precisions, operators usually need to meet the target precision when they are executed. Therefore, the above method can ensure the accuracy of the operator execution results.

[0025] In some embodiments, at least one dataset includes the weight matrix of the matrix multiplication operator in a large language model (LLM).

[0026] In some embodiments, at least one dataset includes the key matrix and value matrix of the attention operator in a large language model (LLM).

[0027] Secondly, this application provides a data processing apparatus, which includes at least one functional module for implementing the data processing method provided by the first aspect or any possible implementation thereof.

[0028] Thirdly, this application provides a chip including a processor and a memory, wherein the processor is configured to execute instructions stored in the memory to cause the chip to perform a data processing method as provided in the first aspect or any possible implementation thereof.

[0029] Fourthly, this application provides a server that includes the chip described in the third aspect above.

[0030] Fifthly, this application provides a data processing system, which includes a host and an acceleration chip. The host is used to control the acceleration chip, and the acceleration chip is used to execute the data processing method provided by the first aspect or any possible implementation thereof.

[0031] Sixthly, this application provides a computer-readable storage medium for storing at least one piece of program code, which, when executed by a chip, causes the chip to implement the data processing method provided by the first aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0032] In a seventh aspect, this application provides a computer program product that, when run on a chip, causes the chip to implement the data processing method provided by the first aspect or any possible implementation thereof. The computer program product can be a software installation package; when the aforementioned data processing method needs to be implemented, the computer program product can be downloaded and executed on the acceleration chip. Attached Figure Description

[0033] Figure 1 is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0034] Figure 2 is a schematic diagram of the structure of a chip provided in an embodiment of this application;

[0035] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0036] Figure 4 is a schematic diagram illustrating the principle of a data processing method provided in an embodiment of this application;

[0037] Figure 5 is a schematic diagram of the chip architecture provided in an embodiment of this application;

[0038] Figure 6 is a flowchart of a data processing method provided in an embodiment of this application;

[0039] Figure 7 is a schematic diagram of a matrix compressed storage provided in an embodiment of this application;

[0040] Figure 8 is a flowchart of another data processing method provided in an embodiment of this application;

[0041] Figure 9 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the matrices and data formats involved in this application were obtained under fully authorized conditions.

[0043] To facilitate understanding, the key terms and concepts involved in this application will be explained below.

[0044] Artificial intelligence (AI) models are a class of mathematical algorithm models that use machine learning concepts to solve practical problems. Typically, AI models include a large number of parameters and calculation formulas (or calculation rules).

[0045] Acceleration chips, also known as accelerators, acceleration devices, or accelerator cards, are a type of specialized hardware device or computer system designed to accelerate computation processes in AI and HPC scenarios. In the embodiments of this application, acceleration chips may be, for example, graphics processing units (GPUs), neural network processing units (XPUs), intelligent processing units (IPUs), tensor processing units (TPUs), domain-specific architecture (DSA) chips, and so on, but are not limited to these.

[0046] An operator (OP) is a computational unit or function that runs on a computing device. In the field of deep learning, neural network layers and even the entire model are composed of operators, which correspond to the computational logic within the neural network layers. For example, a convolutional layer is an operator; the weight summation process in a fully-connected layer (FC layer) is also an operator.

[0047] The attention operator is the core operator of the self-attention layer in the Transformer model. The core idea of ​​the self-attention layer is to make the machine notice the correlations between different parts of the input. Illustratively, for any vector in the input sequence of the self-attention layer (such as words, letters, image features, etc.), a linear transformation is performed on the vector to obtain the corresponding query (Q) matrix, key (K) matrix, and value (V) matrix, which are also the input matrices Q, K, and V of the attention operator. By executing the attention operator, the input matrices are processed to obtain the corresponding output (O) matrix, which represents the correlations between different parts of the input sequence.

[0048] Data format refers to the way data is stored and represented in a computer, such as integers, floating-point numbers (FP), characters, and strings. For example, integers include int8, int16, int32, int64, uint8, uint16, uint32, and uint64, while floating-point numbers include single-precision floating-point numbers (FP32), double-precision floating-point numbers (FP64), half-precision floating-point numbers (FP16), and lower-precision floating-point numbers such as FP8. Illustratively, different data formats correspond to different precisions of data. For example, FP32 can represent approximately 7 significant digits, while FP64 can represent approximately 16 significant digits. Higher precision allows for a wider range of represented values ​​and more accurate representation, but it also consumes more storage space and computing resources.

[0049] Hybrid bit storage, in this application, refers to storing different types of data in a dataset using different bit depths. For example, taking the weight matrix of a matrix multiplication operator in an AI model as an example, for this weight matrix, the matrix elements that have a greater impact on the operator's execution result are stored using more bits (or long bits), such as 8 bits, while the matrix elements that have a smaller impact on the operator's execution result are stored using fewer bits (or short bits), such as 4 bits. It should be understood that for the same data, the precision when storing with long bits is higher than the precision when storing with short bits. Illustratively, if the bit depth is 3, then 3 bits can produce 2^32 binary combinations. 3 = 8 types, which can be regarded as 8 symbols, each symbol representing a value. If the number of bits is 5, then 5 bits can produce 2^36 binary combinations. 5 =32 types, which can be regarded as 32 symbols, each symbol representing a numerical value, which will not be elaborated further. It should be understood that the specific numerical value corresponding to different symbols can be determined based on spatial distribution, probability distribution, and clustering algorithms. For example, taking a bit depth of 5 as an example, if the numerical range is 0-1000, this range can be divided into 32 intervals on average, where 0-31 is represented by 00000, 32-63 by 00001, and so on, up to 968 by 11111. It should be noted that this is only an example. In practical applications, it may be necessary to optimize the interval division and symbol allocation according to the specific distribution of data and application requirements to achieve a better balance between compression effect and data accuracy.

[0050] The application scenarios and implementation environment of this application are described below.

[0051] This application can be applied to model compression and storage, and model inference scenarios. It does not limit the type of model or the application field. Model compression aims to reduce the storage space occupied by the model while maintaining its performance and accuracy as much as possible. Common model compression methods include parameter quantization, which converts the model parameters from high precision (e.g., FP32) to low precision (e.g., int8) to reduce storage requirements. Taking the Large Language Model (LLM) as an example, during the training phase, the LLM weight matrix is ​​usually expressed in high precision (e.g., FP16). To reduce chip memory usage, at the end of training, a quantization algorithm can be used to express the weight matrix in low precision (e.g., FP8) and store it in the chip's memory. During the inference phase, a dequantization algorithm is used to restore the low-precision weight matrix to a high-precision weight matrix to ensure the accuracy of the inference results. However, since the quantized model usually needs to maintain a similar accuracy to the original model (allowing for a small loss), it is difficult to achieve a high compression ratio under this constraint.

[0052] Based on this, this application provides a model compression storage method. By distinguishing relevant model data according to importance, data of different importance are compressed with different precisions and stored in the chip memory. In this way, the model compression rate is maximized while ensuring model accuracy. Moreover, during model inference, when the chip executes relevant operators of the model, it obtains relevant operator data. Since data of different importance corresponds to different precisions, the chip can convert the precision of this data to the precision required to execute the operator before executing the operator, thereby obtaining the execution result of the operator. The chip involved in this application can be applied to various electronic devices with model inference functions. Schematic, referring to Figure 1, Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of this application. As shown in Figure 1, the chip 100 involved in this application can be applied to various electronic devices 200 with model inference functions.

[0053] In some embodiments, the electronic device 200 is a smart terminal device, such as a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) or virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA) and other mobile terminal devices.

[0054] In other embodiments, the electronic device 200 is a server, such as an independent physical server, a server cluster composed of multiple physical servers, or a distributed file system, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc., and this application is not limited thereto.

[0055] Schematic illustration, taking electronic device 200 as a server as an example, this server is equipped with a data processing system, which includes a host and an acceleration chip. The host controls the acceleration chip. For example, the host and the acceleration chip are connected via a peripheral component interconnect express (PCIe) link, and data exchange occurs between them through the PCIe link. For instance, in response to a model's inference request, the host sends execution instructions for each operator in the model to the acceleration chip via the PCIe link to achieve model inference. This process can also be understood as loading the model's inference task into the acceleration chip for execution. Furthermore, the number of hosts can be one or more, and this application does not limit this. The acceleration chip provides computing power for the model's inference task to accelerate the model inference process. For example, the acceleration chip can be a GPU, XPU, IPU, TPU, DSA chip, etc., and this application is not limited to these. Furthermore, the number of acceleration chips can be one or more, and this application does not limit this.

[0056] The electronic device 200 involved in this application is capable of accessing a wired or wireless network. Indicatively, the wireless or wired network uses standard communication technologies and / or protocols. The network includes, but is not limited to, data center networks, storage area networks (SANs), local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some implementations, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or part of the link. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0057] The structure of chip 100 will be described below with reference to Figure 2.

[0058] Figure 2 is a schematic diagram of a chip structure provided in an embodiment of this application. As shown in Figure 2, the chip 100 includes a memory 101, a processor 102, a communication interface 103, and a bus 104. The memory 101, processor 102, and communication interface 103 are interconnected via the bus 104.

[0059] Memory 101 includes, but is not limited to, high bandwidth memory (HBM), double data rate memory (DDR), static random-access memory (SRAM), or other types of dynamic storage devices capable of storing information and instructions, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer. Illustratively, memory 101 is used to store at least a piece of program code. When the program code stored in memory 101 is executed by processor 102, processor 102 performs the steps executed by the chip in the following method embodiments.

[0060] The processor 102 may be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit used to control the execution of the program of the present application. The processor 102 may be a single-core processor or a multi-core processor. The number of processors 102 may be one or more.

[0061] The communication interface 103 uses a transceiver module, such as a transceiver, to enable communication between the chip 100 and other devices or communication networks. For example, data can be acquired through the communication interface 103.

[0062] The memory 101 and the processor 102 can be set separately or integrated together.

[0063] Bus 104 may include a pathway for transmitting information between various components of chip 100 (e.g., memory 101, processor 102, communication interface 103).

[0064] It should be noted that Figure 2 above is only a hardware structure diagram of the chip 100 that can be configured as described above, provided by this application. In some embodiments, the chip 100 may also include other components to achieve more functions. For example, the chip 100 may also include a task scheduler (TS), a controller, etc. This application is not limited to this.

[0065] In addition, this application also provides an electronic device, as shown in FIG3, which is a schematic diagram of the structure of an electronic device provided in this application embodiment. As shown in FIG3, the electronic device 200 includes a chip 100 and an interface 300. The chip 100 is, for example, a CPU, GPU, XPU, IPU, TPU, DSA chip, etc., and the interface 300 is used to interact with external devices of the electronic device 200 by means of instructions. The chip 100 is used to implement the data processing method provided in the following method embodiment.

[0066] The data processing methods provided in this application are described below.

[0067] To facilitate understanding, the principle of the technical solution provided in this application will be introduced below with reference to Figure 4, taking the Large Language Model (LLM) as an example. Figure 4 is a schematic diagram of the principle of a data processing method provided in an embodiment of this application. As shown in Figure 4, for LLM, the weight matrix and the key matrix and value matrix involved in the attention operator often occupy most of the chip's memory space, and the amount of data increases with the expansion of the model size. Analysis of the weight matrix shows that the distribution of each matrix element in the weight matrix has a concentrated characteristic. Most weights are distributed in a small numerical range, while weights with large exponent values ​​account for a small proportion. Furthermore, the contribution of the weight matrix to the model accuracy is uneven, with a small number of weights contributing most of the accuracy. Analysis of the key matrix and value matrix of the attention operator shows that during the execution of the attention operator, its calculation formula is softmax(Q×K)×V. After the softmax normalization operation, a few matrix elements in the key matrix and value matrix will have a significant impact on the operator execution result, while most matrix elements have a limited impact on the operator execution result.

[0068] Based on this, this application provides a method for storing model-related data based on mixed precision. The offline compression process includes two steps: (1) Importance assessment, which refers to distinguishing the importance of data based on its impact on model precision. For example, based on the impact of each matrix element in the weight matrix on model precision, the matrix elements in the weight matrix are distinguished to obtain multiple types of matrix elements. Different types of matrix elements correspond to different levels of importance. Alternatively, based on the distribution patterns of the key and value matrices generated during historical inference, the impact of each matrix element in these matrices on model precision is determined, and then the matrix elements in these matrices are distinguished to obtain multiple types of matrix elements. Different types of matrix elements correspond to different levels of importance. (2) Data precision conversion, which refers to converting the precision of data based on different levels of importance. For example, taking the importance of data as divided into important and unimportant categories, important data can be expressed using high precision (e.g., FP8), and unimportant data can be expressed using low precision (e.g., FP4), thus achieving mixed precision storage. It should be noted that high precision and low precision here are relative, and their value range can be set according to the actual application requirements. This application does not limit this.

[0069] Next, during model inference, since the model-related data stored in the chip memory is stored in mixed-precision format, the chip can perform precision conversion on the obtained matrix based on the precision required by the operator when it obtains the operator's relevance matrix. This process can also be called decompression. Furthermore, because the key and value matrices are repeatedly used to calculate context relevance when executing the attention operator during LLM inference, the chip can compress and store the key and value matrices during inference. That is, it can use a mixed-precision storage method similar to the offline compression process, distinguishing the importance of each matrix element in the key and value matrices and storing them in the chip memory according to different precisions.

[0070] Furthermore, the aforementioned compression and decompression processes can be implemented using software algorithms or accelerated by adding hardware modules to the chip. For example, referring to Figure 5, which is a schematic diagram of a chip architecture provided in an embodiment of this application, the chip includes a first storage unit, a decompression unit, a second storage unit, a calculation unit, and a compression unit. The first storage unit stores the relevant matrices of the model, such as high-bandwidth memory (HBM). The decompression unit decompresses the acquired matrices, or converts the precision of the acquired matrices to the precision required by the calculation unit; the decompression unit is, for example, a dedicated hardware circuit. The second storage unit caches the matrices processed by the decompression unit, such as SRAM or DRAM. The calculation unit executes corresponding operators based on the precision-converted matrices to obtain the execution results of the operators. The compression unit compresses the execution results of the operators, such as the key and value matrices, or stores these matrices with mixed precision; the compression unit is, for example, a dedicated hardware circuit. It should be understood that the aforementioned decompression unit and compression unit are hardware modules added to the chip to accelerate the aforementioned compression and decompression processes, thereby improving the overall data processing efficiency. In this scenario, the decompression unit and compression unit can decompress or compress according to the corresponding instructions received. For example, taking an acceleration chip with the chip as the host as an example, the host sends a data decompression instruction to the chip's decompression unit to decompress the obtained matrix, and the host sends a data compression instruction to the chip's compression unit to compress the obtained matrix, etc. This application does not limit this.

[0071] It should be noted that Figures 4 and 5 above are illustrative examples for explaining the principles of this application and do not constitute a limitation on this application. For example, the data processing method provided in this application does not limit the type, structure, or application scenario of the model, nor does it limit the type or shape of the matrix, nor does it limit the implementation method of precision conversion. Moreover, this application is not limited to mixed-precision storage of matrices; it is similarly applicable to other multidimensional data, such as tensors (which can have any number of dimensions) and high-dimensional arrays, which will not be elaborated further here.

[0072] The data processing method provided in this application will be described in detail below through several method implementation examples.

[0073] Figure 6 is a flowchart of a data processing method provided in an embodiment of this application. As shown in Figure 6, taking the application of this method to a chip as an example, the method includes the following steps 601 to 603.

[0074] 601. Obtain the execution instructions of the operators in the model.

[0075] In the embodiments of this application, an operator refers to any operator in the model, and this application does not limit the type of operator or its position in the model. Illustratively, taking an accelerator chip as the host as an example, the host responds to the model's inference request by sending an operator execution instruction to the accelerator chip. Taking a processing chip as the terminal device as an example, an AI application runs on the terminal device, and the processing chip responds to the inference request initiated by the AI ​​application by generating an operator execution instruction. Furthermore, the chip provided in this application stores a model, and this chip can load and run the model according to the model's inference request to achieve model inference.

[0076] 602. Based on the execution instructions of the operator, obtain at least one data set, which includes multiple types of data, with different types of data corresponding to different precisions.

[0077] In this embodiment, the input data of the operator includes at least one data set, which is stored in the chip memory. Indicatively, the chip reads at least one data set from the chip memory based on the operator's execution instructions. For any given data set, it includes at least one piece of data. This application does not limit the type of data set or the data; for example, the data set may be a matrix, and the data may be matrix elements within the matrix; or, for example, the data set may be a high-dimensional array, and the data may be elements within the high-dimensional array, and so on.

[0078] In some embodiments, taking an LLM model as an example, if the operator is a matrix multiplication operator, the at least one data set can be the weight matrix of the matrix multiplication operator; if the operator is an attention operator, the at least one data set can be the key matrix and value matrix of the attention operator, and this application does not limit this.

[0079] At least one dataset includes multiple types of data, and different types of data correspond to different precisions. This means that in at least one dataset, there exists data expressed with multiple precisions, and different precisions correspond to different types. In this application, the type indicates the importance of the data, which refers to the impact of the data on the execution result of the operator, or it can be the impact on the accuracy of the model. Indicatively, the greater the impact of a certain data on the execution result of the operator, the higher the importance of the data. Correspondingly, using a higher precision to express the data can ensure the accuracy of the model.

[0080] In some embodiments, at least one data set includes multiple types of data, including but not limited to the following: First, there is one data set, which includes multiple types of data, with different types of data corresponding to different precisions. Second, there are multiple data sets, which refer to data sets of various types, with different types of data sets corresponding to different precisions. Third, there are multiple data sets, and for any one data set, this data set includes multiple types of data, with different types of data corresponding to different precisions. It should be understood that at least one data set including multiple types of data can also be other cases, such as some data sets using the same precision, and the remaining data sets using mixed precision, etc., which is not limited in this application. That is, this application does not limit the granularity of data partitioning for mixed precision storage; different precisions can be used to store different data sets, or different data in the same data set can be stored with different precisions, thereby specifically improving the compression rate.

[0081] Furthermore, this application does not limit the method used to implement different precisions for different types of data. For example, different precisions can be represented by the bit width of different data, or by the numerical distribution ratio under the same bit width. The following uses multiple types of data, including first data of the first type and second data of the second type, as examples to introduce the different precisions corresponding to different types of data.

[0082] Schematic, the importance indicated by the first type is lower than that indicated by the second type, and correspondingly, the precision of the first data is lower than that of the second data. Moreover, the first data and the second data can be different data in the same data set, or they can belong to different data sets. For example, taking the data set as a matrix, the first data and the second data can be different matrix elements in the same matrix, or they can belong to different matrices. This application does not limit this.

[0083] In some embodiments, the bit width of the first data and the bit width of the second data are different. It should be understood that the bit width of the data determines the range and precision of the numerical values ​​that can be represented; generally, the larger the bit width, the wider the range of numerical values ​​that can be represented, and the higher the precision. Therefore, the bit width of the first data is smaller than the bit width of the second data, so that the precision of the first data is lower than that of the second data.

[0084] In some embodiments, the bit width of the first data and the bit width of the second data are the same, but the numerical allocation ratio of the first data is different from that of the second data. It should be understood that for floating-point numbers, the numerical allocation of the bit width typically includes a sign bit, an exponent bit, and a mantissa bit. The length of the mantissa bit affects the precision of the data; generally, the more mantissa bits, the finer the fractional part can be represented, and the higher the precision. For example, for a 32-bit floating-point number, one possible allocation is to use more bits to represent the fractional part (i.e., the mantissa bits), which can more accurately represent smaller numerical changes, resulting in higher precision. Another allocation might allocate fewer bits to the mantissa bits and more to the exponent bits, which allows for a wider range of represented values, but is less precise for smaller numerical changes, resulting in lower precision. Therefore, with the same bit width, different precisions of floating-point numbers can be obtained by adjusting the allocation ratio of the sign bit, exponent bit, and mantissa bit. Thus, different numerical allocation ratios with the same bit width can reflect different precisions of the first data and the second data, achieving a lower precision for the first data than for the second data.

[0085] Furthermore, this application does not limit the proportion of different types of data in at least one dataset. In some embodiments, taking the first and second types described above as examples, the proportion of the first type of data in at least one dataset is greater than the proportion of the second type of data in at least one dataset. That is, the proportion of low-precision data is greater than the proportion of high-precision data. For example, the proportion of low-precision data is 80%, and the proportion of high-precision data is 20%. In this way, although the data with higher importance is represented using high precision, its proportion is small and does not affect the overall compression ratio. On the other hand, the use of low precision to represent most of the less important data effectively improves the compression ratio. Moreover, since high precision is used to represent the data with higher importance, the accuracy of the overall model inference can be ensured.

[0086] It should be noted that the first and second types mentioned above are merely illustrative examples, and this application does not limit the number of data types in at least one dataset. For example, it may include two types: important and unimportant, or three types: important, less important, and unimportant, or even more levels of importance types. In practical applications, it can be adaptively adjusted according to model architecture, chip processing power, chip storage capacity, etc. Any other method that uses different precision to express data according to its importance is applicable to this application.

[0087] Based on the principles illustrated in Figures 4 and 5 above, the chip has a compression function, capable of compressing data with different precisions and storing it in the chip's memory. Continuing with the example of the first and second data, the source of at least one data set will be described below. Illustratively, taking the original data of the first data as the third data and the original data of the second data as the fourth data as an example, before executing step 602, the chip can also perform the following steps:

[0088] Step 1: Based on the importance of the third and fourth data, determine whether the third data belongs to the first type and whether the fourth data belongs to the second type.

[0089] The input data of the operator includes third and fourth data. The third and fourth data can be different data elements from the same data set or belong to different data sets; this application does not limit this. Illustratively, the determination of data importance can be based on at least one of numerical distribution, exponential distribution, or spatial arrangement patterns. For example, taking numerical distribution as an example, if the third and fourth data are matrix elements in a 4×4 weight matrix, based on the numerical distribution of each matrix element in the weight matrix, it is determined which matrix elements belong to the first type and which belong to the second type. Since the values ​​on the two diagonals of the weight matrix are more concentrated than those in other positions, the matrix elements on the two diagonals are considered the fourth data, i.e., important data, and the remaining matrix elements are considered the third data, i.e., unimportant data. It should be understood that this is merely an example and does not constitute a limitation of this application.

[0090] For example, taking a matrix multiplication operator, for the weight matrix, based on the probability distribution and spatial arrangement of the exponent digits of the weight matrix, a portion of the matrix elements can be selected as important data (i.e., second-type data) according to a preset ratio, while the remaining portion is considered unimportant data (i.e., first-type data). The preset ratio can be adjusted according to business needs; for example, the preset ratio could be 80% for first-type data and 20% for second-type data. As another example, taking a matrix multiplication operator, the matrix elements in the weight matrix that have a significant impact on the output matrix can be determined as second-type data by reverse calculation based on the output matrix. It should be noted that for LLM, which consists of multiple layers of a Transformer structure stacked sequentially, each layer includes attention operators and multiple matrix multiplication operators of a feed-forward network (FFN). The weight matrix of each matrix multiplication operator can be flexibly adjusted according to the needs of model accuracy and compression ratio. Each weight matrix can use different preset ratios (i.e., the ratio of important to unimportant data), or multiple weight matrices can use the same preset ratio; this application does not impose any limitations on this.

[0091] Step 2: Convert the precision of the third data according to the precision corresponding to the first type to obtain the first data, and convert the precision of the fourth data according to the precision corresponding to the second type to obtain the second data.

[0092] Since different types correspond to different precisions, after determining the types of the third and fourth data, the corresponding data can be precision-converted according to the precision corresponding to each type to obtain the first and second data. In some embodiments, the chip uses a quantization algorithm to implement precision conversion, or the chip uses different bit depths to implement precision conversion. Indicatively, this step includes the following two optional implementation methods:

[0093] (1) According to the first scaling factor corresponding to the first type, the third data is precision-converted to obtain the first data, and according to the second scaling factor corresponding to the second type, the fourth data is precision-converted to obtain the second data. It should be understood that in the quantization algorithm, the scaling factor is used to map the numerical range of the original data to the finite numerical range of the quantized data. By multiplying by the scaling factor, the original data is adjusted to a range suitable for quantization, and then rounding or truncation is performed to obtain the quantized value.

[0094] (2) Based on the first bit count corresponding to the first type, the third data is precision-converted to obtain the first data. Similarly, based on the second bit count corresponding to the second type, the fourth data is precision-converted to obtain the second data. The first bit count is less than the second bit count. It should be understood that for the same data, different bit counts result in different data precisions; generally, the larger the bit count, the higher the precision. Therefore, by setting the first bit count to be less than the second bit count, the precision of the first data obtained after precision conversion of the third and fourth data is lower than that of the second data. For example, the first bit count is 3 and the second bit count is 8. For details on how to represent data using different bit counts, please refer to the aforementioned introduction to key terms; further details will not be repeated here.

[0095] Schematic reference is made to Figure 7, which is a schematic diagram of matrix compression storage provided in an embodiment of this application. As shown in Figure 7, taking the original weight matrix as an example with a size of 4×4 and a bit width of 16, the matrix elements in the matrix are divided into a first type of matrix element and a second type of matrix element by determining the importance of each matrix element. The first type of matrix element is represented by the low bit width of 4 (i.e., bit width of 4), and the second type of matrix element is represented by the high bit width of 8 (i.e., bit width of 8). It should be noted that Figure 7 is only an example and does not constitute a limitation of this application. Moreover, the figure is introduced as an example of obtaining a compressed matrix with mixed bit storage after compressing different matrix elements in a matrix. In some embodiments, based on the foregoing description, at least one data set of the operator can include multiple types of data sets. Therefore, taking the weight matrix shown in Figure 7 as an example, the first type of matrix elements can be combined into one matrix after precision conversion, and the second type of matrix elements can be combined into another matrix after precision conversion.

[0096] 603. Process at least one data set to execute an operator.

[0097] In this embodiment, since at least one data set includes multiple types of data, and different types of data correspond to different precisions, and the operator usually needs to meet the target precision during execution, in this step, the chip performs precision conversion on at least one data set to obtain a target data set. The precision of the target data set is the target precision. The target data set is used as the input data of the operator, the operator is executed, and the execution result of the operator is obtained. It should be understood that the process of performing precision conversion on at least one data set to obtain the target data set can also be understood as the process of decompressing at least one data set. In addition, based on the chip architecture shown in Figure 4 above, it can be seen that in some scenarios, a decompression unit can be added to the chip to accelerate the decompression process. Accordingly, before this step, the chip's decompression unit obtains the data decompression instruction and executes this step based on the data decompression instruction. In some embodiments, where the chip supports operator execution capabilities of different precisions, the chip may not perform precision conversion on at least one data set. For example, the operator is a matrix multiplication operator, and at least one data set includes two matrices A and B represented with different precisions. Matrix A and matrix B are obtained by splitting the weight matrix of the matrix multiplication operator according to data importance and then compressing them separately. After the chip executes the matrix multiplication operator based on matrix A and matrix B respectively, it obtains two execution results. The two execution results are then fused to obtain the execution result of the matrix multiplication operator. In practical applications, whether to perform precision conversion on at least one data set can be set according to requirements, and this application does not impose any limitations on this.

[0098] Furthermore, based on step 602 above, it is clear that data of different types in at least one data set can be different data elements within the same data set, or they can belong to different data sets. Accordingly, this step includes the following two cases:

[0099] (1) Convert the precision of the first data set and the second data set to the target precision to obtain the target data set. The precision of the first data set and the second data set are different. Here, the first data set and the second data set are any two data sets from multiple data sets. For example, in step 602, the chip obtains two matrices. Matrix A is represented by 4 bits and matrix B is represented by 8 bits. The target precision corresponds to 16 bits. Based on this, the chip performs precision conversion on matrix A and matrix B respectively to obtain two converted matrices, both of which are represented by 16 bits.

[0100] (2) Convert the precision of each data in the third data set to the target precision to obtain the target data set. The third data set includes multiple types of data, with different types of data corresponding to different precisions. The third data set can be any one of at least one data set. For example, in step 602, the chip obtains a matrix where matrix element a is represented using 4 bits, matrix element b is represented using 8 bits, and the target precision corresponds to 16 bits. Based on this, the chip performs precision conversion on matrix element a and matrix element b respectively, obtaining two converted matrix elements, both of which are represented using 16 bits.

[0101] As can be seen from step 602 above, during data compression, the chip can use a quantization algorithm to achieve precision conversion, or it can use different bit depths to achieve precision conversion. Similarly, in this step, during data decompression, the chip can also use a quantization algorithm to achieve precision conversion, or it can use different bit depths to achieve precision conversion; these details will not be elaborated further here.

[0102] In summary, in the data processing method provided in this application, during the processing of at least one data set, the at least one data set includes multiple types of data, with different types of data corresponding to different precisions. Because the data in the data set is classified, different types of data correspond to different precisions. Therefore, when storing the data set, it is not necessary to compress it according to a uniform precision, thereby improving the compression ratio and saving chip resources. Furthermore, taking the data set as input data for operators in the model as an example, by distinguishing the relevant data of the model according to their importance, data of different importance is compressed with different precisions before being stored in the chip memory. Thus, while ensuring model accuracy, the model compression ratio is maximized. Moreover, during model inference, when the chip executes the relevant operators of the model, it obtains the relevant data of the operators. Since data of different importance corresponds to different precisions, the chip can convert the precision of this data to the precision required to execute the operators before executing the operators, thereby obtaining the execution results of the operators and ensuring the accuracy of the execution results.

[0103] In addition, as can be seen from the foregoing description of the implementation environment, the data processing method provided in this application is also applicable to a data processing system, which includes a host and an acceleration chip. Referring to Figure 8 below, another data processing method provided in this application will be described.

[0104] Figure 8 is a flowchart of another data processing method provided in an embodiment of this application. As shown in Figure 8, the method is applied to a data processing system, which includes a host and an acceleration chip. The host is used to control the acceleration chip. Schematic, the method includes the following steps 801 to 804.

[0105] 801. In response to the model's inference request, the host sends the operator execution instruction to the acceleration chip.

[0106] The host computer communicates with the acceleration chip, and the host computer can access a wired or wireless network to provide model inference services to users. The acceleration chip can be any acceleration chip connected to the host computer, and this application does not limit this.

[0107] 802. Accelerate the chip to receive the execution instructions of the operator.

[0108] 803. The acceleration chip executes instructions based on operators to obtain at least one data set, which includes multiple types of data, with different types of data corresponding to different precision.

[0109] Steps 802 to 803 are the same as steps 601 and 602 mentioned above, so they will not be described again.

[0110] In some embodiments, taking multiple types of data, including first data of a first type and second data of a second type, as an example, the importance indicated by the first type is lower than the importance indicated by the second type, and correspondingly, the precision of the first data is lower than the precision of the second data. Moreover, the first data and the second data can be different data in the same data set, or they can belong to different data sets.

[0111] In this embodiment, taking the original data of the first data as the third data and the original data of the second data as the fourth data as an example, the source of at least one data set can be achieved through the following steps: the host determines that the third data belongs to a first type and the fourth data belongs to a second type based on the importance of the third data and the fourth data; the host performs precision conversion on the third data according to the precision corresponding to the first type to obtain the first data, and performs precision conversion on the fourth data according to the precision corresponding to the second type to obtain the second data; the host sends the first data and the second data to the acceleration chip. That is, the determination of the importance of the data and the data precision conversion process can be implemented by the host, which can save the computing resources of the acceleration chip. Of course, in other embodiments, the host can also issue computing tasks to the acceleration chip to control the acceleration chip to execute the aforementioned steps of precision conversion of the third data and the fourth data, which is not limited in this application.

[0112] 804. The accelerator chip processes at least one data set to execute operators.

[0113] Step 804 is similar to step 603 above, so it will not be described again. In addition, based on the chip architecture shown in Figure 4 above, it can be seen that in some scenarios, a decompression unit can be added to the chip to accelerate the decompression process. Accordingly, before this step, the decompression unit of the acceleration chip obtains the data decompression command sent by the host, and executes this step based on the data decompression command.

[0114] In summary, in the data processing method provided in this application, during the processing of at least one data set, the at least one data set includes multiple types of data, and different types of data correspond to different precisions. Since the data in the data set is classified, different types of data correspond to different precisions. Therefore, when storing the data set, it is not necessary to compress it according to a uniform precision, thereby improving the compression ratio and saving chip resources.

[0115] Based on the above method embodiments, this application also provides a data processing device, which can implement some or all of the functions of the aforementioned chip through software, hardware, or a combination of both. Schematic, FIG9 is a structural diagram of a data processing device provided in an embodiment of this application. As shown in FIG9, the device includes an acquisition module 901 and a processing module 902.

[0116] The acquisition module 901 is used to acquire at least one data set, which includes multiple types of data, with different types of data corresponding to different precision.

[0117] Processing module 902 is used to process at least one data set.

[0118] In some embodiments, a type is used to indicate the importance of data, and multiple types of data include first data of a first type and second data of a second type, wherein the importance indicated by the first type is lower than that indicated by the second type, and the precision of the first data is lower than that of the second data.

[0119] In some embodiments, the first data and the second data are different data in the same data set; or, the first data and the second data belong to different data sets.

[0120] In some embodiments, the bit width of the first data and the bit width of the second data are different; or, the bit width of the first data and the bit width of the second data are the same, but the numerical allocation ratio of the first data is different from that of the second data.

[0121] In some embodiments, the proportion of the first type of data in at least one dataset is greater than the proportion of the second type of data in at least one dataset.

[0122] In some embodiments, the device further includes a precision conversion module for any of the following:

[0123] The third data is precision-converted according to the first scaling factor corresponding to the first type to obtain the first data, and the fourth data is precision-converted according to the second scaling factor corresponding to the second type to obtain the second data; or,

[0124] The precision of the third data is converted according to the first bit length corresponding to the first type to obtain the first data. The precision of the fourth data is converted according to the second bit length corresponding to the second type to obtain the second data. The first bit length is less than the second bit length.

[0125] In some embodiments, the processing module 902 is configured to:

[0126] Perform precision transformation on at least one dataset to obtain a target dataset, where the precision of the target dataset is the target precision.

[0127] In some embodiments, the processing module 902 is configured to perform any of the following:

[0128] Convert the precision of the first and second data sets to the target precision to obtain the target data set, where the precision of the first and second data sets differs; or,

[0129] The precision of each data point in the third dataset is converted to the target precision to obtain the target dataset. The third dataset includes multiple types of data, with different types of data corresponding to different precisions.

[0130] In some embodiments, at least one dataset includes the weight matrix of the matrix multiplication operator in a large language model (LLM).

[0131] In some embodiments, at least one dataset includes the key matrix and value matrix of the attention operator in a large language model (LLM).

[0132] Through the aforementioned device, during the processing of at least one dataset, which includes multiple types of data with different precisions, the data is categorized to correspond to different precisions. Therefore, when storing the dataset, it is unnecessary to compress it to a uniform precision, thus improving the compression ratio and saving chip resources. Furthermore, taking the dataset as input data for operators in a model as an example, by distinguishing the relevant data of the model according to their importance, data of different importance is compressed with different precisions before being stored in the chip memory. This maximizes the model compression ratio while ensuring model accuracy. Moreover, during model inference, when the chip executes relevant operators of the model, it acquires the relevant data of the operators. Since data of different importance corresponds to different precisions, the chip can convert the precision of this data to the precision required to execute the operators before executing them, thereby obtaining the execution results and ensuring the accuracy of the execution results.

[0133] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0134] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, first data can be referred to as second data, and similarly, second data can be referred to as first data. Both first data and second data can be data, and in some cases, they can be separate and distinct data.

[0135] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple data means two or more data.

[0136] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0137] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of program structure information. This program structure information includes one or more program instructions. When these program instructions are loaded and executed on a computing device, the processes or functions according to the embodiments of this application are generated, in whole or in part.

[0138] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0139] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized by, The method comprises: obtaining at least one data set, the at least one data set comprising a plurality of types of data, different types of data corresponding to different accuracies; processing the at least one data set.

2. The method of claim 1, wherein, The types are used to indicate the importance of the data, and the plurality of types of data comprises first data of a first type and second data of a second type, wherein the importance indicated by the first type is lower than the importance indicated by the second type, and the accuracy of the first data is lower than the accuracy of the second data.

3. The method of claim 2, wherein: the first data and the second data are different data in the same data set; or the first data and the second data belong to different data sets.

4. The method of claim 2 or 3, wherein: the bit width of the first data and the bit width of the second data are different; or the bit width of the first data and the bit width of the second data are the same and the numerical value allocation proportion of the first data and the numerical value allocation proportion of the second data are different.

5. The method according to any one of claims 2 to 4, characterized in that, The proportion of the data of the first type in the at least one data set is greater than the proportion of the data of the second type in the at least one data set.

6. The method according to any one of claims 2 to 5, characterized in that, The method further comprises any one of the following: performing accuracy conversion on third data according to a first scaling factor corresponding to the first type to obtain the first data, and performing accuracy conversion on fourth data according to a second scaling factor corresponding to the second type to obtain the second data; or performing accuracy conversion on the third data according to a first number of bits corresponding to the first type to obtain the first data, and performing accuracy conversion on the fourth data according to a second number of bits corresponding to the second type to obtain the second data, the first number of bits being less than the second number of bits.

7. The method according to any one of claims 1 to 6, characterized in that, The processing of the at least one data set comprises: performing accuracy conversion on the at least one data set to obtain a target data set, the accuracy of the target data set being a target accuracy.

8. The method of claim 7, wherein, The accuracy conversion of the at least one data set to obtain a target data set comprises any one of the following: converting the accuracies of a first data set and a second data set to the target accuracy to obtain the target data set, the accuracies of the first data set and the second data set being different; or converting the accuracies of each data in a third data set to the target accuracy to obtain the target data set, the third data set comprising a plurality of types of data, different types of data corresponding to different accuracies.

9. The method according to any one of claims 1 to 8, characterized in that, The at least one data set comprises a weight matrix of a matrix multiplication operator in a large language model LLM.

10. The method according to any one of claims 1 to 8, characterized in that, The at least one data set comprises a key matrix and a value matrix of an attention operator in a large language model LLM.

11. A data processing apparatus, characterized by, The device comprises: an obtaining module configured to obtain at least one data set, the at least one data set comprising a plurality of types of data, different types of data corresponding to different accuracies; a processing module configured to process the at least one data set.

12. The apparatus of claim 11, wherein, The types are used to indicate the importance of data, the plurality of types of data includes first data of a first type and second data of a second type, wherein the importance indicated by the first type is lower than the importance indicated by the second type, and the precision of the first data is lower than the precision of the second data.

13. The apparatus of claim 12, wherein, the first data and the second data are different data in a same data set; or the first data and the second data belong to different data sets.

14. The apparatus of claim 12 or 13, wherein, the bit width of the first data and the bit width of the second data are different; or the bit width of the first data and the bit width of the second data are the same and the numerical value allocation proportion of the first data and the numerical value allocation proportion of the second data are different.

15. The apparatus of any one of claims 12-14, wherein, The proportion of the data of the first type in the at least one data set is greater than the proportion of the data of the second type in the at least one data set.

16. The apparatus of any one of claims 12-15, wherein, The apparatus further comprises an accuracy conversion module configured to perform any of the following: convert a third data according to a first scaling factor corresponding to the first type to obtain the first data, and convert a fourth data according to a second scaling factor corresponding to the second type to obtain the second data; or convert the third data according to a first number of bits corresponding to the first type to obtain the first data, and convert the fourth data according to a second number of bits corresponding to the second type to obtain the second data, wherein the first number of bits is less than the second number of bits.

17. The apparatus of any one of claims 11 to 16, wherein, The processing module is configured to: convert the accuracy of the at least one data set to obtain a target data set, and the accuracy of the target data set is a target accuracy.

18. The apparatus of claim 17, wherein, The processing module is configured to perform any of the following: convert the accuracy of a first data set and a second data set to the target accuracy to obtain the target data set, wherein the accuracy of the first data set and the second data set is different; or convert the accuracy of each data in a third data set to the target accuracy to obtain the target data set, wherein the third data set includes a plurality of types of data, and different types of data correspond to different accuracies.

19. The apparatus of any of claims 11 to 18, wherein, The at least one data set includes a weight matrix of a matrix multiplication operator in a large language model LLM.

20. The apparatus of any one of claims 11-18, wherein, The at least one data set includes a key matrix and a value matrix of an attention operator in a large language model LLM.

21. A chip, characterized by The chip comprises a processor and a memory, and the processor is configured to execute instructions stored in the memory to enable the chip to perform the data processing method of any one of claims 1 to 10.

22. A server, comprising: The server comprises the chip of claim 21.

23. A data processing system comprising: The system comprises a host and an acceleration chip, wherein the host is configured to control the acceleration chip, and the acceleration chip is configured to perform the data processing method of any one of claims 1 to 10.

Images