Data processing method and apparatus therefor

Abstract

A data processing method and an apparatus therefor, applicable to the field of artificial intelligence. The method comprises: acquiring first data, the first data being obtained by means of a machine learning model on the basis of first input data (501); performing compression processing on the first data to obtain first compressed data, and storing the first compressed data in a memory (502); reading the first compressed data from the memory and decompressing the first compressed data to obtain decompressed first compressed data, wherein the decompressed first compressed data and second data are configured to be input into the machine learning model, and the second data is obtained on the basis of second input data (503); and performing compression processing on the first compressed data or the decompressed first compressed data to obtain second compressed data, and storing the second compressed data in the memory, wherein the precision loss of the first compressed data relative to the first data is less than the precision loss of the second compressed data relative to the first data (504). The present application reduces the storage requirements for intermediate results to be reused while ensuring the inference accuracy of the machine learning model.

Description

A data processing method and device thereof

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on March 4, 2024, with application number 202410246289.9 and application name “A data processing method and device thereof”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method and device thereof. Background Art

[0003] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0004] When input data is processed by a machine learning model, the machine learning model may include an attention layer. The attention layer may perform attention calculations on the input data (e.g., the data is a token, which is referred to as token below for convenience). When the attention layer performs attention calculations on the token, it obtains an intermediate result that needs to be reused when performing attention calculations on the token later. For example, the intermediate result may be KV (key-value) data.

[0005] In this process, when processing a new token, the intermediate results that can be reused can be stored in the memory, so that when they need to be reused for attention calculation of other tokens later, the intermediate results can be read from the memory and the attention calculation of the subsequent token can be performed based on the intermediate results.

[0006] However, the amount of reusable intermediate results that need to be stored grows rapidly as reasoning progresses, resulting in a high demand for storage. Furthermore, overly large intermediate results can also slow the reasoning process significantly, so reusable intermediate results need to be compressed. Summary of the Invention

[0007] In a first aspect, the present application provides a data processing method, the method comprising: obtaining first data, the first data being obtained through a machine learning model based on first input data; compressing the first data to obtain first compressed data, and storing the first compressed data in a memory; reading the first compressed data from the memory and decompressing the first compressed data to obtain decompressed first compressed data, the decompressed first compressed data and second data being used to input into the machine learning model, the second data being obtained based on the second input data; and compressing the first compressed data or the decompressed first compressed data to obtain second compressed data, and storing the second compressed data in the memory, the precision loss of the first compressed data compared to the first data being less than the precision loss of the second compressed data compared to the first data.

[0008] That is, the second compressed data has been compressed more times than the first compressed data. Since the precision loss is smaller with fewer compression times and greater with more compression times, the precision loss of the second compressed data is greater than that of the first compressed data.

[0009] In existing implementations, when compressing the same intermediate result that needs to be reused when performing attention calculations (in order to simplify the description, this embodiment may refer to the intermediate result that needs to be reused as the intermediate result), the same compressed data will be kept in the memory after the first compression. This method will result in: even if compression is performed, since the stored compressed data does not change as the reasoning process proceeds, as more and more intermediate results need to be stored, the required storage capacity is still large. In the embodiment of the present application, the same intermediate result stored in the memory can be compressed with differentiated accuracy as the reasoning process proceeds, that is, the accuracy loss after compression will change, but not It is always unified. Specifically, as the reasoning process proceeds, the precision loss of the compressed intermediate results will increase. Although the precision loss of the compressed intermediate results will increase, as the reasoning proceeds, the importance of the intermediate results generated longer ago to the current reasoning results will gradually decrease (for example, in the language model, in the process of autoregressive generation, the current reasoning result (token) is mainly highly correlated with one or more adjacent tokens, and has little correlation with tokens that are farther away). Therefore, even if the intermediate results generated a long time ago are compressed with a large precision loss, the current reasoning accuracy of the model will not be greatly affected. Furthermore, in the embodiment of the present application, while ensuring the reasoning accuracy of the machine learning model, the storage requirements of the intermediate results that need to be reused are reduced.

[0010] Each time the latest input data is processed by the machine learning model, it is necessary to obtain the intermediate results that have been generated in the past (such as KV data) and input them into the machine learning model to perform attention operations on the latest input data (that is, operations based on the attention mechanism). In this process, intermediate results will still be obtained (that is, data that still needs to be reused when processing subsequent input data). The newly generated intermediate results and the intermediate results that have been generated in the past can be spliced ​​and the attention weight calculation can be performed.

[0011] In one possible implementation, the first data may be data that needs to be reused by the machine learning model introduced above during the operation of subsequent input data.

[0012] In a possible implementation, the first data may be K data or V data. That is, the first data may be K data, the first data may be V data, or the first data may be K data and V data.

[0013] In a possible implementation, the first data may be obtained by compressing the intermediate result.

[0014] In a possible implementation, the first data may be obtained by compressing and decompressing the intermediate result.

[0015] Specifically, in a possible implementation, the first data can be obtained through a machine learning model based on the latest input data (first input data), or it can be the result obtained by compressing and decompressing the intermediate result obtained through a machine learning model based on the latest input data.

[0016] Specifically, in a possible implementation, the first data may be obtained through a machine learning model based on the latest input data, or may be a result obtained by compressing and decompressing an intermediate result obtained through a machine learning model based on the latest input data.

[0017] Specifically, in a possible implementation, the first data may not be obtained through the latest input data (first input data), but may be compressed data obtained from the memory, or may be obtained by decompressing the compressed data obtained from the memory.

[0018] When the first data is obtained based on the latest input data, the first data needs to be compressed and stored in the memory.

[0019] When the first data is not obtained based on the latest input data but is acquired from a memory, it may be necessary to perform compression of other strengths (ie, compression with different precision losses) on the first data and store it in the memory.

[0020] Among them, the second input data can be data input into the machine learning model after the first input data. For example, the second input data can be data input into the machine learning model after and adjacent to the first input data. For example, the second input data can be data input into the machine learning model after the first input data and separated by multiple input data.

[0021] It should be understood that the intermediate results in the embodiments of the present application may be obtained directly from the model, or may be obtained by performing other processing on the results obtained from the model. For example, other processing may include but is not limited to position encoding or other transformations.

[0022] In a possible implementation, storing the second compressed data in the memory includes: replacing the first compressed data stored in the memory with the second compressed data.

[0023] In one possible implementation, the second input data is data currently input to the machine learning model, and the second input data is data input to the machine learning model M th time after the first input data, and storing the second compressed data in the memory includes:

[0024] When at least one of the following conditions is met, the second compressed data is stored in the memory:

[0025] The M is greater than a first preset value; or,

[0026] The attention score obtained by inputting into the machine learning model meets the preset conditions.

[0027] The attention score can be understood as a similarity score calculated from the query matrix and the key matrix, typically performed by matrix multiplication. It can be used to express the relevance of each token in the sequence when processing the current token.

[0028] In a possible implementation, the M is greater than a first preset value, including:

[0029] The M is greater than a first preset value and less than a second preset value.

[0030] In a possible implementation, before storing the second compressed data in the memory, the method further includes:

[0031] storing the second data in the memory;

[0032] Storing the second compressed data in the memory includes:

[0033] When the amount of uncompressed data in the intermediate results obtained by the machine learning model stored in the memory reaches a third preset value, the first compressed data stored in the memory is replaced with the second compressed data, and the uncompressed data includes the second data. The compression accuracy of the data stored in the memory is not updated every time a piece of input data is processed. Instead, the compression accuracy is updated only after a certain amount (the third preset value) of input data has been processed, thereby reducing the frequency of compression processing and lowering computing power overhead.

[0034] In one possible implementation, the method further includes: compressing the second data to obtain third compressed data, and storing the third compressed data in the memory, wherein the precision loss of the third compressed data compared to the second data is less than the precision loss of the second compressed data compared to the first data.

[0035] In one possible implementation, when the third compressed data is stored in a memory, the data stored in the memory includes the third compressed data and the second compressed data. That is, based on this solution, compressed data with different compression precisions may exist in the memory at the same time.

[0036] In one possible implementation, after compressing the first compressed data or the decompressed first compressed data to obtain the second compressed data, the method further includes: reading the second compressed data from the memory and decompressing the second compressed data to obtain the decompressed second compressed data, wherein the decompressed second compressed data and third data are used to input into the machine learning model, and the third data is obtained based on the third input data; compressing the second compressed data or the decompressed second compressed data to obtain fourth compressed data and storing it in the memory; the precision loss of the second compressed data compared to the first data is less than the precision loss of the fourth compressed data compared to the first data. For the same data that needs to be reused (for example, the first data), more compression accuracy levels can be set for it, thereby forming a gradient compression accuracy.

[0037] In a second aspect, the present application provides a data processing device, comprising:

[0038] An acquisition module, configured to acquire first data, where the first data is an intermediate result obtained by performing a calculation using a machine learning model based on the first input data;

[0039] A processing module is used to compress the first data to obtain first compressed data, and store the first compressed data in a memory; after the attention interaction, compress the first compressed data or the decompressed first compressed data to obtain second compressed data, and store the second compressed data in the memory, and the accuracy loss of the first compressed data compared to the first data is less than the accuracy loss of the second compressed data compared to the first data.

[0040] In a possible implementation, the operation is a process of calculating attention weights.

[0041] In a possible implementation, the processing module is specifically configured to: after reading the first compressed data from the memory and decompressing the first compressed data, perform compression processing on the first compressed data or the decompressed first compressed data to obtain second compressed data.

[0042] In a possible implementation, the first data is key data or value data, and the second data is key data or value data.

[0043] In a possible implementation, the processing module is specifically configured to:

[0044] The first compressed data stored in the memory is replaced by the second compressed data.

[0045] In one possible implementation, the second input data is data currently input to the machine learning model, and the second input data is data input to the machine learning model Mth time after the first input data, and the processing module is specifically configured to:

[0046] When at least one of the following conditions is met, the second compressed data is stored in the memory:

[0047] The M is greater than a first preset value;

[0048] The attention score obtained by inputting into the machine learning model meets the preset conditions.

[0049] In a possible implementation, the M is greater than a first preset value, including: the M is greater than the first preset value and less than a second preset value.

[0050] In a possible implementation, before storing the second compressed data in the memory, the processing module is further configured to:

[0051] storing the second data in the memory;

[0052] The processing module is specifically used to:

[0053] When the amount of uncompressed data in the intermediate results obtained by the machine learning model stored in the memory reaches a third preset value, the first compressed data stored in the memory is replaced with the second compressed data, and the uncompressed data includes the second data.

[0054] In a possible implementation, the processing module is further configured to:

[0055] The second data is compressed to obtain third compressed data, and the third compressed data is stored in the memory, wherein the precision loss of the third compressed data compared to the second data is less than the precision loss of the second compressed data compared to the first data.

[0056] In a possible implementation, when the third compressed data is stored in a memory, the data stored in the memory includes the third compressed data and the second compressed data.

[0057] In a possible implementation, after compressing the first compressed data or the decompressed first compressed data to obtain the second compressed data, the processing module is further configured to:

[0058] Reading second compressed data from the memory and decompressing the second compressed data to obtain decompressed second compressed data, wherein the decompressed second compressed data and third data are input into the machine learning model, wherein the third data is obtained based on the third input data;

[0059] The second compressed data or the decompressed second compressed data is compressed to obtain fourth compressed data and stored in the memory; the precision loss of the second compressed data compared to the first data is less than the precision loss of the fourth compressed data compared to the first data.

[0060] In a possible implementation, the acquisition module is specifically configured to:

[0061] Based on the first input data, the attention layer of the machine learning model is used to perform calculations to obtain first data, where the first data is an intermediate result obtained by performing calculations on the machine learning model.

[0062] In a third aspect, an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to perform the first aspect and any optional method thereof.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof.

[0064] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0065] In a sixth aspect, the present application provides a chip system comprising a processor configured to support the execution of a data processing device to implement the functions described in the aforementioned aspects, such as transmitting or processing the data or information described in the aforementioned methods. In one possible design, the chip system further comprises a memory configured to store program instructions and data necessary for executing the device or training the device. The chip system may consist of a single chip or may include a chip and other discrete components. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework;

[0067] Figures 2 to 4 are schematic diagrams of the application system framework of the present invention;

[0068] FIG5 is a flowchart of a data processing method provided in an embodiment of the present application;

[0069] FIG6A is a schematic diagram of a model architecture provided in an embodiment of the present application;

[0070] FIG6B is a schematic diagram of the attention calculation process provided by an embodiment of the present application;

[0071] 7A to 7D are schematic diagrams of an application example provided by an embodiment of the present application;

[0072] FIG8 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0073] FIG9 is a schematic diagram of a structure of an execution device provided in an embodiment of the present application;

[0074] FIG10 is a schematic diagram of a structure of a training device provided in an embodiment of the present application;

[0075] FIG11 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0077] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0078] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0079] As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation, not as terms of degree, and are intended to take into account the inherent variations in measurements or calculations that one of ordinary skill in the art would recognize. Furthermore, the use of "may" when describing embodiments of the present invention refers to "one or more possible embodiments." As used herein, the terms "use," "using," and "used" may be considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively. Additionally, the term "exemplary" is intended to refer to an example or illustration.

[0080] First, let's describe the overall workflow of an AI system. See Figure 1, which shows a schematic diagram of the AI ​​framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0081] (1) Infrastructure

[0082] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0083] (2) Data

[0084] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0085] (3) Data processing

[0086] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0087] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0088] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0089] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0090] (4) General ability

[0091] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0092] (5) Smart products and industry applications

[0093] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0094] The system architecture provided in the embodiment of the present application is described in detail below with reference to FIG2 .

[0095] FIG2 is a schematic diagram of the system architecture provided by an embodiment of the present application. As shown in FIG2 , the system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 , and a data acquisition device 560 .

[0096] The execution device 510 includes a calculation module 511, an I / O interface 512, a pre-processing module 513, and a post-processing module 514. The calculation module 511 may include the target model / rule 501, and the pre-processing module 513 and the post-processing module 514 are optional.

[0097] The data collection device 560 is used to collect training samples. After collecting the training samples, the data collection device 560 stores these training samples in the database 530.

[0098] The training device 520 can train the neural network to be trained (such as the machine learning model in the embodiment of the present application) based on the training samples maintained in the database 530 to obtain the target model / rule 501.

[0099] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.

[0100] It should be noted that, in actual applications, the training samples maintained in the database 530 may not all be collected by the data acquisition device 560, but may also be received from other devices. It should also be noted that the training device 520 may not train the target model / rule 501 entirely based on the training samples maintained in the database 530, but may also obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0101] The target model / rule 501 obtained through training with the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in FIG2 . The execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, an in-vehicle terminal, etc., or a server, etc.

[0102] Specifically, the training device 520 may transfer the trained model to the execution device 510 .

[0103] In FIG. 2 , the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with an external device. A user can input data to the I / O interface 512 through a client device 540 .

[0104] Preprocessing module 513 and preprocessing module 514 are used to preprocess the input data received by I / O interface 512. It should be understood that preprocessing module 513 and preprocessing module 514 may be absent or only one preprocessing module may be present. If preprocessing module 513 and preprocessing module 514 are absent, computing module 511 may be used directly to process the input data.

[0105] When the execution device 510 preprocesses the input data, or when the computing module 511 of the execution device 510 performs calculations and other related processing, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 550.

[0106] Finally, the I / O interface 512 provides the processed results to the client device 540 and thus to the user.

[0107] In the scenario shown in FIG. 2 , the user can manually input data, and this "manual input data" can be operated through the interface provided by I / O interface 512. Alternatively, client device 540 can automatically send input data to I / O interface 512. If user authorization is required for client device 540 to automatically send input data, the user can set the corresponding permissions in client device 540. The user can view the output of execution device 510 on client device 540, which can be presented in a display, sound, action, or other specific form. Client device 540 can also serve as a data acquisition terminal, collecting input data and output results from I / O interface 512 as new sample data and storing them in database 530. Of course, collection can also be performed without client device 540, with I / O interface 512 directly storing the input data and output results from I / O interface 512 as new sample data in database 530.

[0108] It is worth noting that FIG2 is merely a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in FIG2 , the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the execution device 510 can be deployed in the client device 540.

[0109] From the inference side of the model:

[0110] In the embodiment of the present application, the computing module 511 of the above-mentioned execution device 510 can obtain the code stored in the data storage system 550 to implement the steps related to the model reasoning process in the embodiment of the present application.

[0111] In an embodiment of the present application, the computing module 511 of the execution device 510 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0112] Specifically, the computing module 511 of the execution device 510 can be a hardware system with an execution instruction function, and the steps related to the model reasoning process provided in the embodiment of the present application can be software codes stored in the memory. The computing module 511 of the execution device 510 can obtain the software code from the memory and execute the obtained software code to implement the steps related to the model reasoning process provided in the embodiment of the present application.

[0113] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to the model reasoning process provided in the embodiment of the present application can also be implemented by the hardware system that does not have the function of executing instructions in the computing module 511 of the execution device 510, which is not limited here.

[0114] From the training side of the model:

[0115] In an embodiment of the present application, the above-mentioned training device 520 can obtain the code stored in the memory (not shown in Figure 2, which can be integrated into the training device 520 or deployed separately from the training device 520) to implement the steps related to model training in the embodiment of the present application.

[0116] In an embodiment of the present application, the training device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0117] It should be understood that the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided in the embodiments of the present application can also be implemented by the hardware system in the training device 520 that does not have the function of executing instructions, which is not limited here.

[0118] In the embodiments of the present application, the forward propagation process of the model is involved, which can be executed by the execution device 510 or the training device 520 introduced in the above embodiments.

[0119] In addition, the execution device 510 or the training device 520 can process the input data through a machine learning model, wherein the machine learning model may include an attention layer, and the attention layer can perform attention calculation on the input token, and when the attention layer performs attention calculation on the token, it can obtain an intermediate result that needs to be reused when performing attention calculation on the token later. For example, the intermediate result can be KV data. In this process, when processing a new token, the obtained reusable intermediate result can be stored in the memory, so that when it needs to be reused when performing attention calculation on other tokens later, the intermediate result can be read from the memory, and the attention calculation of the subsequent token can be performed based on the intermediate result. However, the amount of reusable intermediate results that need to be stored will increase rapidly as the reasoning proceeds, resulting in a large demand for storage. In addition, an excessively large intermediate result will also make the reasoning process extremely slow, so compressing the reusable intermediate results is particularly important.

[0120] In one implementation, the compression process can be performed by a compression module, which can be centrally deployed with the above-mentioned execution device 510 or training device 520, for example, belonging to the same chip or computing unit of other granularity, or it can be deployed separately, for example, belonging to different chips. For example, the execution device 510 or training device 520 can be an AI chip, and the compression module can belong to the CPU.

[0121] For example, referring to Figures 3 and 4, Figures 3 and 4 are schematic diagrams of the architecture of an embodiment of the present application, wherein the model processing module can obtain an intermediate result by running a machine learning model, the compression module can compress the intermediate result and write the compressed data into a memory, the compression module can read the compressed data from the storage area and decompress it, obtain the decompressed result and pass it to the model processing module. The compression module and the model processing module in Figure 3 are separately deployed on different chips, while the compression module and the model processing module in Figure 4 are centrally deployed on the same chip.

[0122] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0123] (1) Neural Network

[0124] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0125] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0126] (2) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way of extracting features is independent of position. The convolution kernel can be formalized as a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0127] CNN is a very common neural network. The following section will focus on a detailed description of its structure, using Figure 4. As mentioned in the previous section about basic concepts, a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture, which uses machine learning algorithms to perform multiple levels of learning at different levels of abstraction. As a deep learning architecture, a CNN is a feed-forward artificial neural network, in which individual neurons respond to input images.

[0128] (3) Deep Neural Networks

[0129] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. Based on the position of different layers in DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0130] (4) Loss function

[0131] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0132] (5) Backpropagation algorithm

[0133] The back propagation (BP) algorithm can be used to correct the size of the initial model parameters during training, reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial model, thereby converging the error loss. The BP algorithm is a backward propagation movement driven by error loss, aiming to obtain optimal model parameters, such as the weight matrix.

[0134] (6) Large Language Model: Large language models refer to natural language processing models trained on large amounts of data, typically with billions or tens of billions of parameters. These models capture the general characteristics of language by learning from large amounts of text data during the pre-training phase, and can then be fine-tuned on downstream tasks to adapt to specific task requirements.

[0135] (7) Transformer: Transformer is a deep learning model architecture originally used for sequence-to-sequence tasks such as machine translation. It uses a self-attention mechanism to process input sequences and has achieved great success in the field of natural language processing. Most large language models, such as BERT, GPT, and T5, are based on the Transformer architecture.

[0136] (8) KV cache (key-value cache): KV cache refers to a cache structure that stores key-value pairs. In large language models, KV cache is often used to store intermediate results or other useful information when the model processes text to improve efficiency. By using KV cache, the model can avoid repeated calculations when processing text.

[0137] (9) Key-Value cache quantization: KV cache quantization refers to quantizing the values ​​in the KV cache to reduce storage space and computational overhead. In some large language models, in order to adapt the model to limited resources, the values ​​in the KV cache can be quantized to reduce the storage and computational costs of the model.

[0138] (10) Perplexity (PPL): PPL is a metric used to evaluate the performance of a language model, indicating the model's ability to predict a given text sequence. PPL is a positive real number that can be understood as the average difficulty of the model in predicting the observed data sequence. The lower the PPL, the better the model performance.

[0139] (11) Non-Uniform Quantization: Non-Uniform Quantization is a quantization method in which the value range is divided into intervals of different sizes to better adapt to the data distribution. Unlike uniform quantization, non-uniform quantization can assign a different number of value ranges to each interval based on the data distribution.

[0140] (12) Token: In natural language processing, a "token" is the basic unit of text string segmentation. This can be a word, a character, or a subword fragment. Large language models usually need to segment input text into tokens and then convert these tokens into numerical representations (such as word vectors) that the model can understand.

[0141] (13) Sequence: In the context of large language models, "sequence" refers to a sequence of elements with a certain order relationship, and multiple tokens constitute a sequence.

[0142] (14) Incremental Inference: Incremental inference allows the model to process only newly added portions of the input, rather than reprocessing the entire sequence each time. This is achieved by maintaining contextual information in the model's internal state, allowing the model to respond quickly when new input is received. Incremental inference is particularly useful in interactive applications, such as chatbots or real-time translation, as it can significantly reduce latency and computing resource usage.

[0143] The present invention provides a data processing method. The data processing method of the present invention is described in detail below with reference to the accompanying drawings.

[0144] Refer to Figure 5, which is a flow chart of a data processing method provided in an embodiment of the present application. As shown in Figure 5, a data processing method provided in an embodiment of the present application may include steps 501 to 503, and these steps are described in detail below.

[0145] 501. Obtain first data, where the first data is obtained through a machine learning model based on first input data.

[0146] In one possible implementation, the first input data can be processed by a machine learning model based on the first input data to obtain a processing result, wherein the machine learning model may include an attention layer. When the attention layer performs attention calculation on the token, it can obtain an intermediate result (i.e., the first data) that needs to be reused when performing attention calculation on the token later. For example, the intermediate result can be KV data (which can be called key data and value data), which can also be called a KV pair.

[0147] In the above process, each time the latest input data is processed by the machine learning model, it is necessary to obtain the intermediate results that have been generated in the history (such as KV data) and input them into the machine learning model to perform attention operations on the latest input data (that is, operations based on the attention mechanism). In this process, intermediate results will still be obtained (that is, data that still needs to be reused when processing subsequent input data). The newly generated intermediate results and the intermediate results that have been generated in the history can be spliced ​​and the attention weight calculation can be performed.

[0148] For example, referring to FIG6B , K cache and V cache are intermediate results that have been generated and stored in the memory in the history of the attention layer of the machine learning model. The intermediate results stored in the memory can be compressed intermediate results (for example, the first compressed data in the embodiment of the present application). After the token of the latest input data is calculated by the machine learning model (for example, linear transformation is performed by the Wd matrix, the Wk matrix, and the Wv matrix), the Q data, K data, and V data of the latest input data can be obtained. Then, the K data of the latest input data (for example, the second data in the embodiment of the present application) and the K cache (the intermediate result obtained by obtaining and decompressing from the memory, such as the first compressed data decompressed in the embodiment of the present application) can be spliced ​​and transposed, and the V data of the latest input data (for example, the second data in the embodiment of the present application) and the V cache (the intermediate result obtained by obtaining and decompressing from the memory, such as the first compressed data decompressed in the embodiment of the present application) can be spliced. The subsequent operation can be obtained by referring to the relevant introduction of the attention operation in the prior art, which will not be repeated here. The K data and V data of the latest input data can be stored in the memory, and then can be obtained from the memory when the machine learning model performs operations on subsequent input data.

[0149] The machine learning model may be a language model.

[0150] Among them, the attention layer can perform attention operation on the input data. The attention operation is an operation based on the attention mechanism. Since the attention operation requires internal interaction between the K data, Q data and V data obtained from the same input token, the attention operation can also be called self-attention calculation.

[0151] Take, for example, a transformer-based machine learning model based on an attention mechanism. For example, referring to Figure 6A , the machine learning model can be a language model. The transformer is a model architecture based on an attention mechanism, with the self-attention structure as its core component. Each transformer layer consists of two main components: a multi-head self-attention layer and a feedforward neural network (FNN). The self-attention structure allows the model to dynamically focus on information at different positions when processing sequential data. The self-attention structure consists of three main components: query data, key data, and value data. For an input sequence, linear transformations are performed on Q, K, and V, respectively, to obtain Q data, K data, and V data. A softmax operation is performed on the interaction between K and Q data, and the result of the softmax operation is then interacted with V data to obtain the attention distribution of each position towards other positions. Finally, these distributions are weighted to obtain the output for the current position. The self-attention structure is used to process information at different positions in the input data. In self-attention, the representation of each position is a weighted average of all other positions in the sequence, where the weights are determined by computing the relationship between the query, key, and value at the current position. This enables the model to dynamically pay attention to different parts of the input sequence.

[0152] During inference, the self-attention architecture generates a key-value (KV) pair for each position. These KV pairs are used by the attention mechanism to calculate weights for different positions. The KV cache is a collection of KV pairs generated at a specific time step. Storing historical KVs avoids repeated computations and accelerates model inference.

[0153] In one possible implementation, the first data may be data that needs to be reused by the machine learning model introduced above during the operation of subsequent input data.

[0154] In a possible implementation, the first data may be K data or V data. That is, the first data may be K data, the first data may be V data, or the first data may be K data and V data.

[0155] The amount of reusable intermediate results that need to be stored will increase rapidly as the reasoning progresses, resulting in a large demand for storage. In addition, excessively large intermediate results will also make the reasoning process extremely slow, so compressing the reusable intermediate results is particularly important. Therefore, before storing the intermediate results (that is, the first data) that need to be reused when performing attention calculations, they need to be compressed and stored to reduce storage overhead. When using the compressed data, the compressed data needs to be read and decompressed to ensure that the model accuracy is maintained when calculating the attention distribution.

[0156] In a possible implementation, the first data may be obtained by compressing the intermediate result.

[0157] In a possible implementation, the first data is specifically obtained by compressing and decompressing the intermediate result.

[0158] Specifically, in a possible implementation, the first data can be obtained through a machine learning model based on the latest input data (first input data), or it can be the result obtained by compressing and decompressing the intermediate result obtained through a machine learning model based on the latest input data.

[0159] Specifically, in a possible implementation, the first data may be obtained through a machine learning model based on the latest input data, or may be a result obtained by compressing and decompressing an intermediate result obtained through a machine learning model based on the latest input data.

[0160] Specifically, in a possible implementation, the first data may not be obtained through the latest input data (first input data), but may be compressed data obtained from the memory, or may be obtained by decompressing the compressed data obtained from the memory.

[0161] When the first data is obtained based on the latest input data, it is necessary to compress it and store it in the memory.

[0162] When the first data is not obtained based on the latest input data but is acquired from a memory, it may be necessary to perform compression of other strengths (ie, compression with different precision losses) on the first data and store it in the memory.

[0163] 502. Compress the first data to obtain first compressed data, and store the first compressed data in a memory.

[0164] In an embodiment of the present application, after the first data is acquired, the first data may be compressed to obtain first compressed data, and the first compressed data may be stored in a memory, for example, the memory is a memory cache.

[0165] Among them, compression can be but is not limited to quantization, discarding, entropy coding, etc., and the embodiments of this application are not limited.

[0166] As shown in Figure 6A, Figure 6A shows the process of quantizing, storing, and dequantizing the KV cache. Quantizing the KV cache is a common optimization method that can reduce the storage and computational overhead of the model. Quantization typically involves converting floating-point representations into integer representations with fewer bits. Storing the quantized KV cache can reduce the model's memory usage. When the KV cache is needed, it needs to be dequantized, that is, the integer representation is restored to floating-point representation. This ensures that the model's accuracy is maintained when calculating the attention distribution.

[0167] 503. Read the first compressed data from the memory and decompress the first compressed data to obtain the decompressed first compressed data. The decompressed first compressed data and the second data are used to input into the machine learning model, and the second data is obtained based on the second input data.

[0168] When processing the second input data using the machine learning model, it is necessary to retrieve from the memory the compressed intermediate results that have been previously generated and stored and need to be reused to participate in the processing of the second input data. For example, if the compressed intermediate results that have been previously generated and stored and need to be reused are first compressed data, it is necessary to read from the memory and decompress the first compressed data to obtain the decompressed first compressed data.

[0169] For example, when compression is quantized, decompression can be dequantized, which converts the quantized integer representation back to floating-point representation. During model inference, when key-value (KV) pairs are needed, the dequantizer maps the integer representation back to floating-point numbers to ensure model accuracy. The dequantization process switches the model from quantized representation back to standard floating-point representation.

[0170] In one possible implementation, the second data can be obtained by performing calculations through the machine learning model based on the second input data. The second data is the intermediate result obtained by the attention layer. The attention distribution can be calculated based on the decompressed first compressed data and second data.

[0171] In addition, the second data may also be the result obtained by performing other processing on the result obtained by performing calculations based on the second input data through the machine learning model, for example, compressing and decompressing the result obtained by performing calculations based on the second input data through the machine learning model.

[0172] 504. Compress the first compressed data or the decompressed first compressed data to obtain second compressed data, and store the second compressed data in the memory, wherein the precision loss of the first compressed data compared to the first data is less than the precision loss of the second compressed data compared to the first data.

[0173] In the existing implementation, when compressing the same intermediate result that needs to be reused when performing attention calculation (in order to simplify the description, this embodiment can refer to the intermediate result that needs to be reused as the intermediate result), the same compressed data will be kept in the memory after the first compression. This method will result in: even if compression is performed, since the stored compressed data does not change as the reasoning process proceeds, as more and more intermediate results need to be stored, the required storage capacity is still large. In the embodiment of the present application, the same intermediate result stored in the memory can be compressed with different accuracy as the reasoning process proceeds, that is, the accuracy loss after compression will change, instead of It is always unified. Specifically, as the reasoning process proceeds, the precision loss of the compressed intermediate results will increase. Moreover, although the precision loss of the compressed intermediate results will increase, as the reasoning proceeds, the importance of the intermediate results generated longer ago to the current reasoning results will gradually decrease (for example, in the language model, in the process of autoregressive generation, the current reasoning result (token) is mainly highly correlated with one or more adjacent tokens, and has little correlation with tokens that are farther away). Therefore, even if the intermediate results generated a long time ago are compressed with a large precision loss, the current reasoning accuracy of the model will not be greatly affected. Furthermore, in the embodiment of the present application, while ensuring the reasoning accuracy of the machine learning model, the storage requirements of the intermediate results that need to be reused are reduced.

[0174] In a possible implementation, after reading the first compressed data from the memory and decompressing the first compressed data, compression processing may be performed on the first compressed data or the decompressed first compressed data to obtain second compressed data.

[0175] In a possible implementation, the first compressed data or the first compressed data after decompression may be compressed to obtain the second compressed data. For example, the first compressed data may be 8-bit data, and the second compressed data may be 4-bit data.

[0176] In one possible implementation, the first compressed data or the decompressed first compressed data can be compressed to obtain second compressed data, and the second compressed data can be stored in the memory. For example, the first compressed data stored in the memory can be replaced with the second compressed data.

[0177] Among them, the second compressed data obtained by compressing the first compressed data (or the first compressed data after decompression) can be stored in the memory and replace the first compressed data only when the interval between the current input data (that is, the second input data) and the first input data is within a preset range.

[0178] In one possible implementation, the second input data is the data currently input to the machine learning model, and the second input data is the Mth input data to the machine learning model after the first input data. The above-mentioned preset range can be: M is greater than the first preset value, or M is greater than the first preset value and less than the second preset value.

[0179] In one possible implementation, the second input data is the data currently input to the machine learning model, and the second input data is the Mth input data to the machine learning model after the first input data. The above-mentioned preset range can be: M is greater than the first preset value, or M is greater than the first preset value and less than the second preset value.

[0180] In one possible implementation, when the attention score obtained through the attention interaction meets a preset condition, the second compressed data obtained by compressing the first compressed data (or the first compressed data after decompression) can be stored in the memory and replace the first compressed data. For example, for all tokens within a certain range, only the k tokens with the highest cumulative attention scores can be retained, and the rest can be discarded.

[0181] For example, taking the intermediate result as KV cache, the embodiment of the present application can provide differentiated compression accuracy for the KV cache according to the time sequence of token reasoning. For example, when a token is just generated, its KV cache is given a lower compression accuracy, maintaining a higher accuracy, and at the same time, its compression accuracy is changed during the process of continuing reasoning. As shown in Figure 7A, the middle horizontal axis is the time axis of the reasoning process, and the right side is the newer token. There are many windows on the time axis, and each window has a compression level (the window size is not fixed, the compression level is not fixed). The newly generated token t t KV Cache k of (current token) t vt During inference, the window will gradually transition from compression level 1 to compression level x, while the compression level is constantly changing (with increasing loss of precision).

[0182] Exemplarily, compression level 1 can be: no compression, full precision retention, compression level 2 can be: high bit quantization (for example, 8-bit quantization), compression level 3 can be: low bit quantization (for example, 4-bit quantization), compression level 4 can be: low rank decomposition combined with quantization, and compression level 5 can be: discard.

[0183] In one possible implementation, the compression accuracy of the data stored in the memory is not updated every time an input data is processed, but the compression accuracy is updated only after a certain number of input data are processed. For example, the second data can be obtained based on the latest input data. At this time, the second data can be directly stored in the memory without being compressed. When the amount of uncompressed data including the second data reaches a third preset value, the first compressed data stored in the memory can be replaced by the second compressed data, or the second data in the memory can be replaced by the third compressed data obtained by compressing the second data.

[0184] In one possible implementation, when compressing the second data, the second data can be compressed to obtain third compressed data and the third compressed data can be stored in a memory, and the precision loss of the third compressed data compared to the second data is less than the precision loss of the second compressed data compared to the first data.

[0185] In one possible implementation, when the third compressed data is stored in a memory, the data stored in the memory includes both the third compressed data and the second compressed data. That is, based on this solution, compressed data with different compression accuracies may exist in the memory at the same time.

[0186] In one possible implementation, for the same data that needs to be reused (for example, the first data), more compression accuracy levels can be set for it, thereby forming a gradient compression accuracy. For example, the second compressed data can be read from the memory and decompressed to obtain the decompressed second compressed data. The decompressed second compressed data and the third data are used to input into the machine learning model, and the third data is obtained based on the third input data; the second compressed data or the decompressed second compressed data is compressed to obtain fourth compressed data and stored in the memory; the accuracy loss of the second compressed data compared to the first data is less than the accuracy loss of the fourth compressed data compared to the first data.

[0187] Next, several examples are used to introduce the embodiments of this application:

[0188] 7B , this embodiment has two compression levels for compressing data in the memory. The first compression level is 8-bit quantization with a compression window size of w1. The second compression level is 4-bit quantization (for the remaining tokens). The specific process is as follows:

[0189] For a Self Attention layer, the input at the current moment is x t , new token x t Input Self Attention layer to generate Q t K t V t Matrix. For historical KV cache (K i ,V i i≤t) is decompressed to generate KV_cache_dequant: for K i＜t-w1 ,V i＜t-w1 Decompress according to compression method 2 (4-bit inverse quantization), for K t-w1≤i＜t ,V t-w1≤i＜t Decompress according to compression method 1 (8-bit dequantization), and compare KV_cache_dequant with the current K t V t Cat together to generate a new KV cache to perform self attention calculation. Compress the current KV and store it in the historical KV cache. Specifically, you can t V t Compression is performed according to compression method 1 (8-bit quantization). t-w1 ,V t-w1 Compression is performed using compression method 2 (4-bit quantization).

[0190] 7C , this embodiment compresses data in the memory at three levels. The first level uses 8-bit quantization with a compression window size of w1. The second level uses 4-bit quantization with a compression window size of w2. The third level uses discarding (remaining tokens).

[0191] For a Self Attention layer, the input at the current moment is x t , new token x t Input Self Attention layer to generate Q t K t V tMatrix. Decompress the historical KV cache to generate KV_cache_dequant: t-w2≤i＜t-w1 ,V t-w2≤i＜t-w1 Decompress according to compression method 2 (4-bit inverse quantization). Specifically, for K t-w1≤i＜t ,V t-w1≤i＜t Decompress according to compression method 1 (8-bit dequantization), and compare KV_cache_dequant with the current K t V t Cat together to generate a new KV cache to perform self attention calculation. Compress the current KV and store it in the historical KV cache. Specifically, you can t V t Compression is performed according to compression method 1 (8-bit quantization). t-w1 ,V t-w1 Compression is performed according to compression method 2 (4-bit quantization). t-w2 ,V t-w2 Discard, delete from the history Kvcache, and no longer available in subsequent reasoning.

[0192] Referring to FIG7D , this embodiment has three compression levels for compressing data in the memory. The first compression level is 8-bit quantization, and the compression window size is w1. The second compression level is 4-bit quantization, and the compression window size is w2. The third compression level is discarding. For a certain Self Attention layer, the input at the current moment is x t , new token x t Input Self Attention layer to generate Q t K t V t Matrix. Decompress the historical KV cache to generate KV_cache_dequant: for (K i＜t-w1 ∩K i∈Attkeep ), (V i＜t-w1 ,∩V i∈Attkeep ) Decompress according to compression method 2 (4-bit inverse quantization), K i∈Attkeep Indicates that the attention score meets the preset conditions, for example, the score is the highest K, for (K t-w1≤i＜t ∩K i∈Attkeep ), (V t-w1≤i＜t ,∩V i∈Attkeep ) Decompress according to compression method 1 (8bit dequantization), and KV_cache_dequant and the current K t V tCat together to generate a new KV cache to perform self attention calculation. Compress the current KV and store it in the historical KV cache. Specifically, you can t V t Compression is performed according to compression method 1 (8-bit quantization). t-w1 ,V t-w1 Compress according to compression method 2 (4-bit quantization). Discard, delete from the history Kvcache, and no longer available in subsequent reasoning.

[0193] In addition, for the case where the compression level is not updated for each token, but is processed once every M tokens, two compression levels are included, the first compression level: 8-bit quantization, compression window size is w1. The second compression level: 4-bit quantization (for the remaining tokens). For a certain Self Attention layer, the current input is x t , decompress the historical KV_cache according to the compression level of each token itself, and the current K t V t The new KV cache is generated and self-attention is performed. The current KV is stored in the historical KV cache. Specifically, if t is a multiple of M: the KV cache of all tokens in the first compression window is quantized to 8 bits and the compression level is updated. The KV cache of all tokens in the second compression window is quantized to 4 bits and the compression level is updated.

[0194] 8 , which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. As shown in FIG8 , a data processing device provided in an embodiment of the present application, the device 800 includes:

[0195] An acquisition module 801 is configured to acquire first data, where the first data is an intermediate result obtained by performing a calculation on the first input data using a machine learning model.

[0196] For a detailed description of the acquisition module 801 , reference may be made to the description of step 501 in the above embodiment, which will not be repeated here.

[0197] Processing module 802 is used to compress the first data to obtain first compressed data, and store the first compressed data in a memory; read the first compressed data from the memory and decompress the first compressed data to obtain decompressed first compressed data, the decompressed first compressed data and second data are used to input into the machine learning model, and the second data is obtained based on the second input data; and compress the first compressed data or the decompressed first compressed data to obtain second compressed data, and store the second compressed data in the memory, the accuracy loss of the first compressed data compared to the first data is less than the accuracy loss of the second compressed data compared to the first data.

[0198] For a detailed description of the processing module 802 , reference may be made to the description of steps 502 and 503 in the above embodiment, which will not be repeated here.

[0199] In a possible implementation, the first data is specifically an intermediate result obtained by calculating the attention weight using a machine learning model based on the first input data; or,

[0200] The first data is specifically obtained by compressing the intermediate result; or,

[0201] The first data is specifically obtained by compressing and decompressing the intermediate result.

[0202] In one possible implementation, the second data is obtained by compressing the second input data, or the second data is an intermediate result obtained through a machine learning model based on the second input data.

[0203] In a possible implementation, the processing module 802 is specifically configured to: after reading the first compressed data from the memory and decompressing the first compressed data, perform compression processing on the first compressed data or the decompressed first compressed data to obtain second compressed data.

[0204] In a possible implementation, the first data is key data or value data, and the second data is key data or value data.

[0205] In a possible implementation, the processing module 802 is specifically configured to:

[0206] The first compressed data stored in the memory is replaced by the second compressed data.

[0207] In one possible implementation, the second input data is data currently input to the machine learning model, and the second input data is data input to the machine learning model M th time after the first input data. The processing module 802 is specifically configured to:

[0208] When at least one of the following conditions is met, the second compressed data is stored in the memory:

[0209] The M is greater than a first preset value;

[0210] The attention score obtained by inputting into the machine learning model meets the preset conditions.

[0211] In a possible implementation, the M is greater than a first preset value, including: the M is greater than the first preset value and less than a second preset value.

[0212] In a possible implementation, before storing the second compressed data in the memory, the processing module 802 is further configured to:

[0213] storing the second data in the memory;

[0214] The processing module is specifically used to:

[0215] When the amount of uncompressed data in the intermediate results obtained by the machine learning model stored in the memory reaches a third preset value, the first compressed data stored in the memory is replaced with the second compressed data, and the uncompressed data includes the second data.

[0216] In a possible implementation, the processing module 802 is further configured to:

[0217] The second data is compressed to obtain third compressed data, and the third compressed data is stored in the memory, wherein the precision loss of the third compressed data compared to the second data is less than the precision loss of the second compressed data compared to the first data.

[0218] In a possible implementation, when the third compressed data is stored in a memory, the data stored in the memory includes the third compressed data and the second compressed data.

[0219] In a possible implementation, after compressing the first compressed data or the decompressed first compressed data to obtain the second compressed data, the processing module 802 is further configured to:

[0220] Reading second compressed data from the memory and decompressing the second compressed data to obtain decompressed second compressed data, wherein the decompressed second compressed data and third data are input into the machine learning model, wherein the third data is obtained based on the third input data;

[0221] The second compressed data or the decompressed second compressed data is compressed to obtain fourth compressed data and stored in the memory; the precision loss of the second compressed data compared to the first data is less than the precision loss of the fourth compressed data compared to the first data.

[0222] In a possible implementation, the acquisition module is specifically configured to:

[0223] Based on the first input data, the attention layer of the machine learning model is used to perform calculations to obtain first data, where the first data is an intermediate result obtained by performing calculations on the machine learning model.

[0224] Next, a terminal device provided in an embodiment of the present application is introduced. Please refer to Figure 9. Figure 9 is a structural diagram of a terminal device provided in an embodiment of the present application. The terminal device 900 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Specifically, the terminal device 900 includes: a receiver 901, a transmitter 902, a processor 903 and a memory 904 (wherein the number of processors 903 in the terminal device 900 can be one or more, and Figure 9 takes one processor as an example), wherein the processor 903 may include an application processor 9031 and a communication processor 9032. In some embodiments of the present application, the receiver 901, the transmitter 902, the processor 903 and the memory 904 may be connected via a bus or other means.

[0225] The memory 904 may include a read-only memory and a random access memory, and provides instructions and data to the processor 903. A portion of the memory 904 may also include non-volatile random access memory (NVRAM). The memory 904 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0226] Processor 903 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0227] The methods disclosed in the above embodiments of the present application can be applied to or implemented by the processor 903. The processor 903 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 903 or by software instructions. The above processor 903 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 903 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 904. Processor 903 reads information from memory 904 and, in conjunction with its hardware, completes the steps involved in the model training or model inference process in the above method.

[0228] Receiver 901 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 902 can be used to output digital or character information through the first interface. Transmitter 902 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 902 can also include a display device such as a display screen.

[0229] The embodiment of the present application also provides a server. Please refer to Figure 10. Figure 10 is a schematic diagram of the structure of the server provided in the embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances. It may include one or more central processing units (CPU) 1010 (for example, one or more processors) and memory 1032, and one or more storage media 1030 (for example, one or more mass storage devices) for storing application programs 1042 or data 1044. Among them, the memory 1032 and the storage medium 1030 can be temporary storage or permanent storage. The program stored in the storage medium 1030 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1010 can be configured to communicate with the storage medium 1030 to execute a series of instruction operations in the storage medium 1030 on the server 1000.

[0230] The server 1000 may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1058; or one or more operating systems 1041, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0231] In an embodiment of the present application, the central processing unit 1010 is used to execute actions related to model training or model reasoning in the above embodiments.

[0232] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0233] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0234] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0235] Specifically, see Figure 11, which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 1100. NPU 1100 is mounted on the host CPU (host CPU) as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1103, which is controlled by controller 1104 to extract matrix data from memory and perform multiplication operations.

[0236] In some implementations, the arithmetic circuit 1103 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1103 is a two-dimensional systolic array. The arithmetic circuit 1103 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1103 is a general-purpose matrix processor.

[0237] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1102 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1101 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1108.

[0238] Unified memory 1106 is used to store input and output data. Weight data is directly transferred to weight memory 1102 through the Direct Memory Access Controller (DMAC) 1105. Input data is also transferred to unified memory 1106 through the DMAC.

[0239] BIU stands for Bus Interface Unit, i.e., bus interface unit 1110 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1109 .

[0240] The bus interface unit 1110 (BIU) is used for the instruction fetch memory 1109 to obtain instructions from the external memory, and is also used for the storage unit access controller 1105 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0241] DMAC is mainly used to move input data in the external memory DDR to the unified memory 1106 or to move weight data to the weight memory 1102 or to move input data to the input memory 1101.

[0242] The vector calculation unit 1107 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1103, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0243] In some implementations, the vector calculation unit 1107 can store the processed output vector in the unified memory 1106. For example, the vector calculation unit 1107 can apply a linear function or a nonlinear function to the output of the operation circuit 1103, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1107 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1103, for example, for use in subsequent layers in a neural network.

[0244] An instruction fetch buffer 1109 connected to the controller 1104 is used to store instructions used by the controller 1104;

[0245] Unified memory 1106, input memory 1101, weight memory 1102, and instruction fetch memory 1109 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0246] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0247] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0248] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0249] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0250] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining first data, where the first data is obtained through a machine learning model based on first input data; Compressing the first data to obtain first compressed data, and storing the first compressed data in a memory; Reading the first compressed data from the memory and decompressing the first compressed data to obtain decompressed first compressed data, wherein the decompressed first compressed data and second data are input into the machine learning model, wherein the second data is obtained based on the second input data; and The first compressed data or the decompressed first compressed data is compressed to obtain second compressed data, and the second compressed data is stored in the memory, wherein the precision loss of the first compressed data compared to the first data is less than the precision loss of the second compressed data compared to the first data.

2. The method according to claim 1, characterized in that The first data is specifically an intermediate result obtained by calculating the attention weight using a machine learning model based on the first input data; or The first data is specifically obtained by compressing the intermediate result; or, The first data is specifically obtained by compressing and decompressing the intermediate result.

3. The method according to claim 1 or 2, characterized in that The second data is obtained by compressing the second input data, or the second data is an intermediate result obtained through a machine learning model based on the second input data.

4. The method according to any one of claims 1 to 3, characterized in that: The second input data is data input into the machine learning model after the first input data.

5. The method according to any one of claims 1 to 4, characterized in that: The first data is key data or value data, and the second data is key data or value data.

6. The method according to any one of claims 1 to 5, characterized in that: Storing the second compressed data in the memory includes: The first compressed data stored in the memory is replaced by the second compressed data.

7. The method according to any one of claims 1 to 6, characterized in that: The second input data is data currently input to the machine learning model, and the second input data is data input to the machine learning model M th time after the first input data, and storing the second compressed data in the memory includes: When at least one of the following conditions is met, the second compressed data is stored in the memory: The M is greater than a first preset value; or, The attention score obtained by inputting into the machine learning model meets the preset conditions.

8. The method according to claim 7, characterized in that The M is greater than a first preset value, including: The M is greater than a first preset value and less than a second preset value.

9. The method according to any one of claims 1 to 8, characterized in that: Before storing the second compressed data in the memory, the method further includes: storing the second data in the memory; Storing the second compressed data in the memory includes: When the amount of uncompressed data in the intermediate results obtained by the machine learning model stored in the memory reaches a third preset value, the first compressed data stored in the memory is replaced with the second compressed data, and the uncompressed data includes the second data.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: The second data is compressed to obtain third compressed data, and the third compressed data is stored in the memory, wherein the precision loss of the third compressed data compared to the second data is less than the precision loss of the second compressed data compared to the first data.

11. The method according to claim 10, characterized in that The data stored in the memory includes the third compressed data and the second compressed data.

12. The method according to any one of claims 1 to 11, characterized in that: After compressing the first compressed data or the decompressed first compressed data to obtain second compressed data, the method further includes: Reading the second compressed data from the memory and decompressing the second compressed data to obtain decompressed second compressed data, wherein the decompressed second compressed data and third data are input into the machine learning model, wherein the third data is obtained based on the third input data; The second compressed data or the decompressed second compressed data is compressed to obtain fourth compressed data and stored in the memory; the precision loss of the second compressed data compared to the first data is less than the precision loss of the fourth compressed data compared to the first data.

13. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire first data, where the first data is obtained through a machine learning model based on first input data; A processing module is used to compress the first data to obtain first compressed data, and store the first compressed data in a memory; read the first compressed data from the memory and decompress the first compressed data to obtain decompressed first compressed data, the decompressed first compressed data and second data are used to input into the machine learning model, and the second data is obtained based on the second input data; and compress the first compressed data or the decompressed first compressed data to obtain second compressed data, and store the second compressed data in the memory, the accuracy loss of the first compressed data compared to the first data is less than the accuracy loss of the second compressed data compared to the first data.

14. The device according to claim 13, characterized in that The first data is specifically an intermediate result obtained by calculating the attention weight using a machine learning model based on the first input data; or The first data is specifically obtained by compressing the intermediate result; or, The first data is specifically obtained by compressing and decompressing the intermediate result.

15. The device according to claim 13 or 14, characterized in that The second data is obtained by compressing the second input data, or the second data is an intermediate result obtained through a machine learning model based on the second input data.

16. The device according to any one of claims 13 to 15, characterized in that The first data is key data or value data, and the second data is key data or value data.

17. The device according to any one of claims 13 to 16, characterized in that The processing module is specifically used to: The first compressed data stored in the memory is replaced by the second compressed data.

18. The device according to any one of claims 13 to 17, characterized in that The second input data is data currently input to the machine learning model, and the second input data is data input to the machine learning model M th time after the first input data. The processing module is specifically configured to: When at least one of the following conditions is met, the second compressed data is stored in the memory: The M is greater than a first preset value; The attention score obtained by inputting into the machine learning model meets the preset conditions.

19. The device according to claim 18, characterized in that The M is greater than a first preset value, including: the M is greater than the first preset value and less than a second preset value.

20. The device according to any one of claims 13 to 19, characterized in that Before storing the second compressed data in the memory, the processing module is further configured to: storing the second data in the memory; The processing module is specifically used to: When the amount of uncompressed data in the intermediate results obtained by the machine learning model stored in the memory reaches a third preset value, the first compressed data stored in the memory is replaced with the second compressed data, and the uncompressed data includes the second data.

21. The device according to any one of claims 13 to 20, characterized in that The processing module is further configured to: The second data is compressed to obtain third compressed data, and the third compressed data is stored in the memory, wherein the precision loss of the third compressed data compared to the second data is less than the precision loss of the second compressed data compared to the first data.

22. The device according to claim 21, characterized in that The data stored in the memory includes the third compressed data and the second compressed data.

23. The device according to any one of claims 13 to 22, characterized in that After compressing the first compressed data or the decompressed first compressed data to obtain second compressed data, the processing module is further configured to: Reading second compressed data from the memory and decompressing the second compressed data to obtain decompressed second compressed data, wherein the decompressed second compressed data and third data are input into the machine learning model, wherein the third data is obtained based on the third input data; compressing the second compressed data or the decompressed second compressed data to obtain fourth compressed data and storing the fourth compressed data in the memory; The precision loss of the second compressed data compared to the first data is smaller than the precision loss of the fourth compressed data compared to the first data.

24. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers or processors, cause the one or more computers or processors to perform the method of any one of claims 1 to 12.

25. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on a computer device or a processor, cause the computer device or the processor to execute the method according to any one of claims 1 to 12.

26. A system comprising at least one processor and at least one memory; The at least one processor and the at least one memory are connected via a communication bus; The at least one memory is used to store code; The at least one processor is configured to execute the code to perform the method according to any one of claims 1 to 12.

27. A chip comprising a processor, characterized in that: The processor is used to support a data processing device to implement the method according to any one of claims 1 to 12.

Images