Data processing method and electronic device

Abstract

The application discloses a data processing method and an electronic device, and relates to the technical field of large language models. The data processing method comprises the following steps: acquiring input data of a preset large language model. In a model inference process of the large language model based on the input data, the large language model generates first output data, and Value and Key corresponding to the first output data are stored in a preset cache according to a reference position. The reference position comprises the coordinate position of Value and Key corresponding to second output data generated by the previous model inference in the preset cache. The storage structure of the preset cache adopts an image body structure, and the image body structure provides the coordinate position of the stored data. In the application, the operation of copying historical Value and Key from the preset cache is avoided, thereby shortening the time consumption of reading and splicing the historical Value and Key, shortening the model inference time, and improving the model inference speed.

Description

[0001] The present application claims priority from the Chinese Patent Application No. 202410063860.3 filed on January 16, 2024, and entitled "A data optimization method", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the technical field of large language model, and particularly relate to a data processing method and an electronic device. BACKGROUND

[0003] A key-value cache (KV Cache) is a cache mechanism for storing key-value pair data. In the inference process of a large language model, the large language model calculates tokens one by one when outputting a decoder, and after calculating a new token, the previous sequence is added to form a new input sequence for the calculation of a new token. The process is repeated until the output token is a stop symbol token. The token sequence of the current sequence length seq_len is converted by a weight matrix to obtain Value, Key, and Query matrices. In this process, the Value, Key, and Query matrices of the token sequence of length seq_len-1 have been obtained in the previous step, and their values will not change, so only the Value, Key, and Query values of the newly generated token need to be converted, and the previous Value, Key, and Query matrices are spliced to obtain the Value, Key, and Query matrices of the current complete token sequence. In the process of the new token, only the current Query and complete Vaule and Key are related, that is, the KV-cache caches the Value and Key of the previous inference, thereby achieving the effect of reducing calculation and accelerating inference.

[0004] However, as the Value and Key in the KV Cache increase, the amount of data for each splicing also increases, the time for reading data from the KV Cache and splicing increases, and the inference delay of the large language model increases. SUMMARY

[0005] The embodiment of the application provides a data processing method and an electronic device, in a model inference process of a large language model, a preset cache of an image body structure is constructed, a reference position based on the preset cache is realized, generated output data is stored in the preset cache, an operation of copying historical output data from the preset cache is avoided, and therefore the time consumption of reading and splicing of the output data is shortened, the time of model inference is shortened, and the speed of model inference is improved.

[0006] To achieve the above object, the embodiment of the application adopts the following technical scheme:

[0007] In a first aspect, a data processing method is provided, comprising:

[0008] Input data of a preset large language model is obtained. In a model inference process of the large language model based on the input data, the large language model generates first output data, and Value and Key corresponding to the first output data are stored in a preset cache according to a reference position. The reference position includes the coordinate position of Value and Key corresponding to second output data generated by the previous model inference of generating the first output data in the preset cache.

[0009] The storage structure of the preset cache adopts an image body structure, and the image body structure provides the coordinate position of the stored data.

[0010] In the application, since the storage structure of the preset cache adopts the image body structure, the image body structure can provide the coordinate position of the stored data. In the model inference process, when Value and Key corresponding to each new output data are stored in the preset cache, Value and Key corresponding to the new output data can be directly stored according to the reference position. Compared with the storage of Value and Key corresponding to the output data in the prior art based on the preset cache with a one-dimensional linear structure such as buffer, the previous Value and Key in the preset cache need to be copied, and with the increase of the inference number, the historical Value and Key to be copied will gradually increase, and the time consumption of copying will become larger and larger. In the application, the historical Value and Key in the preset cache do not need to be copied out for splicing, the data access amount of the preset cache is reduced, and the time consumption caused by accessing the preset cache for output data copying is shortened, and the speed of model inference is improved.

[0011] In a possible implementation manner of the first aspect, the model inference process of the large language model is a model inference process based on a multi-head attention mechanism; and before Value and Key corresponding to the first output data are stored in the preset cache according to the reference position, the method further comprises:

[0012] According to preset model parameters of the large language model, structure parameters corresponding to the preset cached image body structure are obtained.

[0013] The preset model parameters include the length of the attention head and the number of the attention head, and the structure parameters include the width of the image body and the depth of the image body structure, the depth corresponding to the number of the attention head.

[0014] Based on the structure parameters of the image body structure, the preset cached image body structure is constructed, and the initial sequence length of the image body structure is 0.

[0015] Then, according to the reference position, the Value and the Key corresponding to the first output data are stored in the preset cache, including:

[0016] According to the reference position in the preset cached image body structure, the Value and the Key corresponding to the first output data are stored in the target position of the preset cached image body structure, wherein the target position is the reference position, or the target position is the next position of the reference position in the image body structure.

[0017] In the present application, according to the reference position in the image body structure, the Value and the Key corresponding to the first output data are stored in the preset cache, without copying the existing Value and Key in the preset cache for data splicing, thereby reducing the data access amount of the preset cache, shortening the time consumption caused by accessing the preset cache for output data copying, and improving the speed of model inference.

[0018] In another possible implementation of the first aspect, according to the reference position, the Value and the Key corresponding to the first output data are stored in the preset cache, including:

[0019] According to the reference position of each attention head of the image body structure, the Value and the Key corresponding to the first output data are stored in the target position of each attention head of the image body structure in parallel.

[0020] In the present application, the storage of the Value and the Key corresponding to the output data of each attention head is processed in parallel, which can further speed up the speed of model inference.

[0021] In another possible implementation of the first aspect, the first output data is the first output data generated by the large language model in model inference, the reference position is the position corresponding to the origin coordinate of the image body structure, and the target position is the reference position.

[0022] Then, according to the reference position in the preset cached image body structure, the Value and the Key corresponding to the first output data are stored in the target position of the preset cached image body structure, including:

[0023] store the Value and the Key corresponding to the first output data to a position corresponding to the origin coordinate of the preset cache image body structure.

[0024] In the present application, when the first output data is output data generated by the first model inference, the Value and the Key corresponding to the first output data can be directly stored to the position corresponding to the origin coordinate of the image body structure, simplifying the process of storing output data into the preset cache.

[0025] In another possible implementation of the first aspect, the first output data is non-first output data generated by the large language model in model inference, and the reference position is a coordinate position of the Value and the Key corresponding to the second output data in the image body structure of the preset cache.

[0026] According to the reference position in the preset cache image body structure, the Value and the Key corresponding to the first output data are stored into the target position of the preset cache image body structure, including:

[0027] Then, according to the sequence length of the image body structure, a next position of the reference position in the image body structure is obtained; and the Value and the Key corresponding to the first output data are stored into the next position of the reference position of the preset cache image body structure.

[0028] In the present application, when the first output data is non-first output data generated by the model inference, the Value and the Key corresponding to the first output data can be directly stored into the next position of the image body structure according to the next position of the reference position in the image body structure, without the need to copy the existing Value and Key in the preset cache for data splicing, thereby reducing the data access amount of the preset cache, shortening the time consumption caused by accessing the preset cache for output data copying, and improving the speed of model inference.

[0029] In another possible implementation of the first aspect, after the Value and the Key corresponding to the first output data are stored into the target position of the preset cache image body structure according to the reference position of the preset cache image body structure, the method further includes:

[0030] The sequence length of the preset cache image body structure is increased by 1.

[0031] In the present application, after the Value and the Key corresponding to the first output data are stored into the preset cache image body structure each time, the sequence length of the image body structure is updated in time, so that in subsequent model inference, the reference position can be determined according to the sequence length for storage of the Value and the Key corresponding to the first output data next time, thereby simplifying the process of storing output data into the preset cache.

[0032] In a possible implementation form of the first aspect, the preset cache image structure is constructed based on the structure parameters of the image structure, and the construction includes:

[0033] The preset cache image structure is constructed in a direction of a sequence length as a first dimension, a direction of a width of the image structure as a second dimension, and a direction of an attention head arrangement in the image structure as a third dimension. A coordinate system corresponding to the image structure is a three-dimensional coordinate system constructed based on the first dimension, the second dimension, and the third dimension, with an intersection point of the second dimension and the third dimension as a coordinate origin.

[0034] The width of the image structure is the same as the length of the attention head of the large language model, and the number of the attention heads of the image structure is the same as the number of the attention heads of the large language model.

[0035] In the present application, the image structure corresponding to the three-dimensional data is constructed based on the three-dimensional structure of the large language model, so that the image structure can better store the Value and Key corresponding to the output data obtained by each model inference of the large language model, and the problem of time-consuming model inference caused by data traversal, data copying, and data splicing in the one-dimensional cache structure is avoided.

[0036] In a possible implementation form of the first aspect, after the Value and the Key corresponding to the first output data are stored in the preset cache according to the reference position, the method further includes:

[0037] According to the Value and the Key corresponding to the first output data to the last output data corresponding to the input data in the preset cache image structure, the output result corresponding to the input data is generated.

[0038] In the present application, the Value and the Key corresponding to the output data generated by each model inference are stored in the preset cache, and the first output data corresponding to the input data to the last output data corresponding to the input data are combined to form the output result, so that the process of obtaining the output result avoids repeated calculation and improves the speed of model inference.

[0039] In a second aspect, an electronic device is provided, which includes a memory and one or more processors; the memory is coupled to the processor; the memory stores computer program code, which includes computer instructions, when the computer instructions are executed by the processor, the electronic device executes the method in any one of the above first aspect.

[0040] In a third aspect, a computer readable storage medium is provided, which stores instructions, when the instructions are run on an electronic device, the electronic device can execute the method in any one of the above first aspect.

[0041] In a fourth aspect, a computer program product containing instructions, which, when executed on an electronic device, enable the electronic device to perform the method of any one of the first aspect.

[0042] In a fifth aspect, an embodiment of the present application provides a chip, the chip comprising a processor, the processor being configured to invoke a computer program in a memory to perform the method of the first aspect.

[0043] It can be understood that the beneficial effects achievable by the electronic device of the second aspect, the computer readable storage medium of the third aspect, the computer program product of the fourth aspect, and the chip of the fifth aspect can refer to the beneficial effects of the first aspect and any possible design of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A structure diagram of a buffer structure provided by an embodiment of the present application;

[0045] Figure 2 A process diagram of Value and Key splicing under a buffer structure provided by an embodiment of the present application;

[0046] Figure 3 A structure diagram of a three-dimensional image body structure provided by an embodiment of the present application;

[0047] Figure 4 A diagram of an image body structure and coordinates provided by an embodiment of the present application;

[0048] Figure 5 A diagram of an image body structure provided by an embodiment of the present application;

[0049] Figure 6 A diagram of another image body structure provided by an embodiment of the present application;

[0050] Figure 7 A diagram of newly added Value and Key in an image body structure provided by an embodiment of the present application;

[0051] Figure 8 A hardware structure diagram of an electronic device provided by an embodiment of the present application;

[0052] Figure 9 A possible structure diagram of an electronic device provided by an embodiment of the present application;

[0053] Figure 10 A structure diagram of a chip system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing particular embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims, the singular forms “a,” “an” and “the” are intended to include both singular and plural forms, unless the context clearly indicates otherwise. It will be understood that the terms “at least one of’ or “one or more of’ when used in the context of the following embodiments, means one, two, three, or more (including two). The term “and / or” used in the context of the following embodiments refers to three relationships; for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character “ / ” generally represents an “or” relationship between the associated objects.

[0055] In this specification, the phrase “one embodiment” or “some embodiments” etc. means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, appearances of the phrases “in one embodiment”, “in some embodiments”, “in other embodiments”, “in additional embodiments”, etc. in various places in the specification are not necessarily all referring to the same embodiment, unless otherwise specified. The terms “comprises”, “comprising”, “has”, “having”, “includes” and “including” means “including but not limited to” unless otherwise specified. The term “connected” includes both direct and indirect connections unless otherwise specified. “First”, “second”, etc. are used only to describe one entity from another, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0056] In the embodiments of the present application, the words “exemplary” and “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as “exemplary” or “for example” is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words “exemplary” and “for example” is intended to present concepts in a concrete manner. In the embodiments of the present application, the terms “first”, “second”, etc. are used to describe various objects and are not necessarily used to indicate or imply relative importance or implicitly indicate the number of the indicated technical features.

[0057] Large language models (LLMs) are deep learning models trained on vast amounts of text data. LLMs can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks such as text summarization, question answering, translation, etc. Each layer of the LLM is a self-attention neural network model (Transformer). Transformer is a model that uses attention mechanism to improve the speed of model training. Transformer consists of two parts: encoder and decoder. The encoder is responsible for obtaining the encoding information matrix of all tokens based on the vector matrix of the incoming token. The decoder is responsible for predicting the next token based on the current translated token according to the encoding information matrix. The process of encoder and decoder can also be understood as the model inference process. The process of encoder and decoder includes attention step and feed-forward step, which are used to update the state of each token in the input channel of the Transformer. In the attention step, the token will "look around" to find other tokens with relevant context and share information with each other. In the feed-forward step, each token will "think" about the information collected in the previous attention step and try to predict the next token.

[0058] In the model inference process, the historically generated tokens do not need to produce attention with the subsequent tokens, therefore, in the model inference stage of the large language model, the corresponding Value and Key of the generated token can be cached. Specifically, the large language model calculates one token at a time when outputting the decoder, and after calculating a new token, it is added to the previous sequence to form a new input sequence for the calculation of the new token. Cycle this process until the output token is the stop symbol token. The token sequence of the current sequence length seq_len is converted by the weight matrix to obtain Value, Key, and Query matrices. In this process, the Value, Key, and Query matrices converted by the token sequence of length seq_len-1 have been obtained in the previous step of calculation, and their values will not change, therefore, only the Value, Key, and Query values of the newly generated token need to be converted, and the previous Value, Key, and Query matrices are spliced to obtain the Value, Key, and Query matrices of the current complete token sequence. In the process of generating a new token, only the current Query and complete Vaule and Key are related, that is, the KV-cache caches the Value and Key of the previous round of inference, thereby achieving the effect of reducing calculation and accelerating inference.

[0059] After the KV cache is configured, the model inference process of the large language model can be divided into two stages: the pre-padding stage and the KV cache usage stage. Among them, the pre-padding stage refers to the process before the first output token is calculated in the model inference process, at this time the KV cache is empty, when the token sequence is converted by the weight matrix to obtain Value, Key, and Query matrices, the calculated Value and Key are stored in the KV cache, and the KV cache is filled. The KV cache usage stage refers to the process of calculating the second output token to the last output token in the model inference process, the KV cache is not empty, and in each round of inference, when a new token is obtained, the Value, Key, and Query corresponding to the token are calculated, the Value and Key calculated in the previous round are read from the KV cache, and the Value and Key of the current round of token are spliced with the Value and Key calculated in the previous round.

[0060] An example is given to illustrate the use of KV cache in the model inference process of the large language model.

[0061] Assuming that the user gives an input data "Hello", the large language model generates the output data "Hello" through the first model inference. Based on the output data of the first model inference, the large language model generates the output result for "Hello" through N times of model inference. For example, the output result is "Hello, I am intelligent assistant xx, what can I help you with". In the model inference process of the large language model, one model inference (referred to as inference) generates one token, and the token output by each inference is calculated to obtain the Value, Key and Query corresponding to the token, the Value and Key calculated in the last time are read from the KV cache, and the Value and Key calculated in the current time are spliced with the Value and Key calculated in the last time to be stored in the KV cache. Until the token of the stop symbol is inferred, the output result for the input data is obtained. Since each generated token is spliced with the previously generated tokens as the input data for the next inference, in this embodiment, the tokens obtained by inference and stored in the cache can be collectively referred to as input tokens. The input tokens in this embodiment include tokens corresponding to the input data and tokens generated by each inference in the inference process (output data).

[0062] For example, in the i-th inference process, the input tokens are:

[0063] "Hello, I am intelligent assistant xx".

[0064] The i-th inference generates a new token "please" according to the input tokens. Since the Value and Key corresponding to "Hello, I am intelligent assistant xx" have been stored in the KV cache, the i-th inference only needs to calculate the Value, Key and Query corresponding to the new token "please". It is not necessary to calculate the Value and Key corresponding to these tokens again, but can directly read the Value and Key corresponding to these tokens from the KV cache, splice the Value and Key corresponding to the new token "please" obtained by the current inference with the Value and Key corresponding to the tokens read from the KV cache, and obtain the Value, Key and Query matrix corresponding to the complete token sequence after the current inference. Wherein, the input tokens "Hello, I am intelligent assistant xx" and the new token "please" generated by the current inference are spliced to obtain the complete token sequence after the current inference:

[0065] "Hello, I am intelligent assistant xx, please".

[0066] In the (i+1)th inference process, the Value, Key and Query corresponding to the new token "ask" are calculated. The Value and Key corresponding to the input tokens are read from the KV cache, and the Value and Key corresponding to the new token "ask" obtained in this inference are spliced with the Value and Key corresponding to the tokens read from the KV cache to obtain the Value, Key and Query matrix corresponding to the complete token sequence after this inference. Among them, before this inference, the input tokens (complete token sequence) are:

[0067] "Hello, I am intelligent assistant xx, please";

[0068] The input tokens "Hello, I am intelligent assistant xx, please" are spliced with the new token "ask" generated in this round of inference to obtain the complete token sequence after this inference:

[0069] "Hello, I am intelligent assistant xx, please ask".

[0070] After N times of inference, after obtaining the new tokens "have", "what", "what", "can", "help", "help", "you", "your" respectively, and completing the splicing of the new tokens and the input tokens respectively, the output result composed of all input tokens is obtained:

[0071] "Hello, I am intelligent assistant xx, please ask what can help you".

[0072] During model inference, since the Value and Key corresponding to the last round of tokens are cached in the KV cache, the past Value and Key do not need to be calculated repeatedly during the prediction of new tokens, saving the calculation resources in the subsequent token prediction process.

[0073] Generally, the KV cache adopts a buffer structure. The buffer structure is a one-dimensional structure, which can also be understood as a piece of continuous memory, and all data stored in the KV cache is in a spliced state. The Value and Key of the self-attention (self-attention) in the large language model structure include the sequence length, the length of the attention head and the number of attention heads. The storage method of the one-dimensional buffer for Value and Key data is to copy and splice the Value and Key data in the sequence length direction. Reference Figure 1 , Figure 1A structural diagram of a buffer structure is given. Taking the number of attention heads in the attention structure of a large language model as 32 and the length of the attention heads as 128 as an example, the buffer structure is illustrated. After two inferences, Figure 1 The buffer structure shown has Value and Key with a sequence length of 2 stored in each attention head. The buffer has a sequence length of 2*128*32.

[0074] Due to the one-dimensional structure and the continuous memory characteristics of the buffer, in the KV cache based on the buffer structure, when generating new Value and Key for splicing, the generated Value and Key values need to be copied to the new buffer in sequence according to the sequence, and the newly generated Value and Key values are stored in the structure. Repeat the operation until all the data of the attention heads are spliced. For example, refer to Figure 2 , Figure 2 A process diagram for splicing Value and Key corresponding to tokens in the buffer structure is given. In combination with Figure 1 The structural diagram of the buffer structure given in Figure 2 has Value and Key corresponding to 2 tokens. When predicting a new token, the Value and Key data of the first attention head need to be copied to the new buffer structure (which can also be understood as a new cache), and the data of the first attention head of the Value and Key generated when predicting the new token is spliced after the first attention head of the previous round of Value and Key. In turn, when copying the Value and Key data of the second attention head, it is spliced after the Value and Key data of the first attention head. The data of the first attention head of the Value and Key data of the new token is spliced after the second attention head of the previous round of Value and Key data, and so on, until the data of the first attention head of the Value and Key data of the new token is spliced after the Value and Key data of the 32nd attention head. Finally, the Value and Key of this inference are obtained, which are stored in the one-dimensional linear buffer structure after data splicing.

[0075] Obviously, in the application of the above KV cache based on the one-dimensional linear structure of buffer, the model inference needs to copy the Value and Key of the last round in the KV cache every time when splicing the newly generated token Value and Key with the Value and Key of the last round. With the increase of the number of inferences, the Value and Key to be copied will gradually increase, and the time-consuming of copying will become larger and larger, resulting in slower model inference speed and increased model inference time.

[0076] The present application provides a data processing method, and a three-dimensional image body structure is used for preset caching. The image body structure can provide the coordinate position of the stored output data. In the model inference process, the coordinate position of the Value and Key in the KV cache can be directly obtained, so that the new Value and Key value is spliced into the Value and Key of the last round according to the coordinate position. The Value and Key of the last round in the KV cache does not need to be copied, the data access amount of the KV cache is reduced, the newly generated Value and Key is cached based on the coordinate position of the data in the image body structure, the time-consuming caused by accessing the KV cache and copying the Value and Key is shortened, and the model inference speed is improved.

[0077] The following embodiments illustrate the image body structure used by the KV cache in the embodiments of the present application.

[0078] The image body structure can also be referred to as an image object. The image object is a special memory object used to represent image data, and the image object supports direct access to memory data through coordinates. That is, data caching based on the image body structure can quickly locate and operate the cached data in the image body structure. The image object supports the creation of one-dimensional data, two-dimensional data, three-dimensional data and high-dimensional data. For example, Figure 3 A structural diagram of a three-dimensional image body structure is given. In the scenario of storing image data using a three-dimensional image body structure (image object), the three-dimensional parameters of the image object include the width of the image (image width), the height of the image (height) and the depth of the image (depth). Based on the image body structure, a three-dimensional coordinate can be constructed, so that the image data at the position of each pixel can be accessed through the coordinate of each pixel.

[0079] In some embodiments, based on the image body structure providing the coordinate position for data access, the KV cache using the image body structure is applied to the model inference process of the large language model to improve the model inference speed. It should be noted that the large language model involved in the embodiments of the present application is a model that performs model inference based on a multi-head attention mechanism. Under the multi-head attention mechanism, the preset model parameters of the large language model include the length of the attention head and the number of attention heads, etc. The embodiments of the present application provide a data processing method, which includes:

[0080] S201, obtaining the structure parameters of the image body structure corresponding to the preset cache according to the preset model parameters of the large language model.

[0081] Among them, the preset model parameters include the length of the attention head and the number of attention heads. In some embodiments, the structure parameters of the image body structure of the preset cache can be determined according to the preset model parameters of the large language model. Among them, the structure parameters of the image body structure include the width of the image body structure and the depth of the image body structure. The preset cache is a KV cache.

[0082] In some embodiments, the width of the image body structure can be the same as the length of the attention head of the large language model; the depth of the image body structure is the same as the number of attention heads of the large language model. For example, when the length of the attention head of the large language model is 128, the width of the image body structure is also 128; when the number of attention heads of the large language model is 32, the depth of the image body structure is 32.

[0083] S202, constructing the image body structure of the preset cache based on the structure parameters of the image body structure.

[0084] In the present embodiment, the image body structure is a three-dimensional structure, wherein the direction of the sequence length is the first dimension, the direction of the width of the image body structure is the second dimension, and the direction of the arrangement of the attention head in the image body structure is the third dimension. The image body structure of the preset cache is constructed.

[0085] In some embodiments, the intersection of the first dimension, the second dimension and the third dimension is taken as the coordinate origin. For example, refer to Figure 4 , Figure 4A schematic diagram of an image body structure and coordinates is given. The intersection P of the first dimension, the second dimension and the third dimension is taken as the coordinate origin to construct the coordinate system of the image body structure. The first dimension is the x-axis direction, the second dimension is the y-axis direction, and the third dimension is the z-axis direction. In the process of storing data in the preset cache based on the image body structure, the sequence length of the image body structure in the x direction increases by 1 for each output data stored in the preset cache. In some embodiments, when the preset cache does not store the output data, the initial sequence length of the image body structure is 0.

[0086] S203, obtain input data of the large language model.

[0087] In some embodiments, the input data of the large language model can be user input data or data obtained from a database. For example, in a question and answer scenario, the input data of the large language model can be user input text. Illustratively, the user input text can be "hello", and the input data input to the large language model includes two tokens "you" and "good".

[0088] S204, input the input data into the large language model for model inference to generate first output data.

[0089] In some embodiments, the large language model performs model inference based on the input data,

[0090] Suppose the user gives an input data "hello", the large language model will generate an output result for the input data "hello" after model inference. For example, the output result is "hello, I am intelligent assistant xx, what can I help you with". Among them, the output result "hello, I am intelligent assistant xx, what can I help you with" is obtained after N times of model inference of the large language model.

[0091] In each model inference process of the large language model, when a new output data (token) is obtained, the token corresponding Query, Value and Key are calculated. The Value and Key corresponding to the token this time are spliced to the Value and Key corresponding to the second output data (the token calculated in the last round) in the preset cache. The second output data is the output data generated by the previous model inference to generate the first output data.

[0092] For example, in the i-th inference process, the historical output data (tokens) is "hello, I am intelligent assistant xx, please". The preset cache has stored all the historical calculated Value and Key corresponding to the tokens. The i-th inference generates output data (token) "ask", and calculates the Query, Value and Key corresponding to "ask".

[0093] S205, store the first output data into the preset cache according to the reference position.

[0094] The reference position includes a coordinate position in the preset cache of Value and Key corresponding to second output data generated by a previous model inference generating the first output data.

[0095] There are two cases when the large language model stores Value and Key corresponding to the first output data into the preset cache during model inference. One is the case of storing Value and Key corresponding to the first output data based on the first model inference of the input data. Before storing the Value and Key corresponding to the first output data, the image body structure of the preset cache is empty, and the sequence length is 0. In this case, the reference position is the position corresponding to the origin coordinate of the image body structure.

[0096] The other case is the case of storing Value and Key corresponding to the first output data based on the non-first model inference of the input data. Before storing the Value and Key corresponding to the first output data, there are historical Value and Key in the image body structure of the preset cache, and the sequence length is not 0. In this case, the reference position is the position corresponding to the last round Value and Key (Value and Key corresponding to the second output data) of the image body structure.

[0097] Specifically, in some embodiments, Value and Key corresponding to the first output data are stored according to the image body structure of the preset cache, and S205 includes:

[0098] S2051, store Value and Key corresponding to the first output data into a target position of the image body structure of the preset cache according to the reference position in the image body structure of the preset cache.

[0099] The target position is the reference position, or the target position is the next position of the reference position in the image body structure.

[0100] In this embodiment, corresponding to the above two cases, in the case of storing Value and Key corresponding to the first output data based on the first model inference of the input data, the target position is the reference position, and Value and Key corresponding to the first output data can be stored in the reference position. In the case of storing Value and Key corresponding to the first output data based on the non-first model inference of the input data, the target position is the next position of the reference position (the position of Value and Key corresponding to the second output data), and Value and Key corresponding to the first output data are stored in the next position of the reference position.

[0101] Specifically, in some embodiments, the first output data is first output data generated by the large language model in model inference, the reference position is a position corresponding to the origin coordinate of the image body structure, and the target position is the reference position. Then S2051 includes:

[0102] The Value and the Key corresponding to the first output data are stored in the position corresponding to the origin coordinate of the preset cache image body structure.

[0103] In this embodiment, before the Value and the Key corresponding to the first output data are stored in the preset cache, the sequence length of the preset cache image body structure is 0. The reference position is the position corresponding to the origin coordinate of the image body structure. Figure 5 A schematic diagram of an image body structure is given. In reference Figure 5 (a), the first output data is output data generated by first model inference based on input data, and the position corresponding to the origin coordinate is point P. The number of attention heads of the image body structure is 32, and the width of the image body structure is 128. According to the position of point P, the Value and the Key corresponding to the first output data are stored in the image body structure along the x direction (the direction of the sequence length), and the sequence length of the image body structure is updated from 0 to 1. In reference Figure 5 (b), for example, it is assumed that a user gives an input data "hello", and the large language model generates output data (tokens) of "hello" after first model inference. The Query, Value and Key corresponding to the tokens are calculated. The Value and the Key are stored in the image body structure, and the size of the Value and the Key is 32*2*128. The sequence length is increased from 0 to 2.

[0104] In this embodiment, when the first output data is output data generated by first model inference, the Value and the Key corresponding to the first output data can be directly stored in the position corresponding to the origin coordinate of the image body structure, simplifying the process of storing output data in the preset cache.

[0105] Specifically, in some embodiments, the first output data is non-first output data generated by the large language model in model inference, and the reference position is a coordinate position of the Value and the Key corresponding to the second output data in the preset cache image body structure.

[0106] Then S2051 includes:

[0107] According to the sequence length of the image body structure, the next position of the reference position in the image body structure is obtained, and the Value and the Key corresponding to the first output data are stored in the next position of the reference position in the preset cache image body structure.

[0108] In the embodiment, before storing the Value and Key corresponding to the first output data into the preset cache, there are historical Value and Key in the image body structure of the preset cache, and the sequence length is not 0. The reference position is the coordinate position of the Value and Key corresponding to the second output data in the image body structure of the preset cache. The reference position is determined according to the sequence length of the image body structure. Figure 6 , Figure 6 Another schematic diagram of the image body structure is given. The first output data is the output data generated based on the input data by the non-first model inference, and the origin coordinate corresponding position is point P. According to the sequence length of the image body structure being 2, the reference position is determined to be M. Then, the next position of M is N accordingly. The number of attention heads of the image body structure is 32, and the width of the image body structure is 128. According to the position of N determined by point M, the Value and Key corresponding to the first output data generated by this inference are stored in the image body structure at the position of N along the x direction (the direction of the sequence length), and the sequence length of the image body structure is updated from 2 to 3. In this way, in the N inference processes in the future, the coordinate position of the Value and Key (historical Value and Key) corresponding to the second output data in the image body structure can be determined as the reference position according to the sequence length of the image body structure. Based on the next position of the reference position, the Value and Key corresponding to the first output data generated by the current inference are stored, and the sequence length of the image body structure is updated.

[0109] In the embodiment, when the first output data is the output data generated by the non-first model inference, the Value and Key corresponding to the first output data can be directly stored in the next position of the image body structure according to the next position of the reference position, without the need to copy the historical Value and Key in the preset cache for data splicing, thereby reducing the data access amount of the preset cache, shortening the time consumption caused by accessing the preset cache for output data copying, and improving the speed of model inference.

[0110] In some embodiments, the image body structure is a three-dimensional data structure corresponding to a large language model, which includes a plurality of attention heads, and S205 includes:

[0111] According to the reference position of each attention head of the image body structure, the Value and Key corresponding to the first output data are stored in the target position of each attention head of the image body structure in parallel.

[0112] In some embodiments, after one model inference, the token corresponding to the generated Value and Key are stored in the image body structure, and the Value and Key corresponding to the second output data are stored in the next position of the existing Value and Key in the sequence direction.

[0113] Reference Figure 7 , Figure 7 A schematic diagram of adding new Value and Key in the image body structure is given. Taking the coordinate system of the image body structure shown in Figure 7 As an example, the sequence length of the preset cached image structure body is 2. For each attention head in the image body structure, the Value and Key corresponding to the new token are stored based on the next position of the position of the Value and Key corresponding to the second output data in the x direction (sequence length direction) of the attention head. The splicing storage operation of multiple attention heads can be performed in parallel. The sequence length of the stored new Value and Key increases by 1, and the sequence length of the Value and Key is updated to 3.

[0114] S206, according to the preset cached image body structure, the first output data corresponding to the input data to the last output data corresponding to the input data, the output result corresponding to the input data is generated.

[0115] In this embodiment, the Value and Key data corresponding to the complete tokens sequence are stored in the preset cache. Based on the Value and Key data corresponding to the complete tokens sequence in the preset cache, the complete tokens for the input data can be obtained. For example, in the inference process of the input data "hello" of the large language model, the model inference is stopped when the large language model infers to the stop symbol. According to the conversion of the Value and Key data corresponding to the complete tokens sequence in the preset cache, the complete tokens sequence obtained includes "you", "good", "I", "am", "smart", "assistant", "hand", "x", "x", "please", "ask", "what", "can", "help", "you", "of"; that is, "hello, I am a smart assistant xx, please ask what can help you" as the output result corresponding to the input data.

[0116] This application provides a data processing method in which a preset cache adopts a three-dimensional image volume structure. The image volume structure can provide the coordinate positions of the stored output data. During model inference, for each new output data value and key generated, the value and key corresponding to the new output data can be directly cached in the preset cache based on the reference position in the image volume structure of the preset cache. This eliminates the need to copy and concatenate historical output data values ​​and keys from the preset cache, reducing data access to the preset cache and minimizing the impact of data copying and concatenation (data movement) on memory during model inference. By directly caching the newly generated output data values ​​and keys based on the coordinate positions of the values ​​and keys generated from the output data in the image volume structure, the time consumed by accessing the preset cache and copying historical values ​​and keys is shortened, thus improving the speed of model inference. For example, extensive experimental data shows that using an image volume structure KV cache for data caching during model inference can reduce the overall model inference latency by 2-3 times compared to using a buffer structure KV cache.

[0117] The data processing method provided in this application can be applied to electronic devices. The electronic devices in this application can be portable computers (such as mobile phones), tablets, laptops, personal computers (PCs), wearable electronic devices (such as smartwatches), augmented reality (AR) / virtual reality (VR) devices, in-vehicle computers, servers, server clusters, etc. The following embodiments do not impose any special limitations on the specific form of the electronic device.

[0118] For example, Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. See also... Figure 8 , Figure 8 The illustrated electronic device may include a processor 101, a memory 102, and a bus 103. The processor 101 and the memory 102 are connected via the bus 103. Optionally, in some embodiments, the electronic device 100 may further include a communication interface 104, which is connected to the processor 101 and the memory 102 via the bus 104. Optionally, in some embodiments, the electronic device 100 may further include a display screen 105, which is connected to the processor 101 and the memory 102 via the bus 103.

[0119] The processor 101 is the control center of the electronic device, and can include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), and / or the like. Different processing units can be independent devices or integrated in one or more processors.

[0120] As an example, the processor 101 can include one or more CPUs, such as the CPU 0 and the CPU 1 shown in FIG. 1B. Figure 8

[0121] The NPU is a neural-network (NN) computing processor that processes input information quickly by drawing on the structure of a biological neural network, such as the transmission mode between human brain neurons, and can also constantly self-learn. The NPU can implement intelligent cognitive applications of the electronic device 100, such as image recognition, face recognition, speech recognition, text understanding, and the like. A preset large language model can be deployed in the NPU to process input data corresponding to a user input operation to obtain an output result corresponding to the input data.

[0122] The memory 102 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0123] ​In a possible implementation, the memory 102 can exist independently of the processor 101. The memory 102 can be connected to the processor 101 through the bus 103, for storing data, instructions or program codes. When the processor 101 invokes and executes the instructions or program codes stored in the memory 102, the data processing method provided in the embodiments of the present application can be implemented.

[0124] In another possible implementation, the memory 102 can also be integrated with the processor 101.

[0125] The bus 103 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0126] The communication interface 104 is configured to connect the electronic device to other devices through a communication network, which can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 104 can include a receiving unit configured to receive data, and a sending unit configured to send data.

[0127] The electronic device 100 implements a display function through a GPU, a display screen 105 and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 105 and the application processor. The GPU is configured to perform mathematical and geometric calculations for graphics rendering. The processor 101 can include one or more GPUs, which execute program instructions to generate or change display information.

[0128] The display screen 105 is configured to display images, videos, and the like. The display screen 105 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 105, where N is a positive integer greater than 1.

[0129] It should be noted that the structure shown in the above embodiments does not limit the electronic device, except that Figure 8 The electronic device can include more or fewer components than shown, or combine some components, or have different arrangements of components. Figure 8 The electronic device shown in the above embodiments can include more or fewer components than shown, or combine some components, or have different arrangements of components.

[0130] Figure 9 A possible structural schematic diagram of the electronic device involved in the above embodiments is shown. Figure 9 The electronic device 1000 shown includes a processing module 1001, a display module 1002, and a storage module 1003.

[0131] The processing module 1001 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor can include an application processor and a baseband processor. It can implement or execute various exemplary logical blocks, modules, and circuits described in combination with the disclosure. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, and the like.

[0132] For example, the processing module 1001 can be, for example, Figure 8The processor 101 shown; the display module 1002 can be as follows: Figure 8 The display screen 105 shown; the storage module 1003 can be as follows: Figure 8 The memory 102 shown. The electronic device provided in this application embodiment can be Figure 8 The electronic device 100 shown.

[0133] This application also provides a chip system (e.g., a system-on-a-chip (SoC)). Figure 10 As shown, the chip system includes at least one processor 701 and at least one interface circuit 702. The processor 701 and the interface circuit 702 are interconnected via lines. For example, the interface circuit 702 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 702 can be used to send signals to other devices (e.g., the processor 701 or the camera of an electronic device). Exemplarily, the interface circuit 702 can read instructions stored in the memory and send those instructions to the processor 701. When the instructions are executed by the processor 701, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete components, which are not specifically limited in this application embodiment.

[0134] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the electronic device 100 in the above method embodiment.

[0135] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the electronic device 100 in the above method embodiments. For example, the computer may be the aforementioned electronic device 100.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0137] It should be noted that the personal information used in the technical solution of this application is limited to information for which individual consent has been obtained, including but not limited to notifying and reminding users to read the relevant user agreement (notification) and sign the agreement (authorization) which includes authorization of relevant user information before users use the function.

[0138] The collection, storage, use, processing, transmission, provision and disclosure of the user personal information in the technical solutions disclosed in the application comply with relevant laws and regulations and do not violate public order and good customs.

[0139] In several embodiments provided in the application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. Taking the division of the modules or units as an example, the division can be only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, apparatuses or units, and can be electrical, mechanical or in other forms.

[0140] The units described as separate components can or can not be physically separate, and the components displayed as units can be one physical unit or multiple physical units, i.e., can be located in one place, or can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the application.

[0141] In addition, each functional unit in the embodiments of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software functional units.

[0142] When the integrated unit is implemented in the form of software functional units and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the application essentially, or the part that contributes to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium, and includes several instructions for causing an apparatus (which can be a single chip machine, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store program codes.

[0143] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any change or replacement within the technical scope disclosed by the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized by, The method comprises: acquiring input data of a preset large language model; in a model inference process of the large language model based on the input data, the large language model generates first output data, and Value and Key corresponding to the first output data are stored in a preset cache according to a reference position; the reference position includes the coordinate position of Value and Key corresponding to second output data generated by the previous model inference in the preset cache; wherein the storage structure of the preset cache adopts an image body structure, the image body structure adopts a three-dimensional structure, the three-dimensional structure includes a sequence length of the image body structure, a width of the image body structure and a depth of the image body structure, wherein the sequence length is related to the number of data stored in the image body structure, the width of the image body structure is the same as the length of the attention head of the large language model, the depth of the image body structure is the same as the number of attention heads of the large language model, the image body structure provides the coordinate position of the stored data, and the coordinate position is used to provide an access entry for data.

2. The method of claim 1, wherein, The model inference process of the large language model is a model inference process based on a multi-head attention mechanism; Before storing Value and Key corresponding to the first output data in the preset cache according to the reference position, the method further comprises: acquiring structure parameters of the image body structure corresponding to the preset cache according to preset model parameters of the large language model; the preset model parameters include the length of the attention head and the number of attention heads; the structure parameters include the width of the image body and the depth of the image body structure; the depth corresponds to the number of attention heads; constructing the image body structure of the preset cache based on the structure parameters of the image body structure; the initial sequence length of the image body structure is 0; storing Value and Key corresponding to the first output data in the preset cache according to the reference position, comprising: storing Value and Key corresponding to the first output data in the target position of the image body structure of the preset cache according to the reference position in the image body structure of the preset cache; wherein the target position is the reference position, or the target position is the next position of the reference position in the image body structure.

3. The method of claim 2, wherein, storing Value and Key corresponding to the first output data in the preset cache according to the reference position, comprising: storing Value and Key corresponding to the first output data in the target position of each attention head of the image body structure in parallel according to the reference position of each attention head of the image body structure.

4. The method according to claim 2 or 3, characterized in that, The first output data is the first output data generated by the large language model in the model inference, the reference position is the position corresponding to the origin coordinate of the image body structure, and the target position is the reference position. The storing the Value and Key corresponding to the first output data into the target position in the preset cached image body structure according to the reference position in the preset cached image body structure comprises: The storing the Value and Key corresponding to the first output data into the position corresponding to the origin coordinate of the preset cached image body structure.

5. The method according to claim 2 or 3, characterized in that, The first output data is a non-first output data generated by the large language model in model inference, and the reference position is a coordinate position of Value and Key corresponding to the second output data in the preset cached image body structure. The storing the Value and Key corresponding to the first output data into the target position in the preset cached image body structure according to the reference position in the preset cached image body structure comprises: According to the sequence length of the stored Value and Key in the image body structure, the next position of the reference position in the image body structure is obtained; The first output data is stored in the next position of the reference position in the preset cached image body structure.

6. The method according to claim 2 or 3, characterized in that, After the storing the Value and Key corresponding to the first output data into the target position in the preset cached image body structure according to the reference position in the preset cached image body structure, the method further comprises: The sequence length of the preset cached image body structure is increased by 1.

7. The method of claim 2, wherein, The constructing the preset cached image body structure based on the structure parameters of the image body structure comprises: The preset cached image body structure is constructed in the first dimension of the sequence length direction, the second dimension of the width direction of the image body structure, and the third dimension of the attention head arrangement direction in the image body structure. The coordinate system corresponding to the image body structure is a three-dimensional coordinate system constructed based on the first dimension, the second dimension, and the third dimension, with the intersection point of the second dimension and the third dimension as the coordinate origin.

8. The method of claim 3, wherein, After the storing the Value and Key corresponding to the first output data into the preset cache according to the reference position, the method further comprises: According to the first output data corresponding to the input data to the last output data corresponding to the input data, the output result corresponding to the input data is generated.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 8. The processor executes the computer program to realize the steps of the method of any one of claims 1-8.

10. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the method of any one of claims 1-8.

11. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the method of any one of claims 1-8.

Images