Data processing method and electronic equipment

Abstract

The invention discloses a data processing method and electronic equipment, and relates to the technical field of large language models. The data processing method comprises the steps of obtaining input data of a preset large language model; in the process of performing model reasoning of the large language model based on the input data, the large language model generates first output data, and Value and Key corresponding to the first output data are stored in a preset cache according to the reference position. Wherein the reference position comprises a coordinate position of a Value and a Key in a preset cache, wherein the Value and the Key correspond to second output data generated by previous model reasoning for generating the first output data. Wherein the storage structure of the preset cache adopts an image body structure, and the image body structure provides the coordinate position of the stored data. According to the method and the device, the operation of copying the historical Value and Key from the preset cache is avoided, so that the time consumed for reading and splicing the historical Value and Key is shortened, the time of model reasoning is shortened, and the speed of model reasoning is increased.

Description

[0001] This application claims the priority of a Chinese patent application with the application number 202410063860.3 and the invention title "A Data Optimization Method" filed with the National Intellectual Property Administration on January 16, 2024. The entire content thereof is incorporated herein by reference. Technical Field

[0002] Embodiments of the present application relate to the technical field of large language models, and in particular, to a data processing method and an electronic device. Background Art

[0003] A key-value cache (KV Cache) is a cache mechanism for storing key-value pair data. During the inference process of a large language model, when the large language model outputs during encoding (decoder), it calculates one character (token) at a time in sequence. After each new token is calculated, it is added to the previous sequence to form a new input sequence for calculating the new token. This process is repeated until the output token of the calculation is a token of the stop symbol. The token sequence with the current sequence length of seq_len will be transformed through a weight matrix to obtain three matrices: Value, Key, and Query. In this process, the Value, Key, and Query matrices transformed from the token sequence with the length of seq_len - 1 have been obtained in the previous calculation step, and their values will not change. Therefore, only the Value, Key, and Query values of the newly generated token need to be transformed and concatenated with the previous Value, Key, and Query matrices to obtain the Value, Key, and Query matrices of the current complete token sequence. And during the process of the new token, it is only related to the current Query and the complete Value and Key, that is, the KV-cache caches the Value and Key of the previous round of inference, thereby achieving the effect of reducing calculations and accelerating inference.

[0004] However, as the Value and Key in the KV Cache increase, the amount of data to be concatenated each time also becomes larger and larger, and the time taken to read data from the KV Cache and perform concatenation increases, resulting in an increase in the inference latency of the large language model. Summary of the Invention

[0005] Embodiments of the present application provide a data processing method and an electronic device. During the model inference process of a large language model, a preset cache with an image body structure is constructed. It is possible to store the generated output data in the preset cache based on the reference position in the preset cache, avoiding the operation of copying historical output data from the preset cache. This shortens the time-consuming for reading and splicing the output data, shortens the model inference time, and improves the model inference speed.

[0006] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, a data processing method is provided, and the method includes:

[0008] Obtain the input data of a preset large language model. During the model inference process of the large language model based on the input data, the large language model generates first output data, and stores the Value and Key corresponding to the first output data in the preset cache according to the reference position; the reference position includes the coordinate positions of the Value and Key corresponding to the second output data generated by the previous model inference when generating the first output data in the preset cache.

[0009] Among them, the storage structure of the preset cache adopts an image body structure, and the image body structure provides the coordinate positions of the stored data.

[0010] In the present application, since the storage structure of the preset cache adopts an image body structure, the image body structure can provide the coordinate positions of the stored data. During the model inference process, when storing the Value and Key corresponding to each new output data in the preset cache, the Value and Key corresponding to the new output data can be directly stored according to the reference position. Compared with the prior art where the Value and Key corresponding to the output data are stored in a preset cache with a one-dimensional linear structure like a buffer, it is necessary to copy the Value and Key of the previous round in the preset cache. As the number of inferences increases, the historical Value and Key to be copied will gradually increase, and the copying time-consuming will become larger and larger. In the present application, it is not necessary to copy the historical Value and Key in the preset cache for splicing, reducing the data access volume to the preset cache, and at the same time shortening the time-consuming caused by accessing the preset cache for output data copying, improving the model inference speed.

[0011] In a possible implementation manner of the first aspect, the model inference process of the large language model is a model inference process based on a multi-head attention mechanism; before storing the Value and Key corresponding to the first output data in the preset cache according to the reference position, the method further includes:

[0012] Obtain the structural parameters corresponding to the preset cache's image body structure according to the preset model parameters of the large language model.

[0013] Among them, the preset model parameters include the length and number of attention heads; the structural parameters include the width of the image body and the depth of the image body structure, and the depth corresponds to the number of attention heads.

[0014] Construct the image body structure of the preset cache based on the structural parameters of the image body structure; the initial sequence length of the image body structure is 0.

[0015] Then, store the Value and Key corresponding to the first output data in the preset cache according to the reference position, including:

[0016] Store the Value and Key corresponding to the first output data in the target position of the image body structure of the preset cache according to the reference position in the image body structure of the preset cache; where the target position is the reference position, or the target position is the next position of the reference position in the image body structure.

[0017] In this application, storing the Value and Key corresponding to the first output data in the preset cache according to the reference position in the image body structure does not require copying the existing Value and Key in the preset cache for data splicing, reducing the data access volume to the preset cache, and at the same time shortening the time-consuming caused by accessing the preset cache for output data copying, improving the speed of model inference.

[0018] In another possible implementation manner of the first aspect, storing the Value and Key corresponding to the first output data in the preset cache according to the reference position includes:

[0019] Parallelly store the Value and Key corresponding to the first output data in the target position of each attention head of the image body structure according to the reference position of each attention head of the image body structure.

[0020] In this application, parallelly processing the storage of the Value and Key corresponding to the output data of each attention head can further accelerate the speed of model inference.

[0021] In another possible implementation manner of the first aspect, the first output data is the first output data generated by the large language model during model inference, the reference position is the position corresponding to the origin coordinates of the image body structure, and the target position is the reference position.

[0022] Then, store the Value and Key corresponding to the first output data in the target position of the image body structure of the preset cache according to the reference position in the image body structure of the preset cache, including:

[0023] Store the Value and Key corresponding to the first output data at the position corresponding to the origin coordinates of the image body structure in the preset cache.

[0024] In this application, when the first output data is the output data generated by the first model inference, the Value and Key corresponding to the first output data can be directly stored at the position corresponding to the origin coordinates of the image body structure, which simplifies the process of storing the output data into the preset cache.

[0025] In another possible implementation of the first aspect, the first output data is the non-first output data generated by the large language model in the model inference, and the reference position is the coordinate position of the Value and Key corresponding to the second output data in the image body structure of the preset cache.

[0026] According to the reference position in the preset cache image body structure, store the Value and Key corresponding to the first output data into the target position of the image body structure of the preset cache, including:

[0027] Then, according to the sequence length of the image body structure, obtain the next position of the reference position in the image body structure; store the Value and Key corresponding to the first output data into the next position of the reference position of the image body structure of the preset cache.

[0028] In this application, when the first output data is the output data generated by non-first model inference, the Value and Key corresponding to the first output data can be directly stored into the next position of the image body structure according to the next position of the reference position in the image body structure, without copying the existing Value and Key in the preset cache for data splicing, reducing the data access volume to the preset cache, and at the same time shortening the time-consuming caused by accessing the preset cache for output data copying, improving the speed of model inference.

[0029] In another possible implementation of the first aspect, after storing the Value and Key corresponding to the first output data into the target position of the image body structure of the preset cache according to the reference position of the preset cache image body structure, the method further includes:

[0030] Increase the sequence length of the image body structure of the preset cache by 1.

[0031] In this application, after each time the Value and Key corresponding to the first output data are stored into the image body structure of the preset cache, the sequence length of the image body structure is updated in a timely manner. In the subsequent model inference, the reference position can be determined according to the sequence length, and the Value and Key corresponding to the next first output data can be stored, which simplifies the process of storing the output data into the preset cache.

[0032] In another possible implementation of the first aspect, based on the structural parameters of the image volume structure, a preset cached image volume structure is constructed, including:

[0033] Taking the direction where the sequence length is located as the first dimension, the direction where the width of the image volume structure is located as the second dimension, and the direction where the attention heads are arranged in the image volume structure as the third dimension, a preset cached image volume structure is constructed; the coordinate system corresponding to the image volume structure is a three-dimensional coordinate system constructed based on the first dimension, the second dimension, and the third dimension with the intersection point of the second dimension and the third dimension as the coordinate origin.

[0034] Wherein, the width of the image volume structure is the same as the length of the attention heads of the large language model; the number of attention heads of the image volume structure is the same as the number of attention heads of the large language model.

[0035] In this application, an image volume structure corresponding to three-dimensional data is constructed based on the three-dimensional structure of the large language model, so that the image volume structure can better store the Value and Key corresponding to the output data obtained by each model inference of the large language model, avoiding the problem of large model inference time consumption caused by data traversal, data copying, and data splicing in the one-dimensional cache structure.

[0036] In another possible implementation of the first aspect, after storing the Value and Key corresponding to the first output data in the preset cache according to the reference position, the method further includes:

[0037] According to the first output data corresponding to the input data to the last output data corresponding to the input data in the preset cached image volume structure, an output result corresponding to the input data is generated.

[0038] In this application, according to the Value and Key corresponding to the output data generated by each model inference stored in the preset cache, the first output data corresponding to the input data to the last output data corresponding to the input data are combined into an output result, and the process of obtaining the output result avoids repeated calculations and improves the speed of model inference.

[0039] In a second aspect, an electronic device is provided, which includes a memory and one or more processors; the memory is coupled to the processor; computer program code is stored in the memory, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device is caused to execute the method according to any one of the above first aspects.

[0040] In a third aspect, a computer-readable storage medium is provided, in which instructions are stored, and when it runs on an electronic device, the electronic device can be caused to execute the method according to any one of the above first aspects.

[0041] In a fourth aspect, a computer program product including instructions is provided. When it runs on an electronic device, it enables the electronic device to execute the method described in any one of the above first aspects.

[0042] In a fifth aspect, an embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the method as in the first aspect.

[0043] It can be understood that for the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect, reference can be made to the beneficial effects in the first aspect and any possible design thereof, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic structural diagram of a buffer structure provided by an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of the process of splicing Value and Key under a buffer structure provided by an embodiment of the present application;

[0046] Figure 3 It is a schematic structural diagram of a three-dimensional image volume structure provided by an embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of an image volume structure and coordinates provided by an embodiment of the present application;

[0048] Figure 5 It is a schematic diagram of an image volume structure provided by an embodiment of the present application;

[0049] Figure 6 It is a schematic diagram of another image volume structure provided by an embodiment of the present application;

[0050] Figure 7 It is a schematic diagram of adding Value and Key in an image volume structure provided by an embodiment of the present application;

[0051] Figure 8 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application;

[0052] Figure 9 It is a schematic diagram of a possible structure of an electronic device provided by an embodiment of the present application;

[0053] Figure 10 It is a schematic structural diagram of a chip system provided by an embodiment of the present application. Detailed implementation manners

[0054] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and claims of the present application, the singular forms "a", "the", "above-mentioned", "this" and "this one" are also intended to include, for example, the expression "one or more", unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of the present application, "at least one" and "one or more" mean one or more than two (including two). The term "and / or" is used to describe the association relationship of associated objects and indicates that three relationships may exist; for example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship.

[0055] References in this specification to "one embodiment" or "some embodiments" or the like mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" and the like that appear in different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprise", "include", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connection" includes direct connection and indirect connection, unless otherwise stated. "First" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features.

[0056] In the embodiments of the present application, words such as "exemplary" or "for example" are used to mean for example, illustration or explanation. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or more advantageous than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0057] Large language models (LLMs) are deep learning models trained on vast amounts of text data. LLMs can not only generate natural language text but also deeply understand the meaning of text and handle various natural language tasks such as text summarization, question answering, and translation. Each layer of an LLM is a self-attention neural network model (Transformer), which is a model that uses the attention mechanism to improve the model training speed. The Transformer consists of two parts: an encoder and a decoder. The encoder is responsible for obtaining the encoded information matrix of all prompts based on the vector matrix of the incoming prompts (tokens). The decoder is responsible for predicting the next token to be translated based on the encoded information matrix, successively according to the currently translated prompts (tokens). The processes of the encoder and decoder can also be understood as the model inference process. The processes of the encoder and decoder include attention steps and feed-forward steps, which are used to update the state of each token in the input channels of the Transformer. Among them, in the attention step, the token will "look around" to find other tokens with relevant contexts and share information with each other. In the feed-forward step, each token will "think" about the information collected in the previous attention step and try to predict the next token.

[0058] During the model inference process, the historically generated tokens do not need to have attention with the subsequent tokens. Therefore, during the model inference stage of the large language model, the Values and Keys corresponding to the already generated tokens can be cached. Specifically, when the large language model outputs during encoding (decoder), it calculates one token at a time in sequence. After each new token is calculated, it is added to the previous sequence to form a new input sequence for the calculation of the new token. This process is looped until the output token being calculated is the stop symbol token. The token sequence with the current sequence length of seq_len will be transformed through the weight matrix to obtain three matrices: Value, Key, and Query. In this process, the Value, Key, and Query matrices transformed from the token sequence with a length of seq_len - 1 have been obtained in the previous step calculation, and their values will not change. Therefore, only the Value, Key, and Query values of the newly generated token need to be transformed and concatenated with the previous Value, Key, and Query matrices to obtain the Value, Key, and Query matrices of the current complete token sequence. And during the generation of new tokens, it is only related to the current Query and the complete Value and Key, that is, the KV-cache caches the Value and Key of the previous round of inference, thus achieving the effect of reducing calculations and accelerating inference.

[0059] After the KV cache is configured, the model inference process of the large language model can be divided into two stages: the pre-fill stage and the stage of using the KV cache. Among them, the pre-fill stage means that before calculating the first output token during the model inference process, at this time the KV cache is empty. When outputting the token, the token sequence is transformed through the weight matrix to obtain three matrices: Value, Key, and Query, and the calculated Value and Key are stored in the KV cache, and the KV cache is completed. The stage of using the KV cache means that during the process of calculating the second output token to the last output token during the model inference process, the KV cache is not empty. In each round of inference, when a new token is obtained, calculate the Value, Key, and Query corresponding to this token, read the Value and Key obtained in the previous round of calculation from the KV cache, and concatenate the Value and Key of the current round of token with the Value and Key obtained in the previous round of calculation.

[0060] Take an example to illustrate the use of the KV cache during the model inference process of the large language model.

[0061] Suppose the user gives an input data "Hello", and the large language model will generate an output data "Hello" after the first model inference. Based on the output data of the first model inference, the large language model goes through N model inferences to generate the output result for "Hello". For example, the output result is "Hello, I'm the intelligent assistant xx. How can I help you?". Among them, during the model inference process of the large language model, one model inference (abbreviated as inference) generates one word (token). For each token output by each inference, the corresponding Value, Key, and Query of the token are calculated. The Value and Key obtained from the previous calculation are read from the KV cache, and the Value and Key obtained from the current calculation are concatenated with the Value and Key obtained from the previous calculation and stored in the KV cache. Until the token that is the stop symbol is inferred, the output result for the input data is obtained. Since each generated token will be concatenated with the previously generated tokens as the input data for the next inference, in this embodiment, the tokens inferred and stored in the cache can be collectively referred to as input tokens. The input tokens in this embodiment include the tokens corresponding to the input data and the tokens (output data) generated in each inference process.

[0062] For example, during the i-th inference process, the input tokens are:

[0063] "Hello, I'm the intelligent assistant xx,".

[0064] The i-th inference generates a new token "Please" based on the input tokens. Since the Value and Key corresponding to "Hello, I'm the intelligent assistant xx," have already been stored in the KV cache, the i-th inference only needs to calculate the Value, Key, and Query corresponding to the new token "Please". Instead of calculating the Value and Key corresponding to these tokens again, the Value and Key corresponding to these tokens can be directly read from the KV cache, and the Value and Key corresponding to the new token "Please" obtained from the current inference are concatenated with the Value and Key corresponding to the tokens read from the KV cache to obtain the Value, Key, Query matrix corresponding to the complete token sequence up to the current inference. Among them, the input tokens "Hello, I'm the intelligent assistant xx," are concatenated with the new token "Please" generated in this round of inference to obtain the complete token sequence after this inference:

[0065] "Hello, I'm the intelligent assistant xx, Please".

[0066] During the i+1th reasoning process, calculate the Value, Key, and Query corresponding to the new token "问". Read the Value and Key corresponding to the input tokens from KVcache, concatenate the Value and Key corresponding to the new token "问" obtained in this reasoning with the Value and Key corresponding to the tokens read from KV cache, and obtain the Value, Key, and Query matrix corresponding to the complete token sequence up to this reasoning. Among them, before this reasoning, the input tokens (complete token sequence) are:

[0067] "Hello, I am the smart assistant xx, please";

[0068] Concatenate the input tokens "Hello, I am the smart assistant xx, please" with the new token "问" generated by this round of reasoning to get the complete token sequence after this reasoning:

[0069] "Hello, I am the intelligent assistant xx, how could I please help?"

[0070] After N inferences, we get new tokens “有”, “什”, “什么”, “可”, “以”, “帮”, “助”, “你”, “的” and concatenate the new tokens with the input tokens respectively, and get the output result composed of all the input tokens:

[0071] "Hello, I am the intelligent assistant xx, how can I help you?"

[0072] During the model inference process, since the KV cache caches the Value and Key corresponding to the previous round of tokens, there is no need to repeatedly calculate the past Value and Key in the process of predicting a new token, saving computing resources in the subsequent token prediction process.

[0073] Generally, KV cache uses a buffer structure. The buffer structure is a one-dimensional structure, which can also be understood as a continuous memory. All data stored in the KV cache is in a spliced state. The Value and Key of self-attention in the large language model structure include the sequence length, the length of the attention head, and the number of attention heads. The one-dimensional buffer stores the Value and Key data by copying and splicing the Value and Key data in sequence along the length direction. Figure 1 , Figure 1A schematic structural diagram of a buffer structure is given. Among them, taking the number of attention heads in the attention structure of a large language model as 32 and the length of the attention heads as 128 as an example. After two inferences, Figure 1 Each attention head in the shown buffer structure already stores Values and Keys with a sequence length of 2, and the sequence length of the buffer is 2 * 128 * 32.

[0074] Due to the characteristics of the buffer being a one-dimensional structure and a continuous piece of memory, in the KV cache based on the buffer structure, when generating new Values and Keys for concatenation, it is necessary to copy the already generated Values and Key values to the new buffer in sequence according to the sequence order, and store the newly generated Values and Key values in this structure. Repeat this operation until the data concatenation of all attention heads is completed. Exemplarily, refer to Figure 2 , Figure 2 A schematic diagram of the process of concatenating the Values and Keys corresponding to tokens under a buffer structure is given. Combining Figure 1 with the schematic diagram of the buffer structure given. Figure 2 already stores the Values and Keys corresponding to 2 tokens. When predicting a new token, it is necessary to first copy the data of the Values and Keys of the first attention head to the new buffer structure (which can also be understood as a new cache), and concatenate the data of the first attention head of the Values and Keys generated when predicting the new token behind the data of the first attention head of the Values and Keys in the previous round. Sequentially, when copying the data of the Values and Keys of the second attention head, concatenate it behind the data of the Values and Keys of the first attention head. Concatenate the data of the first attention head of the Values and Keys of the new token behind the data of the second attention head of the Values and Keys in the previous round,..., until the data of the Values and Keys of the new token is concatenated behind the data of the Values and Keys of the 32nd attention head. Finally, obtain the Values and Keys for this inference, and store them in the one-dimensional linear buffer structure after data concatenation.

[0075] Obviously, in the above application of the KV cache based on the one-dimensional linear structure of the buffer, when the model inference concatenates the Value and Key of the newly generated token with the Value and Key of the previous round, it is necessary to copy the Value and Key of the previous round in the KV cache every time. Moreover, as the number of inferences increases, the Value and Key that need to be copied gradually increase, and the copying time-consuming becomes larger and larger, resulting in a slowdown in the model inference speed and an increase in the model inference time.

[0076] The present application provides a data processing method, and the preset cache adopts a three-dimensional image volume structure. Among them, the image volume structure can provide the coordinate positions of the stored output data. During the model inference process, the coordinate positions of the Value and Key in the KV cache in the image volume structure can be directly obtained, so that based on this coordinate position, the new Value and Key values can be concatenated into the Value and Key of the previous round. There is no need to copy the Value and Key of the previous round in the KV cache, reducing the data access volume to the KV cache. The newly generated Value and Key are directly cached based on the coordinate positions of the data in the image volume structure, shortening the time-consuming caused by accessing the KV cache and copying the Value and Key, and improving the model inference speed.

[0077] The following embodiments illustrate the image volume structure adopted by the KV cache in the embodiments of the present application.

[0078] The image volume structure can also be called an image object. An image object is a special memory object used to represent image data, and the image object supports directly accessing memory data through coordinates. That is, data caching based on the image volume structure can quickly locate and operate on the cached data in the image volume structure. The image object supports the creation of one-dimensional data, two-dimensional data, three-dimensional data, and high-dimensional data. Exemplarily, Figure 3 A schematic structural diagram of a three-dimensional image volume structure is given. In the scenario of storing image data using a three-dimensional image volume structure (image object), the three-dimensional parameters of the image object include the width of the image (image width), the height of the image (height), and the depth of the image (depth). Based on the image volume structure, a three-dimensional coordinate can be constructed, so that the image data at the position of each pixel can be accessed through the coordinates of each pixel.

[0079] In some embodiments, based on the feature that the coordinate positions can be provided for data access according to the image body structure, the KV cache adopting the image body structure is applied to the model inference process of the large language model, so as to achieve the purpose of improving the model inference speed. It should be noted that the large language model involved in the embodiments of the present application is a model that performs model inference based on the multi-head attention mechanism. Under the multi-head attention mechanism, the preset model parameters of the large language model include the length of the attention head and the number of attention heads, etc. The embodiments of the present application provide a data processing method, including:

[0080] S201. According to the preset model parameters of the large language model, obtain the structure parameters of the image body structure corresponding to the preset cache.

[0081] Among them, the preset model parameters include the length of the attention head and the number of attention heads. In some embodiments, the structure parameters of the image body structure of the preset cache can be determined according to the preset model parameters of the large language model. Among them, the structure parameters of the image body structure include the width of the image body structure and the depth of the image body structure. The preset cache is the KV cache.

[0082] In some embodiments, the width of the image body structure can be the same as the length of the attention head of the large language model; the depth of the image body structure is the same as the number of attention heads of the large language model. For example, when the length of the attention head of the large language model is 128, the width of the image body structure is also 128; when the number of attention heads of the large language model is 32, the depth depth of the image body structure is 32.

[0083] S202. Based on the structure parameters of the image body structure, construct the image body structure of the preset cache.

[0084] In this embodiment, the image body structure is a three-dimensional structure. Among them, taking the direction where the sequence length is located as the first dimension, the direction where the width of the image body structure is located as the second dimension, and the direction where the attention heads are arranged in the image body structure as the third dimension, construct the image body structure of the preset cache.

[0085] In some embodiments, the intersection point of the first dimension, the second dimension and the third dimension is used as the coordinate origin. Exemplarily, refer to Figure 4 , Figure 4A schematic diagram of an image body structure and coordinates is given. The intersection point P of the first dimension, the second dimension and the third dimension is used as the coordinate origin to construct a coordinate system of the image body structure. Among them, the first dimension is the x-axis direction, the second dimension is the y-axis direction, and the third dimension is the z-axis direction. In the process of storing data based on the preset cache of the image body structure, for each additional output data in the preset cache, the sequence length of the image body structure increases by 1 in the x direction. In some embodiments, when the preset cache does not store output data, the initial sequence length of the image body structure is 0.

[0086] S203: Obtain input data of the large language model.

[0087] In some embodiments, the input data of the large language model can be data input by the user or data obtained from a database. For example, in a question-and-answer scenario, the input data of the large language model can be text input by the user. Exemplarily, the text input by the user can be "hello", and the input data input to the large language model includes two tokens "you" and "good".

[0088] S204: Input the input data into the large language model for model inference to generate first output data.

[0089] In some embodiments, the large language model performs model inference based on input data.

[0090] Assume that the user gives an input data "Hello", the large language model will generate an output result for the input data "Hello" after model inference. For example, the output result is "Hello, I am the smart assistant xx, how can I help you?" Among them, the output result "Hello, I am the smart assistant xx, how can I help you" is obtained after the large language model performs N times of model inference.

[0091] In each model inference process of the large language model, when new output data (token) is obtained, the query, value and key corresponding to the token are calculated. The value and key corresponding to this token are concatenated to the value and key corresponding to the second output data (token obtained in the previous round of calculation) in the preset cache. The second output data is the output data generated by the previous model inference that generated the first output data.

[0092] For example, in the i-th reasoning process, the historical output data (tokens) is: "Hello, I am the intelligent assistant xx, please." The preset cache has stored the Value and Key corresponding to all tokens that have been calculated historically. The output data (token) generated by the i-th reasoning is "Ask", and the Query, Value, and Key corresponding to "Ask" are calculated.

[0093] S205. Store the first output data into a preset cache according to a reference position.

[0094] The reference position includes the coordinate positions of the Value and Key corresponding to the second output data generated by the previous model inference for generating the first output data in the preset cache.

[0095] When the large language model stores the Value and Key corresponding to the first output data into the preset cache during the model inference process, there are two cases. One is the case of storing the first output data obtained from the first model inference based on the input data. Before storing the Value and Key corresponding to the first output data, the image body structure of the preset cache is empty and the sequence length is 0. In this case, the reference position is the position corresponding to the origin coordinates of the image body structure.

[0096] The other case is the case of storing the first output data obtained from a non-first model inference based on the input data. Before storing the Value and Key corresponding to the first output data, there are already historical Value and Key in the image body structure of the preset cache and the sequence length is not 0. In this case, the reference position is the position corresponding to the Value and Key in the previous round (the Value and Key corresponding to the second output data) in the image body structure.

[0097] Specifically, in some embodiments, to store the Value and Key corresponding to the first output data according to the image body structure of the preset cache, S205 includes:

[0098] S2051. According to the reference position in the image body structure of the preset cache, store the Value and Key corresponding to the first output data into the target position in the image body structure of the preset cache.

[0099] The target position is the reference position, or the target position is the next position of the reference position in the image body structure.

[0100] In this embodiment, corresponding to the above two cases, in the case of storing the Value and Key corresponding to the first output data obtained from the first model inference based on the input data, the target position is the reference position, and the Value and Key corresponding to the first output data can be stored in this reference position. In the case of storing the Value and Key corresponding to the first output data obtained from a non-first model inference based on the input data, the target position is the next position of the reference position (the position of the Value and Key corresponding to the second output data), and the Value and Key corresponding to the first output data are stored in the next position of this reference position.

[0101] Specifically, in some embodiments, the first output data is the first output data generated by the large language model during model inference, the reference position is the position corresponding to the origin coordinates of the image volume structure, and the target position is the reference position. Then S2051 includes:

[0102] Store the Value and Key corresponding to the first output data at the position corresponding to the origin coordinates of the image volume structure in the preset cache.

[0103] In this embodiment, before storing the Value and Key corresponding to the first output data in the preset cache, the sequence length of the image volume structure in the preset cache is 0. The reference position is the position corresponding to the origin coordinates of the image volume structure. Figure 5 A schematic diagram of an image volume structure is given. Among them, referring to Figure 5 (a), the first output data is the output data generated by the first model inference based on the input data, and the position corresponding to the origin coordinates is point P. The number of attention heads of the image volume structure is 32, and the width of the image volume structure is 128. According to the position of point P, store the Value and Key corresponding to the first output data into the image volume structure along the x direction (the direction where the sequence length is located), and at the same time update the sequence length of the image volume structure from 0 to 1. Referring to Figure 5 (b), exemplarily, assume that the user gives an input data "Hello", and the large language model will generate an output data (token) of "Hello" after the first model inference. Calculate the Query, Value, and Key corresponding to the tokens. Store the Value and Key into the image volume structure, and the size of the Value and Key is 32 * 2 * 128, and the sequence length increases from 0 to 2.

[0104] In this embodiment, when the first output data is the output data generated by the first model inference, the Value and Key corresponding to the first output data can be directly stored at the position corresponding to the origin coordinates of the image volume structure, which simplifies the process of storing the output data in the preset cache.

[0105] Specifically, in some embodiments, the first output data is the non-first output data generated by the large language model during model inference, and the reference position is the coordinate position of the Value and Key corresponding to the second output data in the image volume structure of the preset cache.

[0106] Then S2051 includes:

[0107] According to the sequence length of the image volume structure, obtain the next position of the reference position in the image volume structure, and store the Value and Key corresponding to the first output data in the next position of the reference position in the image volume structure of the preset cache.

[0108] In this embodiment, before storing the Value and Key corresponding to the first output data into the preset cache, there are already historical Value and Key in the image body structure of the preset cache, and the sequence length is not 0. The reference position is the coordinate position of the Value and Key corresponding to the second output data in the image body structure of the preset cache. Reference Figure 6 , Figure 6 FIG. Figure 6 shows a schematic diagram of another image body structure. Among them, the first output data is the output data generated by the non-first model inference based on the input data, and the position corresponding to the origin coordinate is point P. According to the sequence length of the image body structure being 2, the reference position is determined to be M. Then, correspondingly, the next position of M is N. The number of attention heads of the image body structure is 32, and the width of the image body structure is 128. According to the position of N determined by point M, along the x direction (the direction where the sequence length is located), the Value and Key corresponding to the first output data generated by this inference are stored into the image body structure at the position of N, and at the same time, the sequence length of the image body structure is updated from 2 to 3. And so on, in the subsequent N inference processes, the coordinate position of the Value and Key (historical Value and Key) corresponding to the second output data in the image body structure can be determined as the reference position according to the sequence length of the image body structure. Based on the next position of the reference position, the Value and Key corresponding to the first output data generated by the current inference are stored, and the sequence length of the image body structure is updated.

[0109] In this embodiment, when the first output data is the output data generated by the non-first model inference, the Value and Key corresponding to the first output data can be directly stored into the next position of the image body structure according to the next position of the reference position in the image body structure, without copying the existing historical Value and Key in the preset cache for data splicing, reducing the data access amount to the preset cache, and at the same time shortening the time consumption caused by accessing the preset cache for output data copying, improving the speed of model inference.

[0110] In some embodiments, the image body structure is a three-dimensional data structure corresponding to a large language model, which includes multiple attention heads, and S205 includes:

[0111] According to the reference position of each attention head of the image body structure, the Value and Key corresponding to the first output data are stored into the target position of each attention head of the image body structure in parallel.

[0112] In some embodiments, after a model inference is performed once to obtain the first output data (token), the Value and Key generated corresponding to the token are stored in the image body structure, and the Value and Key corresponding to the second output data are stored in the sequence direction to the next position of the existing Value and Key.

[0113] refer to Figure 7 , Figure 7 A schematic diagram of adding Value and Key to the image body structure is given. Figure 7 Taking the coordinate system of the image body structure shown as an example, the sequence length of the preset cached image structure is 2. For each attention head in the image body structure, the Value and Key corresponding to the second output data in the x direction (the direction of the sequence length) of the attention head are stored in the next position. The splicing and storage operations of multiple attention heads can be executed in parallel. The length of the sequence storing the new Value and Key increases by 1, and the sequence length of Value and Key is updated to 3.

[0114] S206 , generating output results corresponding to the input data according to the first output data corresponding to the input data to the last output data corresponding to the input data in the preset cached image body structure.

[0115] In this embodiment, the preset cache stores the Value and Key data corresponding to the complete tokens sequence. Based on the Value and Key data corresponding to the complete tokens sequence in the preset cache, the complete tokens for the input data can be obtained. For example, in the inference process of the input data of the large language model being "hello", after N inferences, the model inference is stopped when the large language model infers to the stop symbol. According to the Value and Key data corresponding to the complete tokens sequence in the preset cache, the conversion is performed, and the obtained complete tokens sequence includes "you", "good", ",","I", "I", "Is", "smart", "can", "help", "hand", "x", "x", ",","please", "ask", "have", "what", "what", "can", "can", "help", "help", "you", "of"; that is, "Hello, I am the smart assistant xx, how can I help you" is used as the output result corresponding to the input data.

[0116] The present application provides a data processing method, where a preset cache adopts a three-dimensional image volume structure. The image volume structure can provide the coordinate positions of the stored output data. During the model inference process, for each newly generated output data, the corresponding generated Value and Key can directly obtain the reference positions in the image volume structure of the preset cache, and cache the Value and Key corresponding to the newly generated output data into the preset cache. There is no need to copy the Value and Key corresponding to the historical output data in the preset cache for data splicing, reducing the data access volume to the preset cache, reducing the impact on memory caused by data copying and data splicing (data transfer) during model inference, and directly caching the Value and Key corresponding to the newly generated output data based on the coordinate positions of the Value and Key corresponding to the output data in the image volume structure, shortening the time consumed due to accessing the preset cache and copying the historical Value and Key, and improving the speed of model inference. Exemplarily, according to a large amount of experimental data, during the model inference process, using the KV cache with an image volume structure for data caching can reduce the overall model inference latency by 2-3 times compared to using the KV cache with a buffer structure for data caching.

[0117] The data processing method provided by the embodiments of the present application can be applied to an electronic device. The electronic device in the embodiments of the present application can be a portable computer (such as a mobile phone), a tablet computer, a laptop computer, a personal computer (PC), a wearable electronic device (such as a smart watch), an augmented reality (AR) / virtual reality (VR) device, an in-vehicle computer, a server, a server cluster, etc. The following embodiments do not impose special restrictions on the specific form of the electronic device.

[0118] Exemplarily, Figure 8 is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. Refer to Figure 8 , Figure 8 As shown, the electronic device may include: a processor 101, a memory 102, and a bus 103. The processor 101 and the memory 102 can be connected through the bus 103. Optionally, in some embodiments, the electronic device 100 may further include a communication interface 104, and the communication interface 104 is connected to the processor 101 and the memory 102 through the bus 104. Optionally, in some embodiments, the electronic device 100 may further include a display screen 105, and the display screen 105 is connected to the processor 101 and the memory 102 through the bus 103.

[0119] The processor 101 is the control center of the electronic device. The processor 101 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0120] As an example, the processor 101 may include one or more CPUs, such as Figure 8 the CPU 0 and CPU 1 shown in

[0121] Among them, the NPU is a neural-network (NN) computing processor. By learning from the biological neural network structure, such as learning from the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc. A preset large language model can be deployed in the NPU to process the input data corresponding to the user input operation to obtain an output result corresponding to the input data.

[0122] The memory 102 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0123] In a possible implementation, the memory 102 can exist independently of the processor 101. The memory 102 can be connected to the processor 101 through the bus 103 and is used to store data, instructions, or program codes. When the processor 101 calls and executes the instructions or program codes stored in the memory 102, the data processing method provided by the embodiments of the present application can be implemented.

[0124] In another possible implementation, the memory 102 can also be integrated with the processor 101.

[0125] The bus 103 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, Figure 8 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0126] The communication interface 104 is used for the electronic device to be connected to other devices through a communication network. The communication network can be an Ethernet, a Radio Access Network (RAN), a Wireless Local Area Networks (WLAN), etc. The communication interface 104 can include a receiving unit for receiving data and a sending unit for sending data.

[0127] The electronic device 100 implements the display function through a GPU, a display screen 105, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display screen 105 and the application processor. The GPU is used to perform mathematical and geometric calculations and is used for graphics rendering. The processor 101 can include one or more GPUs, which execute program instructions to generate or change display information.

[0128] The display screen 105 is used to display images, videos, etc. The display screen 105 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 105, where N is a positive integer greater than 1.

[0129] It should be noted that Figure 8 the structure shown in Figure 8 does not constitute a limitation on the electronic device. In addition to

[0130] Figure 9 the components shown, the electronic device may further include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements. Figure 9 The illustrated electronic device 1000 includes a processing module 1001, a display module 1002, and a storage module 1003.

[0131] Among them, the processing module 1001 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor may include an application processor and a baseband processor. It can implement or execute various exemplary logic blocks, modules, and circuits described in connection with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on.

[0132] For example, the processing module 1001 can be as Figure 8The processor 101 shown; the display module 1002 can be, for example, Figure 8 the display screen 105 shown; the storage module 1003 can be, for example, Figure 8 the memory 102 shown. The electronic device provided by the embodiments of the present application can be Figure 8 the electronic device 100 shown.

[0133] The embodiments of the present application also provide a chip system (for example, a system on a chip (SoC)), as Figure 10 shown, the chip system includes at least one processor 701 and at least one interface circuit 702. The processor 701 and the interface circuit 702 can be interconnected through a line. For example, the interface circuit 702 can be used to receive signals from other devices (such as the memory of an electronic device). Also for example, the interface circuit 702 can be used to send signals to other devices (such as the processor 701 or the camera of an electronic device). Exemplarily, the interface circuit 702 can read the instructions stored in the memory and send the instructions to the processor 701. When the instructions are executed by the processor 701, the electronic device can be made to execute each step in the above embodiments. Of course, the chip system can also include other discrete devices, and the embodiments of the present application do not make specific limitations thereto.

[0134] The embodiments of the present application also provide a computer-readable storage medium, the computer-readable storage medium includes computer instructions, when the computer instructions run on the above electronic device, the electronic device is made to execute each function or step executed by the electronic device 100 in the above method embodiments.

[0135] The embodiments of the present application also provide a computer program product, when the computer program product runs on a computer, the computer is made to execute each function or step executed by the electronic device 100 in the above method embodiments. For example, the computer can be the above electronic device 100.

[0136] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above division of each functional module is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0137] It should be noted that the personal information used in the technical solution of the present application is limited to the information for which individual consent has been obtained, including but not limited to, before the user uses the function, notifying and reminding the user to read the relevant user agreement (notification), and signing the agreement (authorization) including authorizing the relevant user information.

[0138] In the technical solution disclosed in this application, the processing of the user's personal information, such as collection, storage, use, processing, transmission, provision, and disclosure, all complies with the provisions of relevant laws and regulations and does not violate public order and good customs.

[0139] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separated. The components displayed as units can be one physical unit or multiple physical units, that is, they can be located in one place or distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0142] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of this application. And the aforementioned storage medium includes: USB flash drives, mobile hard disks, read only memory (ROM), random access memory (RAM), magnetic disks or optical discs and other various media that can store program codes.

[0143] The above content is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtaining input data of a preset large language model; During the model inference process of the large language model based on the input data, the large language model generates first output data, and stores the Value and Key corresponding to the first output data in a preset cache according to a reference position; the reference position includes the coordinate positions of the Value and Key corresponding to the second output data generated by the previous model inference for generating the first output data in the preset cache; Wherein, the storage structure of the preset cache adopts an image body structure, and the image body structure provides the coordinate positions of the stored data.

2. The method according to claim 1, wherein The model inference process of the large language model is a model inference process based on a multi-head attention mechanism; Before storing the Value and Key corresponding to the first output data in the preset cache according to the reference position, the method further includes: Obtaining structure parameters corresponding to the image body structure of the preset cache according to the preset model parameters of the large language model; the preset model parameters include the length and number of attention heads; the structure parameters include the width of the image body and the depth of the image body structure; the depth corresponds to the number of attention heads; Constructing the image body structure of the preset cache based on the structure parameters of the image body structure; the initial sequence length of the image body structure is 0; Storing the Value and Key corresponding to the first output data in the preset cache according to the reference position includes: Storing the Value and Key corresponding to the first output data in the target position of the image body structure of the preset cache according to the reference position in the image body structure of the preset cache; wherein, the target position is the reference position, or the target position is the next position of the reference position in the image body structure.

3. The method according to claim 2, characterized in that Storing the Value and Key corresponding to the first output data in the preset cache according to the reference position includes: Parallelly storing the Value and Key corresponding to the first output data in the target position of each attention head of the image body structure according to the reference position of each attention head of the image body structure.

4. The method according to claim 2 or 3, characterized in that, The first output data is the first output data generated by the large language model during model inference, the reference position is the position corresponding to the origin coordinates of the image body structure, and the target position is the reference position; Storing the Value and Key corresponding to the first output data in the target position of the image body structure of the preset cache according to the reference position in the image body structure of the preset cache includes: Storing the Value and Key corresponding to the first output data in the position corresponding to the origin coordinates of the image body structure of the preset cache.

5. The method according to claim 2 or 3, characterized in that, The first output data is a non-first output data generated by the large language model during model inference, and the reference position is the coordinate position of the Value and Key corresponding to the second output data in the image body structure of the preset cache; Storing the Value and Key corresponding to the first output data to the target position in the preset cached image body structure according to the reference position in the preset cached image body structure includes: Obtaining the next position of the reference position in the image body structure according to the sequence length of the stored Value and Key in the image body structure; Storing the Value and Key corresponding to the first output data to the next position of the reference position in the preset cached image body structure.

6. The method according to any one of claims 2-5, characterized in that, After storing the Value and Key corresponding to the first output data to the target position in the preset cached image body structure according to the reference position in the preset cached image body structure, the method further includes: Increasing the sequence length of the preset cached image body structure by 1.

7. The method according to claim 2, characterized in that Constructing the preset cached image body structure based on the structural parameters of the image body structure includes: Constructing the preset cached image body structure with the direction where the sequence length is located as the first dimension, the direction where the width of the image body structure is located as the second dimension, and the arrangement direction of the attention heads in the image body structure as the third dimension; the coordinate system corresponding to the image body structure is a three-dimensional coordinate system constructed based on the first dimension, the second dimension, and the third dimension with the intersection of the second dimension and the third dimension as the coordinate origin; Wherein, the width of the image body structure is the same as the length of the attention heads of the large language model; the number of attention heads of the image body structure is the same as the number of attention heads of the large language model.

8. The method according to any one of claims 3-7, characterized in that After storing the Value and Key corresponding to the first output data to the preset cache according to the reference position, the method further includes: Generating the output result corresponding to the input data according to the first output data of the input data to the last output data corresponding to the input data.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1-8 are implemented.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1-8 are implemented.

Images