Memory management method and device for large language model

Abstract

The embodiment of the invention provides a memory management method and device for a large language model, and the method comprises the steps: obtaining an input lexical element which is a basic unit for processing the large language model; reasoning the input lexical unit through a large language model to obtain a key value cache of a first length, the key value cache of the first length being a key value cache corresponding to a lexical unit needing to be deleted by the input lexical unit; under the condition that the first length is smaller than a second length, the target memory management block is set to store the key value cache of the first length, and the second length is the maximum length, allowed to store the key value cache, of the target memory management block; and when the first length is equal to the second length, setting the target memory management block to cancel the storage of the key value cache of the first length. By means of the method and device, the problem that the memory management stability of the large language model is low is solved, and then the effect of improving the memory management stability of the large language model is achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, to a memory management method and device for a large language model. Background Art

[0002] In the related art, in the process of large language processing reasoning, in order to improve memory management efficiency, it is necessary to continuously discard or selectively save some generated word units, and it is also necessary to discard the frequently discarded key-value cache. Frequently discarding the key-value cache will generate a large amount of video memory fragmentation, which will lead to the problem of low memory management stability of large language models. Therefore, there is a problem of low memory management stability of large language models.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a memory management method and device for a large language model, so as to at least solve the problem of low stability of memory management of a large language model in the related art.

[0005] According to an embodiment of the present application, a memory management method for a large language model is provided, comprising: obtaining an input word, wherein the input word is a basic unit for processing the large language model; performing inference processing on the input word through the large language model to obtain a key-value cache of a first length, wherein the key-value cache of the first length is a key-value cache corresponding to the word to be deleted from the input word; when the first length is less than a second length, setting a target memory management block to store the key-value cache of the first length, wherein the second length is the maximum length allowed for the target memory management block to store the key-value cache; when the first length is equal to the second length, setting the target memory management block to cancel storing the key-value cache of the first length.

[0006] According to another embodiment of the present application, a memory management device for a large language model is provided, including: a first acquisition unit, used to acquire an input word, wherein the input word is a basic unit for processing the large language model; an inference unit, used to perform inference processing on the input word through the large language model to obtain a key-value cache of a first length, wherein the key-value cache of the first length is the key-value cache corresponding to the word to be deleted from the input word; a storage unit, used to set a target memory management block to store the key-value cache of the first length when the first length is less than a second length, wherein the second length is the maximum length allowed for the target memory management block to store the key-value cache; and a setting unit, used to set the target memory management block to cancel storing the key-value cache of the first length when the first length is equal to the second length.

[0007] As an optional solution, the above-mentioned reasoning unit includes: a first reasoning module, which is used to perform reasoning processing on the above-mentioned input word in the pre-filling stage through the above-mentioned large language model.

[0008] As an optional scheme, the above-mentioned reasoning module includes: a splitting sub-module, which is used to split the above-mentioned input word-gram when the number of word-grams of the above-mentioned input word-gram is greater than the upper limit of the reasoning word-gram, and perform reasoning processing on the split input word-grams in sequence, wherein the above-mentioned split input word-gram includes at least two segments of input word-grams, and the number of word-grams of each segment of the above-mentioned at least two segments of input word-grams is less than or equal to the above-mentioned upper limit of the reasoning word-grams.

[0009] As an optional solution, the splitting submodule includes: a setting subunit, which is used to set the input words of the first segment of the n segments of input word-grams to a first word-gram number, set the input word-gram number of the i-th segment of the n segments of input word-grams to a second word-gram number, and set the input word-gram number of the n-th segment of the n segments of input word-grams to a third word-gram number, wherein i is an integer greater than 1 and less than or equal to n, the first word-gram number is an integer multiple of the number corresponding to the second length, the first word-gram number is greater than the sum of the initial word-gram number and the rolling word-gram number, the initial word-gram number and the rolling word-gram number are used to determine the key-value cache to be deleted, the initial word-gram number is an integer multiple of the number corresponding to the second length, the second word-gram number, the initial word-gram number, and the rolling word-gram number are all integer multiples of the number corresponding to the second length, and the sum of the second word-gram number, the initial word-gram number, and the rolling word-gram number is less than or equal to the inference word-gram upper limit.

[0010] As an optional scheme, the above-mentioned setting sub-unit includes: a first reasoning component, which is used to use the first reasoning method to sequentially reason and process the above-mentioned split input word units when n is greater than 2; a second reasoning component, which is used to use the second reasoning method to sequentially reason and process the above-mentioned split input word units when n is equal to 2, wherein the above-mentioned second reasoning method is different from the above-mentioned first reasoning method.

[0011] As an optional scheme, the above-mentioned first reasoning component includes: a first reasoning sub-component, used to perform reasoning processing on the f input words in the first segment of the above-mentioned n-segment input word-grams, and obtain f key-value caches corresponding to the above-mentioned f input words; a second reasoning sub-component, used to perform reasoning processing on the s input words in the i-th segment of the above-mentioned n-segment input word-grams in combination with the above-mentioned f key-value caches, and obtain s key-value caches corresponding to the above-mentioned s input words, wherein i is an integer greater than 1 and less than n; a third reasoning sub-component, used to perform reasoning processing on the m input words in the n-th segment of the above-mentioned n-segment input word-grams in combination with the above-mentioned s key-value caches, and obtain m key-value caches.

[0012] As an optional solution, the second reasoning subcomponent includes: a first determination component, used to determine, from the f key-value caches, a first target number of key-value caches including the initial number of word units and the last number of rolling word units; a first reasoning component, used to perform reasoning processing on the s input words in combination with the first target number of key-value caches to obtain the s key-value caches.

[0013] As an optional solution, the third reasoning subcomponent includes: a second determination component, used to determine the second target number of key value caches including the initial word unit number and the last rolling word unit number from the first target number of key value caches and the s key value caches; a second reasoning component, used to perform reasoning processing on the m input words in combination with the second target number of key value caches to obtain the m key value caches.

[0014] As an optional scheme, the above-mentioned second reasoning component includes: a fourth reasoning sub-component, used to perform reasoning processing on the f input words in the first segment of the above-mentioned n-segment input word units, and obtain f key-value caches corresponding to the above-mentioned f input words; a fifth reasoning sub-component, used to perform reasoning processing on the m input words in the nth segment of the above-mentioned n-segment input word units in combination with the above-mentioned f key-value caches, and obtain m key-value caches.

[0015] As an optional scheme, the above-mentioned fifth reasoning sub-component includes: a third determination component, used to determine a second target number of key value caches from the above-mentioned f key value caches, wherein the above-mentioned second target number of key value caches is the key value cache after the above-mentioned f key value caches lose the third target number of key value caches, and the above-mentioned target number is the minimum number of the quotient of m and the number corresponding to the above-mentioned second length, rounded up; a third reasoning component, used to perform reasoning processing on the above-mentioned m input words in combination with the above-mentioned second target number of key value caches to obtain the above-mentioned m key value caches.

[0016] As an optional solution, the above-mentioned reasoning module includes: a decoding submodule, which is used to perform reasoning processing in the decoding stage on the key-value cache obtained in the above-mentioned pre-filling stage through the above-mentioned large language model.

[0017] As an optional scheme, the above-mentioned reasoning sub-module includes: a deletion sub-unit, which is used to delete the key value cache obtained in the above-mentioned pre-filling stage when the number of key value caches obtained in the above-mentioned pre-filling stage is greater than the upper limit of the key value cache, so as to obtain the deleted key value cache, wherein the number of the key value caches after deletion is less than or equal to the upper limit of the key value cache.

[0018] As an optional solution, the deletion subunit includes: a deletion component, which is used to delete the key value cache from the initial word unit number to the rolling word unit number in the key value cache obtained in the pre-filling stage to obtain the deleted key value cache, wherein the initial word unit number and the rolling word unit number are used to determine the key value cache to be deleted, and the initial word unit number and the rolling word unit number are both integer multiples of the number corresponding to the second length.

[0019] As an optional solution, the deletion subunit includes: a first output component, which is used to output the current new word and the current new key value cache through the large language model, in combination with the key value cache after the deletion, and the word obtained in the pre-filling stage; an acquisition component, which is used to acquire the output word when the current new word meets the termination condition, wherein the output word includes the word obtained in the pre-filling stage and the new word obtained in the pre-filling stage; a second output component, which is used to output the next new word and the next new key value cache through the large language model, in combination with the current new word and the current new key value cache, when the current new word does not meet the termination condition, and use the next new word as the current new word, and use the next new key value cache as the current new key value cache.

[0020] As an optional solution, the first acquisition unit includes: a first acquisition module, used to acquire an input word corresponding to a single word processing request; a second acquisition module, used to acquire a first input word corresponding to the first word processing request, and a second input word corresponding to the second word processing request; a combination module, used to combine the first input word and some of the words in the second input word through the large language model when the word amount of the first input word is less than the word processing upper limit, wherein the sum of the word amount of the first input word and the word amount of some of the words in the second input word is less than the word processing upper limit.

[0021] As an optional solution, the above-mentioned combination module includes: a combination sub-module, which is used to combine the above-mentioned first input word and part of the word-grams in the above-mentioned second input word-gram through the above-mentioned large language model when the word-gram quantity of the above-mentioned first input word-gram is less than the upper limit of the processing of the above-mentioned second word-gram.

[0022] As an optional solution, the above-mentioned inference unit includes: a pause module, which is used to suspend the inference processing of the above-mentioned key-value cache when the remaining space corresponding to the memory management space of the above-mentioned large language model is less than a preset threshold; and a continue module, which is used to continue the inference processing of the key-value cache whose inference processing has been suspended when the remaining space corresponding to the memory management space of the above-mentioned large language model is greater than or equal to the preset threshold.

[0023] As an optional solution, the above-mentioned device also includes: a second acquisition unit, used to obtain model parameters and data accuracy of the above-mentioned large language model; a third acquisition unit, used to obtain the memory space required for each of the above-mentioned key-value caches based on the above-mentioned model parameters and the above-mentioned data accuracy; a fourth acquisition unit, used to use the memory space required for each of the above-mentioned key-value caches and the memory occupied by the above-mentioned large language model according to the preset weight, to obtain the maximum length of the key-value cache allowed to be stored in the above-mentioned target memory management block.

[0024] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0025] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0026] Through the present application, an input word is obtained, wherein the input word is a basic unit for large language model processing; through the large language model, the input word is inferred and processed to obtain a key-value cache of a first length, wherein the key-value cache of the first length is the key-value cache corresponding to the word to be deleted from the input word; when the first length is less than the second length, the target memory management block is set to store the key-value cache of the first length, wherein the second length is the maximum length allowed for the target memory management block to store the key-value cache; when the first length is equal to the second length, the target memory management block is set to cancel the key-value cache storing the first length.

[0027] Specifically, after the large language model performs inference processing on the input word to obtain a key value cache of the first length, when the first length is less than the second length, the key value cache of the first length is stored, and when the key value cache to be deleted in the key value cache of the first length is equal to the number corresponding to the second length, the key value cache of the first number is cancelled, so that only when the key value cache of the first length is equal to the second length, the key value cache of the first number is cancelled, which reduces the frequency of discarding the key value cache and improves the stability of the memory management of the large language model. Since the problem of low stability of the memory management of the large language model can be solved, the technical effect of improving the stability of the memory management of the large language model is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of an application environment of a memory management method for a large language model according to an embodiment of the present application;

[0029] Figure 2 is a flowchart of a memory management method for a large language model according to an embodiment of the present application;

[0030] Figure 3 is a schematic diagram of a memory management method for a large language model according to an embodiment of the present application;

[0031] Figure 4 is a schematic diagram of a memory management method for a large language model according to an embodiment of the present application;

[0032] Figure 5 is a schematic diagram of a memory management method for a large language model according to an embodiment of the present application;

[0033] Figure 6 is a schematic diagram of a memory management method for a large language model according to an embodiment of the present application;

[0034] Figure 7 It is a structural block diagram of a memory management device for a large language model according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0037] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a memory management method for a large language model in an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0038] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the memory management method of the large language model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] In this embodiment, a memory management method for a large language model is provided. Figure 2 is a flowchart of a memory management method for a large language model according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0041] Step S202, obtaining an input word unit, wherein the input word unit is a basic unit for processing a large language model;

[0042] In an exemplary embodiment, the input word unit may be, but is not limited to, the smallest unit extracted from the input text, may be, but is not limited to, one or more characters, words, sub-word units or symbols, used for model input, may be, but is not limited to, obtained from an input sequence input into a large language model, and the input sequence may be, but is not limited to, a sequence input by a user into the large language model for reasoning.

[0043] In an exemplary embodiment, the large language model may refer to, but is not limited to, a deep learning model that can understand and generate complex natural language.

[0044] Step S204, performing inference processing on the input word unit through the large language model to obtain a key value cache of a first length, wherein the key value cache of the first length is a key value cache corresponding to the word unit to be deleted from the input word unit;

[0045] In an exemplary embodiment, the inference process may be understood as, but not limited to, a process of allowing a large language model to generate outputs based on input dimensions, and may include, but not limited to, understanding of inputs and prediction of outputs.

[0046] In an exemplary embodiment, the key-value cache may be, but is not limited to, a cache used to store generated keys and values ​​during large language model inference to avoid repeated calculations when generating subsequent word units. The first number of key-value caches may be, but is not limited to, the number of key-value caches generated and required to be stored in the inference step.

[0047] It should be noted that the key-value cache of the first length can be understood as, but is not limited to, a certain number of key-value caches. For further example, assuming that the length of the key-value cache is 8, it can be understood as 8 key-value caches.

[0048] In addition, when processing very long sequences, in order to avoid excessive memory consumption, it is necessary to periodically discard some key-value caches that are no longer needed. The number of key-value caches to be deleted can be, but is not limited to, the number of key-value caches that need to be removed in order to maintain efficient memory utilization under the current inference stage or memory management strategy.

[0049] In an exemplary embodiment, the memory management block may be, but is not limited to, a space unit for storing a specific number of key-value caches, and may be, but is not limited to, allowing the storage of a maximum number of key-value caches corresponding to the second length.

[0050] Step S206, when the first length is less than the second length, setting the target memory management block to store the key value cache of the first length, wherein the second length is the maximum length allowed by the target memory management block to store the key value cache;

[0051] In an exemplary embodiment, the target memory management block may be understood, but is not limited to, as a single memory management block designated for storing a key-value cache of a first length.

[0052] Step S208: When the first length is equal to the second length, set the target memory management block to cancel the key value cache storing the first length.

[0053] It should be noted that when the total amount of key-value cache is equal to the capacity of a single memory management block, that is, the second length, it means that the single memory management block is full. At this time, all key-value caches need to be discarded, and the memory management block needs to be cleared. The operation when the storage space reaches the upper limit and needs to be released is specified to ensure the continuity of the process and the efficiency of memory management.

[0054] Through the above steps, an input word is obtained, wherein the input word is a basic unit for large language model processing; through the large language model, the input word is inferred to obtain a key-value cache of a first length, wherein the key-value cache of the first length is the key-value cache corresponding to the word to be deleted from the input word; when the first length is less than the second length, the target memory management block is set to store the key-value cache of the first length, wherein the second length is the maximum length allowed for the target memory management block to store the key-value cache; when the first length is equal to the second length, the target memory management block is set to cancel the key-value cache storing the first length.

[0055] Specifically, after the large language model performs inference processing on the input word to obtain a key value cache of the first length, when the first length is less than the second length, the key value cache of the first length is stored, and when the key value cache to be deleted in the key value cache of the first length is equal to the number corresponding to the second length, the key value cache of the first number is cancelled, so that only when the key value cache of the first length is equal to the second length, the key value cache of the first number is cancelled, which reduces the frequency of discarding the key value cache and improves the stability of the memory management of the large language model. Since the problem of low stability of the memory management of the large language model can be solved, the technical effect of improving the stability of the memory management of the large language model is achieved.

[0056] The execution subject of the above steps may be a server, a terminal, etc., but is not limited thereto.

[0057] As an optional solution, a large language model is used to perform reasoning on the input word, including:

[0058] The large language model is used to perform inference processing on the input word in the pre-filling stage.

[0059] In an optional embodiment, the pre-filling stage may be understood as, but not limited to, an initial stage of the large language model reasoning process, and may include, but not limited to, a process in which the large language model receives and processes input word units, and infers and generates a series of word units.

[0060] Through the embodiment of the present application, the input word is subjected to inference processing in the pre-filling stage through a large language model, thereby achieving the technical effect of pre-filling the input word and generating the word.

[0061] As an optional solution, in the process of performing inference processing on the input word in the pre-filling stage by using the large language model, the method further includes:

[0062] When the number of word units of an input word unit is greater than the upper limit of inference word units, the input word unit is split and the split input word units are inferred sequentially, wherein the split input word unit includes at least two segments of input word units, and the number of word units of each segment of the at least two segments of input word units is less than or equal to the upper limit of inference word units.

[0063] In an optional embodiment, the inference word-gram upper limit may refer to, but is not limited to, the maximum number of word-grams that a model or hardware can process at one time during a large language model inference process.

[0064] In an optional embodiment, the split input word-grams can be understood, but not limited to, as when the number of word-grams in the input sequence exceeds the inference word-gram upper limit, the original input sequence is split to generate multiple subsequences, and the number of word-grams in each subsequence does not exceed the inference word-gram upper limit.

[0065] It should be noted that the upper limit of inference tokens is usually determined by the limitations of computing resources, memory size or model architecture. If this upper limit is exceeded, the inference performance of the large language model will be affected, or memory overflow may occur. When the total number of input tokens exceeds the upper limit that the model can process at one time, the input token sequence must be split into smaller parts, and then reasoning is performed one by one in the order of these smaller parts. The splitting and sequential reasoning strategy can significantly improve the practicality and efficiency of large language models in processing long texts or high-concurrency scenarios, while ensuring the accuracy and completeness of model reasoning.

[0066] Through the embodiment of the present application, when the number of word units of an input word unit is greater than the upper limit of the inference word unit, the input word unit is split, and the split input word units are sequentially inferred, wherein the split input word unit includes at least two segments of input word units, and the number of word units of each segment of the at least two segments of input word units is less than or equal to the upper limit of the inference word unit. The technical purpose of splitting the input word unit sequence into smaller parts and then inferring and processing them one by one in the order of these smaller parts is achieved, thereby achieving the technical effect of ensuring the accuracy and integrity of model reasoning.

[0067] As an optional solution, the split input word unit includes n input word units, wherein n is an integer greater than 1, and the input word unit is split, including:

[0068] The input words of the first segment of n input word-grams are set to the first word-gram number, the input word-gram number of the i-th segment of n input word-grams is set to the second word-gram number, and the input word-gram number of the n-th segment of n input word-grams is set to the third word-gram number, wherein i is an integer greater than 1 and less than or equal to n, the first word-gram number is an integer multiple of the number corresponding to the second length, the first word-gram number is greater than the sum of the initial word-gram number and the rolling word-gram number, the initial word-gram number and the rolling word-gram number are used to determine the key-value cache to be deleted, the initial word-gram number is an integer multiple of the number corresponding to the second length, the second word-gram number, the initial word-gram number, and the rolling word-gram number are all integer multiples of the number corresponding to the second length, and the sum of the second word-gram number and the initial word-gram number and the rolling word-gram number is less than or equal to the inference word-gram upper limit.

[0069] In an optional embodiment, the first word unit number may be but is not limited to being the word unit number of the first input word unit among n input word units, and the first word unit number may be but is not limited to being an integer multiple of the number corresponding to the second length.

[0070] In an optional embodiment, the second word number may be, but is not limited to, referring to the number of word units in each segment of the input word unit except the first segment and the nth segment after the input word unit sequence is split, and may be, but is not limited to, an integer multiple of the number corresponding to the second length.

[0071] In an optional embodiment, the third word number may be, but is not limited to, the word number of the last segment of input word units after the input word unit sequence is split.

[0072] In an optional embodiment, the initial word number may refer to, but is not limited to, the number of word units retained from the beginning of the sequence to ensure the accuracy of model reasoning when processing the input sequence. These word units may be used as, but are not limited to, auxiliary information in the subsequent reasoning process to assist the large language model in understanding the text information of the input sequence.

[0073] In an optional embodiment, the rolling word number may refer to, but is not limited to, the number of recent word units retained after each reasoning in order to maintain the consistency of model reasoning when processing an input sequence. These word units may be used as, but are not limited to, the latest input in subsequent reasoning to ensure the model's understanding and generation of the input sequence. Coherence.

[0074] It should be noted that the input word sequence is reasonably divided into multiple segments, and the number of words in each segment is set to adapt to the processing power and memory limitations of the large language model. This can ensure the accuracy of large language model reasoning and reduce processing delays when processing long texts or high-concurrency requests.

[0075] Through the embodiment of the present application, the input words of the first segment in the n-segment input word-gram are set to the first word-gram number, the input word-gram number of the i-th segment in the n-segment input word-gram is set to the second word-gram number, and the input word-gram number of the n-th segment in the n-segment input word-gram is set to the third word-gram number, wherein i is an integer greater than 1 and less than or equal to n, the first word-gram number is an integer multiple of the number corresponding to the second length, the first word-gram number is greater than the sum of the initial word-gram number and the rolling word-gram number, the initial word-gram number and the rolling word-gram number are used to determine the key value cache to be deleted, the initial word-gram number is an integer multiple of the number corresponding to the second length, the second word-gram number, the initial word-gram number, and the rolling word-gram number are all integer multiples of the number corresponding to the second length, and the sum of the second word-gram number and the initial word-gram number and the rolling word-gram number is less than or equal to the upper limit of the inference word-gram. The technical purpose of reasonably splitting the input word-gram sequence into multiple segments and setting the word-gram number of each segment is achieved, thereby achieving the technical effect of ensuring the accuracy of large language model reasoning.

[0076] The split input word-grams include n segments of input word-grams, where n is an integer greater than 1. The split input word-grams are sequentially inferred and processed, including:

[0077] S1-1, when n is greater than 2, the first reasoning method is adopted to perform reasoning on the split input word units one by one;

[0078] S1-2, when n is equal to 2, the second reasoning method is adopted to perform reasoning processing on the split input word units in sequence, wherein the second reasoning method is different from the first reasoning method.

[0079] In an optional embodiment, the first reasoning method may be, but is not limited to, an inference processing strategy adopted by a large language model when the input word sequence is split into more than two segments.

[0080] In an optional embodiment, the second reasoning method may be, but is not limited to, referring to the reasoning processing strategy adopted by the large language model when the input word sequence is split into only two segments.

[0081] It should be noted that by selecting different inference methods according to the number of splits of the input word units, it is possible to flexibly adapt to the inference processing solutions of different scenarios, ensuring that when the large language model processes ultra-long input sequences, no matter how many segments the sequence is split into, it can perform inference processing efficiently and accurately. At the same time, it optimizes memory usage, avoids resource waste, and improves the inference performance and response speed of the large language model.

[0082] Through the embodiment of the present application, when n is greater than 2, the first reasoning method is adopted to perform reasoning processing on the split input word units in sequence; when n is equal to 2, the second reasoning method is adopted to perform reasoning processing on the split input word units in sequence, wherein the second reasoning method is different from the first reasoning method. The purpose of being able to perform reasoning processing efficiently and accurately regardless of how many segments the sequence is split into is achieved, thereby achieving the technical effect of improving the reasoning performance and response speed of the large language model.

[0083] As an optional solution, the first reasoning method is adopted to perform reasoning processing on the split input word units in sequence, including:

[0084] S2-1, perform inference processing on the f input words in the first segment of the n input word units to obtain f key-value caches corresponding to the f input words;

[0085] S2-2, combining the f key-value caches, performing inference processing on the s input words in the i-th segment of the n-segment input word unit, and obtaining s key-value caches corresponding to the s input words, where i is an integer greater than 1 and less than n;

[0086] S2-3, combining s key-value caches, performing inference processing on the m input words in the nth segment of the n-segment input word units to obtain m key-value caches.

[0087] It should be noted that by using the key-value cache generated by the reasoning in the first segment to perform reasoning on the input word of the i-th segment and obtain S new key-value caches, the continuous understanding and reasoning of the input sequence by the large language model is ensured, so that the reasoning result of the middle segment depends not only on the input word of this segment, but also on the key-value cache generated by the previous reasoning, thereby maintaining the coherence of the reasoning process.

[0088] Furthermore, by using the s key-value caches generated by inference to perform inference processing, m key-value caches are generated, the reasoning of the input word is completed, and complete information is provided for the subsequent decoding stage.

[0089] Through the embodiment of the present application, the f input words in the first segment of the n-segment input word unit are inferred and processed to obtain f key value caches corresponding to the f input words; combined with the f key value caches, the s input words in the i-th segment of the n-segment input word unit are inferred and processed to obtain s key value caches corresponding to the s input words, where i is an integer greater than 1 and less than n; combined with the s key value caches, the m input words in the n-th segment of the n-segment input word unit are inferred and processed to obtain m key value caches. The technical purpose of obtaining S new key value caches by inferring the i-th segment input word unit using the key value cache generated by the inference of the first segment is achieved, thereby ensuring the consistency of the reasoning of the large language model on the input sequence.

[0090] As an optional solution, combined with f key-value caches, the s input words in the i-th segment of the n-segment input word unit are inferred and processed to obtain s key-value caches, including:

[0091] S3-1, determining a key value cache having a first target number of initial word units and a final rolling word unit from among f key value caches;

[0092] S3-2, combining the first target number of key-value caches, performing inference processing on the s input words to obtain s key-value caches.

[0093] It should be noted that after processing the f input words in the first paragraph and generating f key-value caches, a specific number of key-value caches, namely the key-value caches corresponding to the initial word unit number and the rolling word unit number, are selected from these key-value caches as the basis for subsequent reasoning. This can ensure the consistency and reasoning accuracy in the subsequent reasoning process, while avoiding unnecessary memory usage and improving resource utilization.

[0094] Furthermore, in the reasoning process of the i-th segment, the key-value cache of the first target number determined previously is used to perform reasoning on the s input words of the current segment, and generate new s key-value caches for subsequent reasoning, thereby ensuring the consistency and efficiency of the model during the reasoning process. By utilizing the key cache information in the previous reasoning results, the large language model can understand and generate word units more accurately, avoiding repeated calculations and waste of resources.

[0095] Through the embodiment of the present application, the key value caches of the first target number of initial word units and the last rolling word unit number are determined from the f key value caches; combined with the key value caches of the first target number, the s input words are inferred and processed to obtain s key value caches. The technical purpose of using the key value caches corresponding to the initial word unit number and the rolling word unit number as the basis for subsequent reasoning is achieved, and the technical effect of avoiding resource waste is realized.

[0096] As an optional solution, combined with s key-value caches, the m input words in the nth segment of the n-segment input word unit are inferred and processed to obtain m key-value caches, including:

[0097] S4-1, determining a second target number of key value caches having a first target number of key value caches and a final number of rolling word caches from the first target number of key value caches and s key value caches;

[0098] S4-2, combining the second target number of key-value caches, performing inference processing on the m input words to obtain m key-value caches.

[0099] It should be noted that after processing the s input words in the middle segment and generating s key-value caches, it is necessary to filter out a specific number of caches from these s caches and the first target number of key-value caches retained. This part of the cache contains key-value pairs corresponding to the initial word unit number and the rolling word unit number for use in the last stage of reasoning processing, ensuring that the model can use the most relevant information during the last stage of reasoning processing, while reducing unnecessary memory usage and improving resource utilization and reasoning efficiency.

[0100] Furthermore, in the last stage of reasoning processing, the second target number of key-value caches screened out are combined with the m input words of the current stage for reasoning processing to generate m key-value caches, completing the reasoning of the entire input sequence, thereby ensuring the coherence and accuracy of the entire input sequence reasoning process. Especially when processing ultra-long sequence inputs, by effectively managing and utilizing the key-value cache, the model can understand and generate word units more accurately, and at the same time, it is more efficient in memory management, avoiding repeated calculations and waste of resources, and improving the reasoning efficiency of large language models.

[0101] Through the embodiment of the present application, the number of initial word units and the second target number of key value caches of the last rolling word units are determined from the first target number of key value caches and s key value caches; combined with the second target number of key value caches, m input words are inferred and processed to obtain m key value caches. The technical purpose of more accurately understanding and generating word units by effectively managing and utilizing key value caches when processing ultra-long sequence inputs is achieved, and the technical effect of improving the inference efficiency of large language models is achieved.

[0102] As an optional solution, the second reasoning method is used to perform reasoning on the split input word units one by one, including:

[0103] S5-1, performing inference processing on the f input words in the first segment of the n input word units to obtain f key-value caches corresponding to the f input words;

[0104] S5-2, combining f key-value caches, performing inference processing on the m input words in the nth segment of the n-segment input word units to obtain m key-value caches.

[0105] It should be noted that in the pre-filling stage, the first segment of the input sequence is inferred and processed to generate a key-value cache equal to the number of input words for subsequent reasoning. A part of the key-value cache can be generated in advance to reduce the computational overhead of the subsequent decoding stage and improve the response speed and processing efficiency of the model.

[0106] Furthermore, when processing the last segment of input words, the existing f key-value caches are used as context information to perform inference on this part of the input words, generate corresponding m key-value caches, and complete the inference process of the entire input sequence, ensuring the coherence and contextual integrity of the last segment of the inference process, avoiding repeated calculations, and improving memory resource utilization, thereby enhancing the performance and response speed of the large language model when processing long sequence inputs.

[0107] Through the embodiment of the present application, the f input words in the first segment of the n-segment input word-gram are inferred and processed to obtain f key-value caches corresponding to the f input words; combined with the f key-value caches, the m input words in the nth segment of the n-segment input word-gram are inferred and processed to obtain m key-value caches. The technical purpose of using the existing f key-value caches as context information to infer this part of the input word-gram and generate the corresponding m key-value caches when processing the last segment of the input word-gram is achieved, thereby achieving the technical effect of improving the utilization rate of memory resources.

[0108] As an optional solution, combined with f key-value caches, the m input words in the nth segment of the n-segment input word unit are inferred and processed to obtain m key-value caches, including:

[0109] S6-1, determining a second target number of key value caches from the f key value caches, wherein the second target number of key value caches is the key value cache after the f key value caches lose the third target number of key value caches, and the target number is the minimum number of the quotient of m and the number corresponding to the second length, rounded up;

[0110] S6-2, combining the second target number of key-value caches, performing inference processing on the m input words to obtain m key-value caches.

[0111] In an optional embodiment, the second target number of key-value caches is first but not limited to being understood as a group of key-value caches selected from f key-value caches for accelerating reasoning when processing the last segment of input word units.

[0112] In an optional embodiment, the third target number of key-value caches: the third target number of key-value caches is the number of key-value caches that need to be discarded from the f key-value caches when determining the second target number of key-value caches.

[0113] It should be noted that when processing the last segment of input tokens, the third target number of key-value caches to be eliminated are determined from the f key-value caches generated in the pre-filling phase according to a specific calculation rule, i.e., the minimum number of the quotient of m and the number corresponding to the second length rounded up. The remaining key-value caches, i.e., the second target number of key-value caches, will be used for the last segment of reasoning processing, ensuring that when processing the last segment of input tokens, the model can accurately use the most relevant and necessary contextual information for reasoning, while avoiding excessive memory usage and improving resource utilization efficiency and reasoning speed.

[0114] Furthermore, the second target number of key-value caches selected from the f key-value caches are used to perform inference processing on the m input words in the nth input sequence, generate m key-value caches, and complete the processing of the entire input sequence, thereby ensuring the accuracy and consistency of model reasoning. Even when processing the last segment of input word units, the key-value cache information can be fully utilized to reduce repeated calculations, accelerate the reasoning process, and improve the reasoning performance of large language models when processing long sequence inputs.

[0115] Through the embodiment of the present application, the second target number of key value caches is determined from the f key value caches, wherein the second target number of key value caches is the key value cache after the f key value caches lose the third target number of key value caches, and the target number is the quotient of m and the number corresponding to the second length, rounded up to the smallest number; combined with the second target number of key value caches, the m input words are inferred and processed to obtain m key value caches. The technical purpose of making full use of key value cache information, reducing repeated calculations, and accelerating the inference process when processing the last segment of input word units is achieved, thereby improving the inference performance of large language models when processing long sequence inputs.

[0116] As an optional solution, after the input word is subjected to inference processing in the pre-filling phase by the large language model, the method further includes:

[0117] The key-value cache obtained in the pre-filling phase is used through a large language model to perform inference processing in the decoding phase.

[0118] In an optional embodiment, the decoding stage can be understood, but is not limited to, as a stage in which, after an initial word element is generated in the pre-filling stage, new word elements are gradually generated based on the existing key-value cache until a specific termination condition is met.

[0119] It should be noted that the goal of the decoding stage is to generate new words or complete sequence generation based on the existing key-value cache. Through the inference processing of the decoding stage, the model can quickly generate new words based on the key-value cache generated in the pre-filling stage. This not only speeds up the generation process, but also improves the response speed and processing efficiency of the model. When dealing with real-time and streaming data processing needs, it can significantly improve user experience and large language model performance. In addition, during the decoding process, by utilizing the key-value cache generated in the pre-filling stage, it is also possible to avoid repeated calculations, reduce memory usage, improve real-time response capabilities and overall processing speed, and improve the inference performance of large language models.

[0120] Through the embodiment of the present application, the key value cache obtained in the pre-filling stage is subjected to inference processing in the decoding stage through the large language model. The technical purpose of quickly generating new word units based on the key value cache generated in the pre-filling stage is achieved, thereby achieving the technical effect of improving the inference performance of the large language model.

[0121] As an optional solution, the key-value cache obtained in the pre-filling phase is processed by a large language model for inference in the decoding phase, including:

[0122] When the number of key value caches obtained in the pre-filling stage is greater than the upper limit of the key value cache, the key value caches obtained in the pre-filling stage are deleted to obtain a deleted key value cache, wherein the number of key value caches after deletion is less than or equal to the upper limit of the key value cache.

[0123] It should be noted that when the number of key-value caches generated in the pre-filling phase exceeds the key-value cache upper limit set for the large language model, the cache needs to be deleted. At this time, the most relevant key-value cache information should be retained, and the non-critical part should be discarded to ensure that the memory resource limit is not exceeded. This can effectively avoid memory overflow and reduce resource waste. It ensures that when processing very long sequences or high-concurrency requests, the model can automatically adjust the storage of the key-value cache to avoid excessive consumption of memory resources and cause the reasoning performance of the large language model to decline.

[0124] Through the embodiment of the present application, when the number of key-value caches obtained in the pre-filling stage is greater than the upper limit of the key-value cache, the key-value caches obtained in the pre-filling stage are deleted to obtain a deleted key-value cache, wherein the number of deleted key-value caches is less than or equal to the upper limit of the key-value cache. The storage of the key-value cache is automatically adjusted, and the technical effect of avoiding excessive consumption of memory resources and resulting in a decrease in the reasoning performance of the large language model is achieved.

[0125] As an optional solution, the key-value cache obtained in the pre-filling phase is deleted to obtain a deleted key-value cache, including:

[0126] The key value cache from the initial word unit number to the rolling word unit number in the key value cache obtained in the pre-filling stage is deleted to obtain a deleted key value cache, wherein the initial word unit number and the rolling word unit number are used to determine the key value cache to be deleted, and the initial word unit number and the rolling word unit number are both integer multiples of the number corresponding to the second length.

[0127] It should be noted that after the pre-filling stage is completed, the key-value cache from the initial word number to the rolling word number is deleted from the key-value cache, where the initial word number and the rolling word number are both integer multiples of the number corresponding to the second length, so as to avoid retaining unnecessary intermediate key-value caches, reduce memory usage, and ensure the consistency of context information. It can effectively manage memory resources, avoid memory overflow and fragmentation, and ensure that large language models retain key context information during the reasoning process, thereby improving reasoning efficiency and memory resource utilization when processing ultra-long sequence inputs or streaming data.

[0128] Through the embodiment of the present application, the key value cache from the initial word number to the rolling word number in the key value cache obtained in the pre-filling stage is deleted to obtain the deleted key value cache, wherein the initial word number and the rolling word number are used to determine the key value cache to be deleted, and the initial word number and the rolling word number are both integer multiples of the number corresponding to the second length. The technical purpose of discarding unnecessary intermediate key value caches and reducing memory usage is achieved, thereby achieving the technical effect of improving the reasoning efficiency of large language models and memory resource utilization.

[0129] As an optional solution, after deleting the key value cache obtained in the pre-filling phase to obtain the deleted key value cache, the method further includes:

[0130] S7-1, execute the following steps until an output word is obtained:

[0131] S7-2, output the current new word and the current new key value cache through the large language model, combined with the deleted key value cache and the word obtained in the pre-filling stage;

[0132] S7-3, when the current new word element meets the termination condition, obtaining an output word element, wherein the output word element includes the word element obtained in the pre-filling stage and the new word element obtained in the pre-filling stage;

[0133] S7-4, when the current new word does not meet the termination condition, the next new word and the next new key value cache are output through the large language model in combination with the current new word and the current new key value cache, and the next new word is used as the current new word, and the next new key value cache is used as the current new key value cache.

[0134] In an optional embodiment, the termination condition may be understood as, but is not limited to, a condition for determining when to stop generating new word units during the decoding process.

[0135] In an optional embodiment, at the beginning of the decoding phase, the large language model may, but is not limited to, generate new word units and corresponding key-value caches based on the key-value caches and word unit inputs generated in the pre-filling phase.

[0136] In an optional embodiment, when the newly generated word element reaches a predefined termination condition, which may include but is not limited to the generation of a terminator or the reaching of an output length limit, the word element generated in the pre-filling stage and the decoding stage is used as the final output word element.

[0137] In an optional embodiment, when the new word does not meet the termination condition, the large language model can, but is not limited to, use the existing word and key-value cache data to generate the next word and the corresponding key-value cache, and then use the generated word and key-value cache for the next word generation process until the output requirements are met.

[0138] It should be noted that based on the optimized key-value cache and word-unit input, new words are continuously generated until the predefined termination condition is reached, ensuring that the text generated by the model is not only coherent but also accurately reflects the context of the input sequence. By dynamically managing the key-value cache, unnecessary repeated calculations and memory consumption are avoided, improving the efficiency of model reasoning and computing resource utilization.

[0139] Through the embodiment of the present application, the current new word and the current new key value cache are output through the large language model in combination with the deleted key value cache and the word obtained in the pre-filling stage; when the current new word meets the termination condition, the output word is obtained, wherein the output word includes the word obtained in the pre-filling stage and the new word obtained in the pre-filling stage; when the current new word does not meet the termination condition, the next new word and the next new key value cache are output through the large language model in combination with the current new word and the current new key value cache, and the next new word is used as the current new word, and the next new key value cache is used as the current new key value cache. The technical purpose of continuously generating new words based on the optimized key value cache and word input until the termination condition is met is achieved, and the technical effect of improving the efficiency of large language model reasoning and the utilization rate of computing resources is achieved.

[0140] As an optional solution, obtaining input tokens includes:

[0141] S8-1, obtaining an input word corresponding to a single word processing request;

[0142] The method also includes:

[0143] S8-2, obtaining a first input word corresponding to the first word processing request and a second input word corresponding to the second word processing request;

[0144] S8-3, when the word size of the first input word is less than the upper limit of word processing, the first input word and part of the word sizes in the second input word are combined and processed through the large language model, wherein the sum of the word size of the first input word and the word size of part of the word sizes in the second input word is less than the upper limit of word processing.

[0145] In an optional embodiment, when the number of word units in the first word unit request is less than the processing upper limit, the request can be combined with the word units in the second word unit request through a large language model, but is not limited to being input into the model as a whole for processing. This can more effectively utilize computing resources, avoid resource waste caused by allocating independent resources to each request, and improve the inference speed and response time of the model.

[0146] It should be noted that when the number of tokens in a single token processing request is less than the processing limit, the tokens in multiple requests can be combined and processed as a whole. By combining requests for processing, resource consumption can be significantly reduced, the processing speed and responsiveness of the model can be improved, while maintaining the quality of reasoning. This can play an important role in dialogue systems, text generation services, and multi-task reasoning scenarios, and can optimize the resource allocation of the model to ensure that large language models can run at a relatively high efficiency when processing multiple requests.

[0147] Through the embodiment of the present application, the input word corresponding to a single word processing request is obtained; the first input word corresponding to the first word processing request and the second input word corresponding to the second word processing request are obtained; when the word quantity of the first input word is less than the word processing upper limit, the first input word and some of the words in the second input word are combined and processed by the large language model, wherein the sum of the word quantity of the first input word and the word quantity of some of the words in the second input word is less than the word processing upper limit. The technical purpose of combining the request with the word of the second word request through the large language model when the word quantity of the first word request is less than the processing upper limit is achieved, and the technical effect of improving the processing efficiency of the large language model is achieved.

[0148] As an optional solution, a single word-gram processing request corresponds to a first word-gram processing upper limit, and multiple word-gram processing requests correspond to a second word-gram processing upper limit, wherein the second word-gram processing upper limit is less than or equal to a word-gram processing upper limit, and when the word-gram amount of the first input word-gram is less than the word-gram processing upper limit, the first input word-gram and part of the word-grams in the second input word-gram are combined and processed by the large language model, including:

[0149] When the word quantity of the first input word is less than the upper limit of processing the second word, the first input word and part of the word in the second input word are combined and processed by the large language model.

[0150] In an optional embodiment, when the number of word units in the first word unit request is less than the word unit upper limit set by the large language model to simultaneously process two or more requests, the word units in the first word unit and the second word unit request can be combined, but not limited to, and then processed through the large language model. This can improve the utilization of computing resources, reduce delays in the model inference process, and process multiple word unit requests at the same time, thereby improving the overall processing efficiency and throughput of the large language model, ensuring that the model can handle more concurrent requests while running efficiently.

[0151] It should be noted that in multi-task or high-concurrency scenarios, computing resources can be used more effectively, resource waste can be reduced, and the model's reasoning speed and response time can be improved. Through combined processing, data from multiple requests can be packaged, but not limited to, to make the processing of each batch more efficient. Combined processing can also alleviate bottleneck problems in resource-constrained environments to a certain extent, providing a more flexible solution for the deployment and operation of large language models.

[0152] Through the embodiment of the present application, when the word size of the first input word is less than the upper limit of the second word processing, the first input word and part of the word in the second input word are combined and processed by the large language model. The technical purpose of processing multiple word requests at the same time is achieved, thereby achieving the technical effect of improving the processing efficiency of the large language model.

[0153] As an optional solution, the process of reasoning about input words through a large language model includes:

[0154] S9-1, when the remaining space corresponding to the memory management space of the large language model is less than a preset threshold, suspending the reasoning processing of the key-value cache;

[0155] S9-2, when the remaining space corresponding to the memory management space of the large language model is greater than or equal to a preset threshold, the reasoning process is continued for the key-value cache for which the reasoning process has been suspended.

[0156] In an optional embodiment, when the remaining memory in the memory management space used by the large language model is lower than a preset threshold, further reasoning processing on the key-value cache can be suspended, but is not limited to, to avoid memory overflow or excessive use of resources. This can effectively prevent the computing device from crashing due to insufficient memory, ensure the stability and reliability of the large language model, and avoid excessive consumption of resources to maintain the efficient operation of the large language model.

[0157] In an optional embodiment, once the remaining memory in the memory management space is restored to or above a preset threshold, the reasoning process of the previously suspended key-value cache can be restarted, but is not limited to, to continue generating output tokens. By dynamically monitoring memory usage and intelligently adjusting the processing flow according to the remaining space, the large language model can flexibly respond to different loads and memory requirements, ensuring that services can continue to be provided under resource-constrained conditions and avoiding reasoning interruptions caused by insufficient memory.

[0158] It should be noted that by real-time monitoring of memory usage and dynamically adjusting the inference strategy, it is possible to ensure the quality of model inference while effectively avoiding performance degradation or service interruption caused by insufficient memory. Application scenarios that require real-time response and large-scale deployment can significantly improve the stability and user experience of large language models. In addition, by real-time monitoring of memory usage and dynamic adjustment of inference strategies, large language models can also handle more requests and tasks under limited computing resources.

[0159] Through the embodiment of the present application, when the remaining space corresponding to the memory management space of the large language model is less than the preset threshold, the reasoning process of the key value cache is suspended; when the remaining space corresponding to the memory management space of the large language model is greater than or equal to the preset threshold, the reasoning process of the key value cache for which the reasoning process has been suspended is continued. The technical purpose of enabling the large language model to process more requests and tasks under limited computing resources is achieved, thereby achieving the technical effect of improving the efficiency of the large language model in processing requests and tasks.

[0160] As an optional solution, the memory management method of the large language model further includes:

[0161] S10-1, obtaining model parameters and data accuracy of the large language model;

[0162] S10-2, acquiring the memory space required for each key-value cache according to the model parameters and the data precision;

[0163] S10-3, using the memory space required for each of the key-value caches and the memory occupied by the large language model according to the preset weight, obtain the maximum length of the key-value cache allowed to be stored in the target memory management block.

[0164] In an optional embodiment, the model parameters may include, but are not limited to, the number of layers, the number of key-value pair attention heads, and the dimension of the head.

[0165] In an optional embodiment, the data precision may be, but is not limited to, the representation precision during the calculation process, and may be, but is not limited to, a 32-bit floating point number or a 16-bit floating point number.

[0166] It should be noted that the capacity of the memory management block will affect the number of disk I / O operations, and thus affect the speed of continuous data reading. The larger the memory management block capacity, the fewer disk I / O operations and the faster the continuous data reading speed. However, if the capacity of the memory management block is too large, it may lead to a waste of memory resources. In addition, if the capacity of the memory management block is too small, it will cause excessive memory fragmentation, thus affecting the normal performance of the system. Therefore, for the overall performance of the system, it is necessary to set an appropriate memory management block capacity.

[0167] Furthermore, in order to set an appropriate memory management block capacity, it is necessary to obtain the size of the key-value cache corresponding to the word unit, which can be understood as the need to obtain the unit memory. In this embodiment, the unit memory is obtained through the following formula: Unit memory = number of layers × number of key-value pair attention heads × dimension of the head × bytes occupied by data precision × 2, where each key-value pair includes a key and a value, and therefore, the bytes occupied by data precision should be multiplied by 2.

[0168] Through the embodiment of the present application, the model parameters and data precision of the large language model are obtained; according to the model parameters and the data precision, the memory space required for each key value cache is obtained; using the memory space required for each key value cache and the memory occupied by the large language model according to the preset weight, the maximum length of the target memory management block allowed to store the key value cache is obtained. Thus, the technical purpose of setting the appropriate memory management block capacity is achieved, and then the technical effect of improving the overall performance of the system is achieved.

[0169] As an optional solution, the above-mentioned memory management method of the large language model is applied to the scenario of large language model reasoning.

[0170] In an optional embodiment, in the prefill phase, if the number of input tokens exceeds the upper limit of the number of tokens that can be inferred and calculated by the board at one time or exceeds the set threshold of the number of tokens inferred at one time, the input is split and inferred, and the first token is generated; otherwise, the first token is directly inferred;

[0171] In an optional embodiment, in the decoding stage (decode stage), if the kvcache length of the request exceeds a certain limit, the kvcache corresponding to a portion of the middle block is discarded, and reasoning continues to generate a new token.

[0172] To further illustrate, the optional Figure 3 As shown, the specific steps are as follows:

[0173] S302, in the prefill stage: if the number of requested input tokens exceeds the upper limit, the input is split and inferred, and the first token is generated; otherwise, the first token stage is directly inferred;

[0174] S304, in the decoding stage: if the length of the key value cache (kvcache) of the request exceeds a certain limit, the kvcache corresponding to a portion of the intermediate memory management block (block) is discarded, and reasoning continues to generate a new token;

[0175] S306, when the generated token is a terminator or the number of generated tokens reaches a set threshold, the reasoning is terminated.

[0176] In an optional embodiment, it is assumed that when a certain board uses paged-attention technology to schedule multi-task reasoning of a large language model, the size of the block is set to bn, that is, each block can store bn kvcaches. In approximate reasoning, the number of initial tokens retained is set to in (where in % bn = 0), and the corresponding number of rolling tokens is set to rn (where rn % bn = 0), where the paged-attention technology can be understood as, but not limited to, an attention algorithm inspired by the concept of virtual memory and paging in traditional operating systems.

[0177] In an optional embodiment, the steps of multi-task reasoning of a large language model are as follows:

[0178] Step 1: In the prefill phase, if the number of input request tokens qn exceeds the upper limit of the number of tokens that the board can calculate at one time or exceeds the upper limit of the number of input tokens that can be inferred at one time (the upper limit is represented by tn. In paged-attention mode, tn%bn=0), then go to step 1.3 to split the input and generate the first token; otherwise, go to step 1.2 to directly infer the first token.

[0179] Step 1.2: Pass the input request and the corresponding mask and position_id into the large language model to obtain the kvcache of all input requests and the first token for decoding stage reasoning, where mask is a qn×qn lower triangular matrix and position_id is a vector from 0 to qn-1.

[0180] Step 1.3, split the input request, perform reasoning separately, and generate the first token, as follows;

[0181] Step 1.3.1, divide the request with token number qn into n segments (n>1), where the length of the first segment is fn, the length of the second to n-1 segments is sn, and the length of the nth segment is the remaining length nn, that is, nn=qn-fn-(n-2)*sn; where fn % bn = 0, fn> in + rn, fn ≤ tn, sn % bn = 0; sn + in + rn ≤ tn;

[0182] Step 1.3.2, pass the first segment request and the corresponding mask and position_id into the large language model, and obtain the kvcache corresponding to fn requests for subsequent segment request calculations, where mask is a lower triangular matrix of fn×fn, and position_id is a vector from 0 to fn-1. If n>2, proceed to step 1.3.3, otherwise skip step 1.3.3 and go directly to step 1.3.4.

[0183] Step 1.3.3, for each segment from the 2nd segment to the n-1th segment, called the i-th segment (2≤i≤n-1), perform the following operations in sequence: for all previous kvcaches (actually the kvcaches corresponding to fn or sn+in+rn tokens), only keep the kvcaches corresponding to the first in and the last rn tokens, discard the kvcaches corresponding to the middle tokens, obtain the kvcaches corresponding to in+rn tokens, and pass them and the i-th segment request and the corresponding mask and position_id into the large language model, and newly obtain sn kvcaches for the subsequent segment request calculation, where mask is a special matrix such as Figure 4 As shown, the matrix sn×(in+rn+sn) is formed by concatenating the all-1 matrix sn×(in+rn) and the lower triangular matrix sn×sn in the row direction, and position_id is a vector from fn+(i-2)*sn to fn+(i-1)*sn-1.

[0184] Step 1.3.4, for the nth segment input request of length nn, let dnn = ⌈nn / bn⌉×bn, round up ⌈x⌉ to represent the smallest integer greater than or equal to x, for all previous kvcaches (actually the kvcaches corresponding to fn or sn+in+rn tokens), discard the kvcaches corresponding to the in tokens of dnn, and obtain the kvcaches corresponding to lnn (lnn is actually fn-dnn or sn+in+rn-dnn) tokens, and pass it and the nth segment request and the corresponding mask and position_id into the large language model, and newly obtain nn kvcaches and corresponding tokens for subsequent decoding stage calculations, where mask is a special matrix, which is a nn×lnn all-1 matrix and a nn×nn lower triangular matrix concatenated in the row direction to form a nn×(nn+lnn) matrix, and position_id is a vector from fn+(n-2)*sn to qn-1.

[0185] It should be noted that the discarding operation in steps 1.3.3 and 1.3.4 is only for the convenience of expression. In the actual reasoning process, the relevant kvcache can be directly not saved during the calculation process according to the above logic.

[0186] To further illustrate, the calculation logic diagram of the entire prefill phase is as follows Figure 5 As shown:

[0187] S502, determine whether the number of requested tokens exceeds the threshold, if yes, execute S504, otherwise execute S506;

[0188] S504, divide the request with the token number qn into n segments (n>1), where the length of the first segment is fn, the length of the second to n-1 segments is sn, and the length of the nth segment is the remaining length nn, that is, qn-fn-(n-2)*sn;

[0189] S506: Pass the input request and the corresponding mask and position_id into the large language model to obtain the kvcache of all input requests and the first token for decoding stage reasoning, where mask is a qn×qn lower triangular matrix and position_id is a vector from 0 to qn-1.

[0190] S508, calculating the kvcache of the first segment length fn;

[0191] S510, determine whether n is greater than 2, if yes, execute S512, otherwise directly execute S514;

[0192] S512, for each segment from the 2nd segment to the n-1th segment, called the i-th segment (2≤i≤n-1), perform the following operations in sequence: for all previous kvcaches, only keep the kvcaches corresponding to the first in and the last m tokens, and pass them and the i-th segment request and the corresponding mask and position_id to the large language model, and the new ones are in sn kvcaches;

[0193] S514, for the n-th segment input request with a length of nn, set dnn=[nn / bn]×bn. For all previous kvcaches, discard the kvcaches corresponding to the tokens of in after dnn, and pass them and the n-th segment request and the corresponding mask and position_id into the large language model, and obtain nn new kvcaches and the first token for reasoning in the decode stage.

[0194] Step 2: This step describes the computational reasoning process in the decode phase. If the exit condition mentioned in step 3 is not met, multiple reasonings are performed in the decode phase. For details of each reasoning, see steps 2.1 to 2.2.

[0195] Step 2.1: In the decode phase, if the number of existing kvcaches reaches the set support upper limit tn, delete the kvcaches corresponding to tokens in to in+bn (starting from 1) and proceed to step 2.2; if the existing kvcache does not reach the set support upper limit tn, proceed directly to step 2.2;

[0196] Step 2.2, assuming that there are currently a total of din tokens corresponding to the kvcache, then the retained kvcache, the new token obtained from the last reasoning, and the corresponding mask and position_id are used as the input of the large language model, where mask is a 1×(1+din) two-dimensional all-one matrix, position_id is a one-dimensional vector containing only din data, and the output is a new token and the corresponding kvcache, which can be used for the next reasoning.

[0197] Step 3 describes the exit judgment condition for the end of large language model inference. That is, before each inference in step 2, the following judgment is made: whether the newly generated token is the terminator of the model, if it is the terminator, the inference ends; whether the total number of generated tokens reaches the set threshold, if it reaches the threshold, the inference ends.

[0198] To further illustrate, the reasoning process of step 2 and step 3 is as follows Figure 6 As shown, the specific steps are as follows:

[0199] S602, whether the token is a terminator or the total number of tokens reaches a threshold, if yes, execute S604, otherwise execute S610;

[0200] S604, whether the number of existing kvcaches reaches the set support upper limit tn, if yes, execute S606, otherwise execute S608;

[0201] S606, delete the kvcache corresponding to tokens in to in+bn;

[0202] S608, assuming that there are currently din tokens corresponding to kvcaches, the retained kvcache, the new token obtained by the last reasoning, and the corresponding mask and position_id are used as the input of the large language model, and the output is a new token and the corresponding kvcache;

[0203] S610, end reasoning.

[0204] In an optional embodiment, if the upper limit of the number of tokens input for one reasoning is tn, in the case of multiple batches, the upper limit of the number of tokens input for one reasoning in a single batch is stn (stn≤tn), then in the prefill phase, if the length of a request is less than tn, it can be spliced ​​with other requests or part of other requests for continue batch, so the space occupied by the spliced ​​request is block_num*bn and is not greater than tn, where block_num is the number of blocks occupied by kvcache.

[0205] Furthermore, in the decode phase, multiple batches can be decoded and reasoned at the same time. If there is still a certain amount of kvcache space remaining, a new request can be added to combine the decode phase and the prefill phase for reasoning. The mask and position_id of each request are consistent with those described in the scheme in Section 2.2.1. In the decode reasoning phase, when the kvcache space is insufficient, the kvcache of some requests is transferred from the accelerated computing card to the CPU, and the corresponding request reasoning is suspended; when the kvcache still has a certain amount of space, the kvcache on the CPU is transferred back to the computing acceleration card to continue reasoning for related requests.

[0206] It should be noted that the embodiment of the present application can process multiple requests at the same time, thereby improving the processing speed of the requests. In addition, even if the number of tokens input into a single request is less than tn but greater than stn, the approximate reasoning scheme proposed in this embodiment can improve the computing performance to a certain extent compared to the existing computing scheme.

[0207] It should be noted that each block can store kvcache information corresponding to a certain number of tokens. The corresponding number can be called block_size, and the memory space occupied is called block_mem. Each block can be assigned a unique block ID corresponding to a segment of address; the number of blocks that can be allocated in memory is block_num. Each block only stores the kvcache of the same request, and does not store the kvcache of other requests.

[0208] Furthermore, for different computing accelerator cards and different models, the block_size value is generally 32 / 64 / 128 or 256, which is generally a power of 2. This size generally results in a faster reading speed for the board.

[0209] For different large language models, the space required for each token corresponding to the kvcache is different, and its value is unit_mem = num_hidden_layers×num_key_value_heads×head_dim×bytes×2, where num_hidden_layers is the number of layers of the transformers structure in the large language model, num_key_value_heads and head_dim are the number of parameters kv heads and head dimensions in the multi-head self-attention mechanism in the large language model, bytes is the number of bytes occupied by calculating data accuracy, and 2 means one copy of k and one copy of v; num_hidden layers, num_key_value_heads, and head_dim may be different for each model.

[0210] For different models, since the space required for each token in kvcache is different, even if the parameter block_size in the block is the same, the size of block_mem corresponding to the block is different. The size of the available allocated block in the corresponding memory is Memb, which is "total memory of the board - memory occupied by weights - memory occupied by temporary calculations and the system", then block_num = Memb divided by block_mem, and rounded down.

[0211] For a specific model, block_size determines block_mem, and the space available for allocating blocks in the board affects the size of block_num. block_size should not be too small, as it will affect the speed of data reading and storage; block_size should not be too large, as it will lead to too small block_num, which will reduce the number of blocks available for allocation to different requests, which is not conducive to memory scheduling, and will also lead to too few blocks that can be discarded. Each time, a relatively large amount of kvcache is stored before it can be discarded.

[0212] In addition, the method in this embodiment is not only applicable to single-card reasoning, but also to multi-card reasoning (including multi-card pipeline parallelism, multi-card tensor parallelism, and multi-card mixed parallelism) or multi-machine reasoning.

[0213] Through the embodiments of the present application, the reasoning of the prefill stage and the decode stage of the large language model are optimized respectively, and the existing large language model reasoning service system based on the paged-attention technology can be directly connected, achieving the technical purpose of only modifying the requested kvcache space management strategy without modifying the calculation logic and the underlying operators, thereby achieving the technical effect of improving the computing performance of the large language model.

[0214] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0215] In this embodiment, a memory management device for a large language model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0216] Figure 7 is a structural block diagram of a memory management device for a large language model according to an embodiment of the present application, such as Figure 7 As shown, the device comprises:

[0217] A first acquisition unit 702 is used to acquire an input word, wherein the input word is a basic unit for large language model processing;

[0218] The inference unit 704 is used to perform inference processing on the input word by using the large language model to obtain a key-value cache of a first length, wherein the key-value cache of the first length is a key-value cache corresponding to the word to be deleted from the input word;

[0219] The storage unit 706 is used to set the target memory management block to store the key value cache of the first length when the first length is less than the second length, wherein the second length is the maximum length allowed by the target memory management block to store the key value cache;

[0220] The setting unit 708 is used to set the target memory management block to cancel the key value cache storing the first length when the first length is equal to the second length.

[0221] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0222] As an optional solution, the above-mentioned reasoning unit 704 includes: a first reasoning module, which is used to perform reasoning processing on the above-mentioned input word in the pre-filling stage through the above-mentioned large language model.

[0223] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0224] As an optional scheme, the above-mentioned reasoning module includes: a splitting sub-module, which is used to split the above-mentioned input word-gram when the number of word-grams of the above-mentioned input word-gram is greater than the upper limit of the reasoning word-gram, and perform reasoning processing on the split input word-grams in sequence, wherein the above-mentioned split input word-gram includes at least two segments of input word-grams, and the number of word-grams of each segment of the above-mentioned at least two segments of input word-grams is less than or equal to the above-mentioned upper limit of the reasoning word-grams.

[0225] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0226] As an optional solution, the splitting submodule includes: a setting subunit, which is used to set the input words of the first segment of the n segments of input word-grams to a first word-gram number, set the input word-gram number of the i-th segment of the n segments of input word-grams to a second word-gram number, and set the input word-gram number of the n-th segment of the n segments of input word-grams to a third word-gram number, wherein i is an integer greater than 1 and less than or equal to n, the first word-gram number is an integer multiple of the number corresponding to the second length, the first word-gram number is greater than the sum of the initial word-gram number and the rolling word-gram number, the initial word-gram number and the rolling word-gram number are used to determine the key-value cache to be deleted, the initial word-gram number is an integer multiple of the number corresponding to the second length, the second word-gram number, the initial word-gram number, and the rolling word-gram number are all integer multiples of the number corresponding to the second length, and the sum of the second word-gram number, the initial word-gram number, and the rolling word-gram number is less than or equal to the inference word-gram upper limit.

[0227] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0228] As an optional scheme, the above-mentioned setting sub-unit includes: a first reasoning component, which is used to use the first reasoning method to sequentially reason and process the above-mentioned split input word units when n is greater than 2; a second reasoning component, which is used to use the second reasoning method to sequentially reason and process the above-mentioned split input word units when n is equal to 2, wherein the above-mentioned second reasoning method is different from the above-mentioned first reasoning method.

[0229] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0230] As an optional scheme, the above-mentioned first reasoning component includes: a first reasoning sub-component, used to perform reasoning processing on the f input words in the first segment of the above-mentioned n-segment input word-grams, and obtain f key-value caches corresponding to the above-mentioned f input words; a second reasoning sub-component, used to perform reasoning processing on the s input words in the i-th segment of the above-mentioned n-segment input word-grams in combination with the above-mentioned f key-value caches, and obtain s key-value caches corresponding to the above-mentioned s input words, wherein i is an integer greater than 1 and less than n; a third reasoning sub-component, used to perform reasoning processing on the m input words in the n-th segment of the above-mentioned n-segment input word-grams in combination with the above-mentioned s key-value caches, and obtain m key-value caches.

[0231] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0232] As an optional solution, the second reasoning subcomponent includes: a first determination component, used to determine, from the f key-value caches, a first target number of key-value caches including the initial number of word units and the last number of rolling word units; a first reasoning component, used to perform reasoning processing on the s input words in combination with the first target number of key-value caches to obtain the s key-value caches.

[0233] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0234] As an optional solution, the third reasoning subcomponent includes: a second determination component, used to determine the second target number of key value caches including the initial word unit number and the last rolling word unit number from the first target number of key value caches and the s key value caches; a second reasoning component, used to perform reasoning processing on the m input words in combination with the second target number of key value caches to obtain the m key value caches.

[0235] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0236] As an optional scheme, the above-mentioned second reasoning component includes: a fourth reasoning sub-component, used to perform reasoning processing on the f input words in the first segment of the above-mentioned n-segment input word units, and obtain f key-value caches corresponding to the above-mentioned f input words; a fifth reasoning sub-component, used to perform reasoning processing on the m input words in the nth segment of the above-mentioned n-segment input word units in combination with the above-mentioned f key-value caches, and obtain m key-value caches.

[0237] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0238] As an optional scheme, the above-mentioned fifth reasoning sub-component includes: a third determination component, used to determine a second target number of key value caches from the above-mentioned f key value caches, wherein the above-mentioned second target number of key value caches is the key value cache after the above-mentioned f key value caches lose the third target number of key value caches, and the above-mentioned target number is the minimum number of the quotient of m and the number corresponding to the above-mentioned second length, rounded up; a third reasoning component, used to perform reasoning processing on the above-mentioned m input words in combination with the above-mentioned second target number of key value caches to obtain the above-mentioned m key value caches.

[0239] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0240] As an optional solution, the above-mentioned reasoning module includes: a decoding submodule, which is used to perform reasoning processing in the decoding stage on the key-value cache obtained in the above-mentioned pre-filling stage through the above-mentioned large language model.

[0241] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0242] As an optional scheme, the above-mentioned reasoning sub-module includes: a deletion sub-unit, which is used to delete the key value cache obtained in the above-mentioned pre-filling stage when the number of key value caches obtained in the above-mentioned pre-filling stage is greater than the upper limit of the key value cache, so as to obtain the deleted key value cache, wherein the number of the key value caches after deletion is less than or equal to the upper limit of the key value cache.

[0243] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0244] As an optional solution, the deletion subunit includes: a deletion component, which is used to delete the key value cache from the initial word unit number to the rolling word unit number in the key value cache obtained in the pre-filling stage to obtain the deleted key value cache, wherein the initial word unit number and the rolling word unit number are used to determine the key value cache to be deleted, and the initial word unit number and the rolling word unit number are both integer multiples of the number corresponding to the second length.

[0245] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0246] As an optional solution, the deletion subunit includes: a first output component, which is used to output the current new word and the current new key value cache through the large language model, in combination with the key value cache after the deletion, and the word obtained in the pre-filling stage; an acquisition component, which is used to acquire the output word when the current new word meets the termination condition, wherein the output word includes the word obtained in the pre-filling stage and the new word obtained in the pre-filling stage; a second output component, which is used to output the next new word and the next new key value cache through the large language model, in combination with the current new word and the current new key value cache, when the current new word does not meet the termination condition, and use the next new word as the current new word, and use the next new key value cache as the current new key value cache.

[0247] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0248] As an optional solution, the first acquisition unit 702 includes: a first acquisition module, used to acquire an input word corresponding to a single word processing request; a second acquisition module, used to acquire a first input word corresponding to the first word processing request, and a second input word corresponding to the second word processing request; a combination module, used to combine the first input word and some of the words in the second input word through the large language model when the word amount of the first input word is less than the word processing upper limit, wherein the sum of the word amount of the first input word and the word amount of some of the words in the second input word is less than the word processing upper limit.

[0249] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0250] As an optional solution, the above-mentioned combination module includes: a combination sub-module, which is used to combine the above-mentioned first input word and part of the word-grams in the above-mentioned second input word-gram through the above-mentioned large language model when the word-gram quantity of the above-mentioned first input word-gram is less than the upper limit of the processing of the above-mentioned second word-gram.

[0251] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0252] As an optional solution, the above-mentioned reasoning unit 704 includes: a pause module, which is used to pause the reasoning processing of the above-mentioned key-value cache when the remaining space corresponding to the memory management space of the above-mentioned large language model is less than a preset threshold; and a continue module, which is used to continue the reasoning processing of the key-value cache whose reasoning processing has been suspended when the remaining space corresponding to the memory management space of the above-mentioned large language model is greater than or equal to the preset threshold.

[0253] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0254] As an optional solution, the above-mentioned device also includes: a second acquisition unit, used to obtain model parameters and data accuracy of the above-mentioned large language model; a third acquisition unit, used to obtain the memory space required for each of the above-mentioned key-value caches based on the above-mentioned model parameters and the above-mentioned data accuracy; a fourth acquisition unit, used to use the memory space required for each of the above-mentioned key-value caches and the memory occupied by the above-mentioned large language model according to the preset weight, to obtain the maximum length of the key-value cache allowed to be stored in the above-mentioned target memory management block.

[0255] For specific embodiments, reference may be made to the example shown in the memory management method for a large language model described above, which will not be described in detail in this example.

[0256] It should be noted that the above-mentioned virtual devices (modules, units, sub-modules, sub-units, components, etc.) can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above-mentioned virtual devices are all located in the same processor; or, the above-mentioned virtual devices are located in different processors in any combination.

[0257] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0258] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0259] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0260] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0261] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0262] Obviously, those skilled in the art should understand that the above-mentioned virtual devices or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0263] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A memory management method for a large language model, characterized in that: include: Obtaining an input word unit, wherein the input word unit is a basic unit for large language model processing; Performing inference processing on the input word unit through the large language model to obtain a key-value cache of a first length, wherein the key-value cache of the first length is a key-value cache corresponding to the word unit to be deleted from the input word unit; When the first length is less than the second length, setting the target memory management block to store the key value cache of the first length, wherein the second length is the maximum length allowed by the target memory management block to store the key value cache; When the first length is equal to the second length, the target memory management block is set to cancel the key value cache storing the first length.

2. The method according to claim 1, characterized in that The inference processing of the input word by the large language model includes: The input word is subjected to inference processing in a pre-filling phase through the large language model.

3. The method according to claim 2, characterized in that In the process of performing inference processing of the input word in the pre-filling phase by the large language model, the method further includes: When the word number of the input word is greater than the upper limit of inference word, the input word is split and the split input word is sequentially inferred, wherein the split input word includes at least two segments of input word, and the word number of each segment of the at least two segments of input word is less than or equal to the upper limit of inference word.

4. The method according to claim 3, characterized in that The split input word unit includes n input word units, wherein n is an integer greater than 1, and the input word unit is split, including: The input words of the first segment of the n segments of input word-grams are set to the first word-gram number, the input word-gram number of the i-th segment of the n segments of input word-grams is set to the second word-gram number, and the input word-gram number of the n-th segment of the n segments of input word-grams is set to the third word-gram number, wherein i is an integer greater than 1 and less than or equal to n, the first word-gram number is an integer multiple of the number corresponding to the second length, the first word-gram number is greater than the sum of the initial word-gram number and the rolling word-gram number, the initial word-gram number and the rolling word-gram number are used to determine the key-value cache to be deleted, the initial word-gram number is an integer multiple of the number corresponding to the second length, the second word-gram number, the initial word-gram number, and the rolling word-gram number are all integer multiples of the number corresponding to the second length, and the sum of the second word-gram number and the initial word-gram number and the rolling word-gram number is less than or equal to the inference word-gram upper limit.

5. The method according to claim 4, characterized in that The split input word units include n input word units, where n is an integer greater than 1, and the sequential reasoning processing of the split input word units includes: When n is greater than 2, the first reasoning method is adopted to perform reasoning processing on the split input word units one by one; When n is equal to 2, a second reasoning method is used to perform reasoning processing on the split input word units in sequence, wherein the second reasoning method is different from the first reasoning method.

6. The method according to claim 5, characterized in that The first reasoning method is adopted to perform reasoning processing on the split input word units in sequence, including: Performing inference processing on the f input words in the first segment of the n input word units to obtain f key-value caches corresponding to the f input words; In combination with the f key-value caches, the s input words in the i-th segment of the n input word units are inferred to obtain s key-value caches corresponding to the s input words, where i is an integer greater than 1 and less than n; In combination with the s key-value caches, the m input words in the nth segment of the n input word units are inferred to obtain m key-value caches.

7. The method according to claim 6, characterized in that The combining the f key-value caches and performing inference processing on the s input words in the i-th segment of the n input word units to obtain s key-value caches includes: Determine, from the f key value caches, a key value cache having the initial word unit number and a first target number of the rolling word unit number; In combination with the first target number of key-value caches, the s input words are inferred to obtain the s key-value caches.

8. The method according to claim 6, characterized in that The combining of the s key-value caches and performing inference processing on the m input words in the nth segment of the n input word units to obtain m key-value caches includes: Determine the first target number of key value caches and the s key value caches to obtain the first target number of word units and the second target number of rolling word units; In combination with the second target number of key-value caches, the m input words are subjected to inference processing to obtain the m key-value caches.

9. The method according to claim 5, characterized in that The second reasoning method is used to perform reasoning processing on the split input word units in sequence, including: Performing inference processing on the f input words in the first segment of the n input word units to obtain f key-value caches corresponding to the f input words; In combination with the f key-value caches, the m input words in the nth segment of the n input word units are inferred to obtain m key-value caches.

10. The method according to claim 9, characterized in that The combining the f key-value caches and performing inference processing on the m input words in the nth segment of the n input word units to obtain m key-value caches includes: Determine a second target number of key value caches from the f key value caches, wherein the second target number of key value caches is the key value caches after the f key value caches lose the third target number of key value caches, and the target number is the minimum number of the quotient of m and the number corresponding to the second length, rounded up; In combination with the second target number of key-value caches, the m input words are subjected to inference processing to obtain the m key-value caches.

11. The method according to claim 2, characterized in that After the input word is subjected to inference processing in the pre-filling phase by the large language model, the method further includes: The key-value cache obtained in the pre-filling stage is subjected to inference processing in the decoding stage through the large language model.

12. The method according to claim 11, characterized in that The key-value cache obtained in the pre-filling stage is subjected to inference processing in the decoding stage by using the large language model, including: When the length of the key value cache obtained in the pre-filling stage is greater than the upper limit of the key value cache, the key value cache obtained in the pre-filling stage is deleted to obtain a deleted key value cache, wherein the length of the deleted key value cache is less than or equal to the upper limit of the key value cache.

13. The method according to claim 12, characterized in that The deleting the key value cache obtained in the pre-filling stage to obtain the deleted key value cache includes: The key value cache from the initial word unit number to the rolling word unit number in the key value cache obtained in the pre-filling stage is deleted to obtain the deleted key value cache, wherein the initial word unit number and the rolling word unit number are used to determine the key value cache to be deleted, and the initial word unit number and the rolling word unit number are both integer multiples of the number corresponding to the second length.

14. The method according to claim 12, characterized in that After deleting the key-value cache obtained in the pre-filling stage to obtain the deleted key-value cache, the method further includes: Perform the following steps until the output word is obtained: Outputting the current new word and the current new key value cache by combining the deleted key value cache and the word obtained in the pre-filling stage through the large language model; When the current new word-gram satisfies the termination condition, obtaining the output word-gram, wherein the output word-gram includes the word-gram obtained in the pre-filling stage and the new word-gram obtained in the pre-filling stage; When the current new word does not satisfy the termination condition, the large language model is used to combine the current new word and the current new key value cache to output the next new word and the next new key value cache, and the next new word is used as the current new word, and the next new key value cache is used as the current new key value cache.

15. The method according to claim 1, characterized in that The obtaining of input word-grams includes: obtaining input word-grams corresponding to a single word-gram processing request; The method further comprises: Obtaining a first input word corresponding to a first word processing request and a second input word corresponding to a second word processing request; When the word size of the first input word is less than the upper limit of word processing, the first input word and some of the word sizes in the second input word are combined and processed by the large language model, wherein the sum of the word size of the first input word and the word size of some of the word sizes in the second input word is less than the upper limit of word processing.

16. The method according to claim 15, characterized in that The single word-gram processing request corresponds to a first word-gram processing upper limit, and the multiple word-gram processing requests correspond to a second word-gram processing upper limit, wherein the second word-gram processing upper limit is less than or equal to the word-gram processing upper limit, and when the word-gram amount of the first input word-gram is less than the word-gram processing upper limit, combining and processing the first input word-gram and part of the word-grams in the second input word-gram by using the large language model, includes: When the word quantity of the first input word is less than the upper limit of processing the second word, the first input word and part of the word in the second input word are combined and processed by the large language model.

17. The method according to any one of claims 1 to 16, characterized in that The process of performing reasoning processing on the input word by using the large language model includes: When the remaining space corresponding to the memory management space of the large language model is less than a preset threshold, suspending the inference processing of the key-value cache; When the remaining space corresponding to the memory management space of the large language model is greater than or equal to a preset threshold, the reasoning process is continued for the key-value cache for which the reasoning process has been suspended.

18. The method according to any one of claims 1 to 16, characterized in that The method further comprises: Obtaining model parameters and data accuracy of the large language model; According to the model parameters and the data precision, obtaining the memory space required for each key-value cache; The maximum length of the key-value cache allowed to be stored in the target memory management block is obtained by using the memory space required for each of the key-value caches and the memory occupied by the large language model according to the preset weight.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 17 when executed by a processor.

20. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 17 are implemented.

Images