Key-value cache reuse method, and related apparatus

Abstract

Disclosed in the embodiments of the present application are a key-value cache reuse method, and a related apparatus, which can be applied to scenarios such as cloud technology, artificial intelligence, intelligent transportation, assisted driving and the Internet of Things. The method comprises: after acquiring an ith-round prompt, on the basis of historical token groups respectively generated by a plurality of tokens included in the ith-round prompt with the previous (i-1)th-round prompt, and according to token arrangement orders, matching tokens having the same token arrangement order, so as to obtain a prefix token group and remaining token groups; acquiring from a graphics processing unit a key-value cache of the prefix token group, and if the acquisition fails, acquiring from a central processing unit a key-value cache corresponding to the prefix token group; and sending the key-value cache of the prefix token group and the remaining token groups to an inference engine, such that the inference engine performs inference on the basis of the key-value cache of the prefix token group and the remaining token groups to obtain response content for the ith-round prompt. Thus, an inference engine directly reuses a key-value cache of a prefix token group without requiring recalculation, thereby improving the speed of inference computation.

Description

Method and related device for reusing key-value cache

[0001] The present application claims priority from the Chinese patent application No. 2024109030896 filed on July 5, 2024, and entitled "Method and related device for reusing key-value cache", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, in particular to the reuse of key-value cache. BACKGROUND

[0003] With the development of artificial intelligence technology, large language models (LLM) have been continuously highly concerned due to their strong generation, understanding, reasoning and other capabilities. However, large language models require a large amount of computing resources when providing reasoning services, resulting in slow real-time reasoning speed.

[0004] In related technologies, not only is the model reasoning calculation accelerated by using a graphics processing unit (GPU), but also data that needs to be accessed multiple times in the model reasoning process is stored in the graphics processing unit in the form of a key-value cache (KV Cache), so that the model can directly access the key-value cache in the subsequent reasoning process without the need for recalculation, that is, the reasoning speed is improved by sacrificing storage space.

[0005] However, the graphics processing unit has limited video memory, which needs a large amount of video memory to support the calculations required in the model reasoning process, resulting in less video memory available for storing key-value cache. In related technologies, the key-value cache generated in each reasoning is generally released from the graphics processing unit after each reasoning, resulting in a small number of reusable key-value caches and limiting the application scenarios. SUMMARY

[0006] To solve the above technical problems, the present application provides a method and related device for reusing key-value cache, which can increase the number of reusable key-value caches while ensuring the reasoning speed and expanding the range of application scenarios.

[0007] The present application discloses the following technical solutions:

[0008] In one aspect, the present application provides a method for reusing key-value cache, comprising:

[0009] obtaining an i-th prompt word, i being an integer greater than 1;

[0010] According to the plurality of segmented words included in the i th round of prompt words and the historical segmented word group, segmented words with the same segmented word arrangement order are matched according to the segmented word arrangement order, to obtain a prefix segmented word group and a remaining segmented word group, the historical segmented word group is segmented words with the segmented word arrangement order obtained based on the first i-1 rounds of prompt words, the first i-1 rounds of prompt words have a correlation relationship with the i th round of prompt words, the prefix segmented word group includes segmented words in the i th round of prompt words that are matched successfully for the first time, and the remaining segmented word group includes segmented words in the i th round of prompt words other than the prefix segmented word group;

[0011] If the key-value cache corresponding to the prefix segmented word group fails to be obtained from the graphics processor, the key-value cache corresponding to the prefix segmented word group is obtained from the central processor;

[0012] The key-value cache of the prefix segmented word group and the remaining segmented word group are sent to the inference engine, so that the inference engine obtains response content for the i th round of prompt words according to the key-value cache of the prefix segmented word group and the remaining segmented word group.

[0013] In another aspect, an embodiment of the present application provides a key-value cache multiplexing device, the device comprising: an acquisition unit, a matching unit, a multiplexing unit, and a sending unit;

[0014] The acquisition unit is configured to acquire i th round of prompt words, i being an integer greater than 1;

[0015] The matching unit is configured to match segmented words with the same segmented word arrangement order according to the plurality of segmented words included in the i th round of prompt words and the historical segmented word group according to the segmented word arrangement order, to obtain a prefix segmented word group and a remaining segmented word group, the historical segmented word group being segmented words with the segmented word arrangement order obtained based on the first i-1 rounds of prompt words, the first i-1 rounds of prompt words having a correlation relationship with the i th round of prompt words, the prefix segmented word group including segmented words in the i th round of prompt words that are matched successfully for the first time, and the remaining segmented word group including segmented words in the i th round of prompt words other than the prefix segmented word group;

[0016] The multiplexing unit is configured to obtain the key-value cache corresponding to the prefix segmented word group from the central processor if the key-value cache corresponding to the prefix segmented word group fails to be obtained from the graphics processor;

[0017] The sending unit is configured to send the key-value cache of the prefix segmented word group and the remaining segmented word group to the inference engine, so that the inference engine obtains response content for the i th round of prompt words according to the key-value cache of the prefix segmented word group and the remaining segmented word group.

[0018] In another aspect, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory:

[0019] The memory is configured to store the computer program and transmit the computer program to the processor.

[0020] The processor is configured to execute the method according to the instructions in the computer program.

[0021] In another aspect, the embodiments of the present application provide a computer readable storage medium configured to store a computer program, the computer program being configured to execute the method according to the above aspects.

[0022] In another aspect, the embodiments of the present application provide a computer program product or a computer program, the computer program product or the computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the method according to the above aspects.

[0023] From the above technical solution, if the video memory of the graphics processor cannot store the key-value cache, the key-value cache is no longer directly released, but a part of the key-value cache is stored in the central processor, so that the key-value cache is stored by the graphics processor and the central processor together, to further increase the number of reusable key-value caches. Moreover, compared with other storage spaces, the central processor and the graphics processor have faster interaction speed to ensure the inference speed. Thus, while ensuring the inference speed, the number of reusable key-value caches is increased, and the range of application scenarios is expanded. The process of reusing the key-value cache is described below.

[0024] When the i-th round of prompt words is obtained, according to the plurality of words included in the i-th round of prompt words and the historical word groups generated by the first i-1 rounds of prompt words, the words with the same word arrangement order are matched according to the word arrangement order, to obtain a prefix word group and a remaining word group, that is, the reusable words and the non-reusable words in the i-th round of prompt words are determined by matching. Then, the key-value cache of the prefix word group is obtained from the graphics processor, if the acquisition fails, the key-value cache of the prefix word group may be moved to the central processor for storage, and the key-value cache corresponding to the prefix word group is obtained from the central processor. Finally, the key-value cache of the prefix word group and the remaining word group are sent to the inference engine, so that the inference engine performs inference according to the key-value cache of the prefix word group and the remaining word group, to obtain the response content for the i-th round of prompt words. Thus, in the calculation process of the inference engine, the key-value cache of the prefix word group does not need to be recalculated, but is directly reused, and only the key-value cache of the remaining word group needs to be calculated, so that the inference speed is improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] FIG. 1 is a schematic diagram of an application scenario of a key-value cache multiplexing method according to an embodiment of the present application;

[0026] FIG. 2 is a schematic diagram of a multi-turn dialogue request scenario of a key-value cache multiplexing method according to an embodiment of the present application;

[0027] FIG. 3 is a schematic diagram of a code completion scenario of a key-value cache multiplexing method according to an embodiment of the present application;

[0028] FIG. 4 is a schematic diagram of a long text inquiry scenario of a key-value cache multiplexing method according to an embodiment of the present application;

[0029] FIG. 5 is a schematic diagram of a flow of a key-value cache multiplexing method according to an embodiment of the present application;

[0030] FIG. 6 is a schematic diagram of a multi-turn dialogue request scenario according to an embodiment of the present application;

[0031] FIG. 7 is a schematic diagram of an application scenario of a key-value cache multiplexing method according to an embodiment of the present application;

[0032] FIG. 8 is a schematic diagram of an application scenario of a key-value cache multiplexing method according to an embodiment of the present application;

[0033] FIG. 9 is a schematic diagram of a result of end-to-end delay decomposition according to an embodiment of the present application;

[0034] FIG. 10 is a diagram of experimental results of a multi-turn interaction scenario according to an embodiment of the present application;

[0035] FIG. 11 is a schematic diagram of a structure of a key-value cache multiplexing apparatus according to an embodiment of the present application;

[0036] FIG. 12 is a schematic diagram of a structure of a server according to an embodiment of the present application;

[0037] FIG. 13 is a schematic diagram of a structure of a terminal device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] Embodiments of the present application will be described below with reference to the accompanying drawings.

[0039] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is merely for distinguishing between the similar objects and the same can be referenced by different terms and can be mutually exchanged under proper circumstances without changing the scope and spirit of the present application. The terms "comprise", "comprising", "include", "including", and the like, used in the present description and in the claims are inclusive rather than exclusive, that is, specific integers, steps, operations, or elements are mentioned not excluding any additions thereof. Any processes, methods, systems, products, or devices described herein can have additional steps, methods, systems, products, or devices, and not every aspect or embodiment described herein has every feature or step described. It is also possible for one or more features, steps or functions to be combined in a single or shared item or step.

[0040] In the related art, in order to improve the number of reusable key-value caches while ensuring inference speed, a paged attention mechanism is introduced, that is, the key-value cache is managed in the form of paging, so that continuous key-values can be stored in non-continuous video memory space, that is, as many key-value caches as possible are stored in limited storage space. In this way, internal or external fragmentation of key-value caches can be avoided, thereby improving the number of requests that can be simultaneously inferred, and ultimately improving the overall utilization of the graphics processor.

[0041] It should be noted that whether it is to store key-value caches or to use the paged attention mechanism, the key-value caches stored are generally the key-value caches within a request (i.e., within a round of conversation), which will be deleted after each inference ends. Although the storage space increases after the introduction of the paged attention mechanism, it can not be necessary to immediately release the key-value caches generated in this inference from the graphics processor after each inference ends. However, when a new space needs to be allocated on the video memory, the previous key-value caches will be immediately released.

[0042] That is, even if more key-value caches are stored in limited storage space as much as possible through the paged attention mechanism, the previous key-value caches will still be immediately released when a new space needs to be allocated on the video memory. Thus, the number of reusable key-value caches is still small, which limits the application scenarios.

[0043] For example, if the paging attention mechanism is used, the key-value cache generated in each round of dialogue does not need to be deleted immediately after each inference. For example, in a graphics processor, only the key-value cache generated in the i-1th round of dialogue and the i-2th round of dialogue can be stored, and when the key-value cache generated in the i th round of dialogue needs to be stored, the key-value cache generated in the i-2th round of dialogue also needs to be released. Therefore, even in the application scenario of multiple rounds of dialogue, the relevance of the query statement is greater, such as the number of repeated segmented words in the query statement of the i th round of dialogue and the query statement of the i-2th round of dialogue. In the inference process for the i th round of dialogue, because the key-value cache generated in the i-2th round of dialogue has been released, it cannot be reused and can only be recalculated, thereby limiting the number of reusable key-value caches.

[0044] Based on this, the embodiment of the application provides a key-value cache reuse method. For the application scenario in which the key-value cache needs to be reused more, if the graphics processor cannot store the key-value cache, the key-value cache is not directly released, but a part of the key-value cache is stored in the central processor, so that the key-value cache is stored by the graphics processor and the central processor together, thereby further increasing the number of reusable key-value caches. Moreover, compared with other storage spaces, the central processor and the graphics processor have a faster interaction speed to ensure the inference speed. Therefore, while ensuring the inference speed, the number of reusable key-value caches is increased, and the range of application scenarios is expanded.

[0045] The key-value cache reuse method provided by the application can be applied to a computer device with a key-value cache reuse capability, such as a terminal device and a server.

[0046] The terminal device can be a desktop computer, a notebook computer, a smart phone, a tablet computer, an Internet of Things device, and a portable wearable device. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, and the like. The smart vehicle device can be a vehicle navigation terminal and a vehicle computer. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, and the like, but is not limited thereto.

[0047] The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server or a server cluster providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, and the like. The terminal device and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the application.

[0048] To facilitate understanding of the key-value cache reuse method provided in the embodiments of the present application, the application scenario of the key-value cache reuse method is exemplarily introduced below taking the server as an example of an execution subject of the key-value cache reuse method.

[0049] Referring to FIG. 1, it is a schematic diagram of an application scenario of a key-value cache reuse method provided in the embodiments of the present application. As shown in FIG. 1, the application scenario includes a terminal device 110, a server 120 and an inference engine 130. The terminal device 110 and the server 120, and the server 120 and the inference engine 130 can communicate through a communication network. The communication network uses standard communication technology and / or protocol, and is usually the Internet, but can also be any network, including but not limited to Bluetooth, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, a dedicated network or any combination of a virtual private network. In some embodiments, custom or dedicated data communication technology can be used instead of or in addition to the above data communication technology.

[0050] The terminal device 110 is installed with a client that provides services such as a conversation service, a code completion service and the like, and a user can input corresponding content through the corresponding client. Taking the conversation service as an example, the client can be a client that provides the conversation service, and the user can input a query statement through the client. The terminal device 110 sends the content input by the user through the client for the i-th time to the server 120.

[0051] The server 120 is installed with services corresponding to the client in the terminal device 110, such as a conversation service, a code completion service and the like. The server 120 generates the i-th round of prompt words according to the content sent by the terminal device 110, matches the same word arrangement order of the words in the i-th round of prompt words according to the word arrangement order of the words included in the i-th round of prompt words and the historical word groups generated by the previous i-1 rounds of prompt words, respectively, to obtain a prefix word group and a remaining word group, that is, to determine the reusable words and the non-reusable words in the i-th round of prompt words through matching.

[0052] In the embodiments of the present application, if the video memory of the graphics processor cannot store the key-value cache, the key-value cache is not directly released, but a part of the key-value cache is stored in the central processor, so that the key-value cache is stored by the graphics processor and the central processor together, to further increase the number of reusable key-value caches. Moreover, compared with other storage spaces, the central processor and the graphics processor have faster interaction speed, to ensure the inference speed. Thus, while ensuring the inference speed, the number of reusable key-value caches is increased, and the range of application scenarios is expanded. The process of reusing the key-value cache is described below.

[0053] When the server 120 determines the reusable word segmentation, i.e., the prefix word segmentation group, the key-value cache of the prefix word segmentation group is obtained from the graphics processor. If the acquisition fails, the key-value cache corresponding to the prefix word segmentation group is obtained from the central processor.

[0054] Finally, the server 120 sends the key-value cache of the prefix word segmentation group and the remaining word segmentation group to the inference engine 130, so that the inference engine 130 performs inference according to the key-value cache of the prefix word segmentation group and the remaining word segmentation group to obtain the response content for the i-th round of prompt words. The inference engine 130 sends the response content to the terminal device 110 through the server 120, so that the user obtains the response content for the content input by the i-th word. Thus, the inference engine does not need to recalculate the key-value cache of the prefix word segmentation group in the calculation process, but directly reuses the key-value cache of the prefix word segmentation group, and only calculates the key-value cache of the remaining word segmentation group, thereby improving the inference speed.

[0055] The key-value cache reuse method provided in the present application can be applied to various fields, including but not limited to cloud technology, artificial intelligence, intelligent transportation, assisted driving, Internet of Things, financial services, and medical services. The following takes three scenarios as examples.

[0056] Scenario one, multi-round dialogue request scenario.

[0057] The multi-round dialogue request scenario refers to that the user continuously writes in query statements, and generates the corresponding response content based on each round of query statements. A query statement and its corresponding response content are referred to as a round of dialogue. In the multi-round dialogue process, the key-value cache generated by the query statement of the previous round of dialogue can be reused to avoid repeated calculation and improve the inference speed.

[0058] Referring to FIG. 2, which is a schematic diagram of a key-value cache reuse method provided in an embodiment of the present application in a multi-round dialogue request scenario.

[0059] Taking the i-th round of dialogue as an example, the terminal device obtains the query statement of the i-th round of dialogue and sends it to the server. The server constructs the i-th round of prompt word based on the query statement of the i-th round of dialogue. For example, the query statement of the 1st round of dialogue, the response content of the 1st round of dialogue and the query statement of the 2nd round of dialogue are spliced to obtain the 2nd round of prompt word. The plurality of words included in the i-th round of prompt word are matched with the historical word group generated based on the previous i-1 rounds of prompt word to obtain the reusable word (prefix word group) and the non-reusable word (remaining word group) in the i-th round of prompt word. Then, the server obtains the key-value cache of the prefix word group from the graphics processor, and if the obtaining fails, obtains the key-value cache corresponding to the prefix word group from the central processor. Finally, the key-value cache of the prefix word group and the remaining word group are sent to the inference engine, so that the inference engine performs inference according to the key-value cache of the prefix word group and the remaining word group to obtain the response content for the i-th round of prompt word, that is, the reply statement based on the query statement of the i-th round of dialogue.

[0060] Scenario two, code completion scenario.

[0061] During the process of inputting characters, the cursor is always located behind the last input character, that is, the cursor is constantly changing. During the process of constantly changing the cursor, the inference engine predicts the code after the cursor based on the code before the cursor, thereby realizing the code completion function. During the inference process, the key-value cache generated by the already predicted character can be reused to avoid repeated calculation and improve the inference speed.

[0062] Referring to FIG. 3, which is a schematic diagram of the key-value cache reuse method provided by the embodiment of the present application in the code completion scenario.

[0063] Taking the i-th prediction as an example, the terminal device obtains the code before the i-th stop position of the cursor and sends it to the server. The server generates the i-th round of prompt word according to the code, matches the plurality of words included in the i-th round of prompt word with the historical word group generated based on the previous i-1 rounds of prompt word to obtain the reusable word (prefix word group) and the non-reusable word (remaining word group) in the i-th round of prompt word. Then, the server obtains the key-value cache of the prefix word group from the graphics processor, and if the obtaining fails, obtains the key-value cache corresponding to the prefix word group from the central processor. Finally, the server sends the key-value cache of the prefix word group and the remaining word group to the inference engine, so that the inference engine performs inference according to the key-value cache of the prefix word group and the remaining word group to obtain the response content for the i-th round of prompt word, that is, the subsequent code predicted for the input code before the cursor.

[0064] Scenario three, long text inquiry scenario.

[0065] The user can input a long text as a query segment, and the reasoning engine generates corresponding response content based on the query segment. However, although the query segment includes contextual semantics, the entire query segment does not need to be learned in one reasoning process, and enough semantics can be learned. That is, the query segment can be divided into multiple segments based on a sliding window, thereby reducing the text length of the query segment and improving the reasoning speed. There are repeated word segmentation between the sliding windows of adjacent reasoning processes, so the key-value cache that has been calculated can be reused to avoid repeated calculation and improve the reasoning speed.

[0066] Referring to FIG. 4, it is a schematic diagram of the reuse method of the key-value cache provided by the embodiment of the application in a long text query scenario.

[0067] Taking the i-th reasoning as an example, the terminal device obtains the long text content input by the user and sends it to the server. The server constructs the text included in the sliding window corresponding to the i-th reasoning, i.e., the i-th round of prompt words, according to the text, matches the multiple word segments included in the i-th round of prompt words with the historical word segment groups generated based on the previous i-1 rounds of prompt words, and obtains the reusable word segments (prefix word segment group) and the non-reusable word segments (remaining word segment group) in the i-th round of prompt words. Then, the server obtains the key-value cache of the prefix word segment group from the graphics processing unit, and if the obtaining fails, obtains the key-value cache corresponding to the prefix word segment group from the central processing unit. Finally, the server sends the key-value cache of the prefix word segment group and the remaining word segment group to the reasoning engine, so that the reasoning engine performs reasoning according to the key-value cache of the prefix word segment group and the remaining word segment group, and obtains the response content for the i-th round of prompt words, i.e., the reply content for the i-th reasoning. When all the reply contents are obtained, the server sends the reply content for the long text to the terminal device.

[0068] It should be noted that the above application scenarios are only examples, and the reuse method of the key-value cache provided by the embodiment of the application can also be applied to other scenarios, which are not limited herein.

[0069] The reuse method of the key-value cache provided by the embodiment of the application can be executed by the server. However, in other embodiments of the application, the terminal device can also have similar functions as the server, so as to execute the reuse method of the key-value cache provided by the embodiment of the application, or the terminal device and the server jointly execute the reuse method of the key-value cache provided by the embodiment of the application, which is not limited by the embodiment of the application.

[0070] The reuse method of the key-value cache provided by the embodiment of the application will be described in detail through the method embodiment.

[0071] Referring to FIG. 5, which is a flowchart of a key-value cache multiplexing method provided by an embodiment of the present application. For ease of description, the following embodiments are still described by taking a server as an example of an execution subject of the key-value cache multiplexing method. As shown in FIG. 5, the key-value cache multiplexing method includes S501-S504.

[0072] S501: Obtain an i th round of prompt words.

[0073] i is an integer greater than 1, and multiple-round dialogue request scenarios, code completion scenarios, long text inquiry scenarios, and the like all need to obtain multiple rounds of prompt words, so as to generate corresponding response content based on each round of prompt words, thereby realizing multiple interactions with the user.

[0074] There is generally an association relationship between multiple rounds of prompt words. Taking a multiple-round dialogue request scenario as an example, the query statement of the 1st round of dialogue is “today's weather is how”, and the reply statement of the 1st round of dialogue is “today is 31℃, cloudy and sunny”. The query statement of the 2nd round of dialogue is “what should I wear”, and actually in the 2nd round of dialogue, the user really wants to ask is “today is 31℃, cloudy and sunny, what should I wear”. Based on this, in order to better understand the user's semantics, the 2nd round of prompt words will be generated based on the historical dialogue and the query statement of the 2nd round, such as “today's weather is how; today is 31℃, cloudy and sunny; what should I wear”. The first round of prompt words has no historical dialogue and is generally itself, such as “today's weather is how”. As can be seen, there is generally an association relationship such as semantic association and repeated words between multiple rounds of prompt words.

[0075] In related technologies, the key (K) value (V) cache involved in the same round of prompt words can be multiplexed. Taking a large language model (LLM) as an example. A round of prompt words will output one response content, and multiple inferences will be performed in the process of generating the response content. One inference only generates one token, and the output token is spliced with the input multiple tokens to serve as the input of the next inference. Each token needs to rely on the previously generated token, and the inference is ended until the condition is met, and the response content for the round of prompt words is obtained. Taking the prompt word “today's weather is how” as an example, the 1st round of inference will generate the token “today” for “today's weather is how”, the 2nd round of inference will generate the token “weather” for “today's weather is how; today”, and “today” is obtained. By analogy, until “today is 31℃, cloudy and sunny” is obtained.

[0076] However, as the reasoning proceeds, the number of input word pieces will increase, resulting in an increase in reasoning computation, which will seriously affect the reasoning speed. Meanwhile, in the reasoning process, the calculation of the i-th attention will repeatedly calculate the KV values of the first i-1 word pieces, so as to store the KV values in the form of KV cache in the graphic processing unit (GPU), and directly read the KV cache in the next round of reasoning calculation, so as to improve the reasoning speed.

[0077] As can be seen from the foregoing, there may be an association relationship between the continuous multiple rounds of prompt words. In order to further improve the reasoning speed, the key value cache that is only reused within one round of prompt words can be changed to the key value cache that is reused between multiple rounds of prompt words, that is, instead of releasing the key value cache generated in this reasoning from the graphic processing unit after each reasoning, as much as possible, the key value cache generated in multiple rounds of prompt words is stored in the range of the graphic processing unit memory, so as to be directly used subsequently.

[0078] Further, in the embodiment of the present application, if the graphic processing unit memory cannot store the key value cache any more, the key value cache is no longer directly released, but a part of the key value cache is stored in the central processing unit (CPU), so that the key value cache is stored by the graphic processing unit and the central processing unit together, so as to further improve the number of reusable key value caches. The storage process will be described later, and will not be described here.

[0079] S502: According to the multiple word pieces included in the i-th round of prompt words and the historical word piece group, the word pieces with the same word piece arrangement order are matched according to the word piece arrangement order, to obtain a prefix word piece group and a remaining word piece group.

[0080] The i-th round of prompt words includes multiple word pieces, and the multiple word pieces are arranged according to the word piece arrangement order. Taking the second round of prompt words as an example, "today, how is the weather today; today, 31 degrees, cloudy and sunny, what should I wear", which includes multiple word pieces arranged according to the word piece arrangement order, can be "today", "weather", "how", "today", "31 degrees", "cloudy and sunny", "I", "wear", "compare", "suitable".

[0081] The historical word piece group also includes multiple historical word pieces, and the historical word pieces are obtained based on the first i-1 rounds of prompt words, and the multiple historical word pieces have an arrangement order. Taking the first round of prompt words as an example, "today, how is the weather today", which includes multiple historical word pieces arranged according to the word piece arrangement order, can be "today", "weather", "how". In addition, the first i-1 rounds of prompt words have an association relationship with the i-th round of prompt words, such as i rounds of query statements from the same user.

[0082] It should be noted that the segmentation arrangement order of the plurality of segments in the historical segmentation group is consistent with the segmentation arrangement order of the plurality of segments included in the i-th round prompt word, such as being determined based on time sequence, being determined based on semantic sequence, etc. The segmentation arrangement order determined based on this kind of manner is generally consistent with the order of generating segments in the reasoning process of the subsequent reasoning engine, thereby ensuring the feasibility of the key-value cache reuse of the subsequent segmentation.

[0083] The corresponding matching between the segments is implemented based on the segmentation arrangement order, that is, the segments with the same segmentation arrangement order are matched according to the segmentation arrangement order, to obtain a prefix segmentation group and a remaining segmentation group. The prefix segmentation group includes the segments in the i-th round prompt word that are matched successfully when the first matching fails, that is, the reusable segments in the i-th round prompt word, so as to directly use the key-value cache corresponding to the segments in the subsequent process. The remaining segmentation group includes the segments in the i-th round prompt word other than the prefix segmentation group, that is, the segments that cannot be reused.

[0084] For example, the segments included in the i-th round prompt word are ABCDF, and the segments included in the historical segmentation group are ABEDG. The prefix segmentation group is AB, but not AB and D, because in the process of matching according to the segmentation arrangement order, C and E will fail to match, and F and G are the second time of matching failure. The remaining segmentation group is CDF. It should be noted that even if D is matched successfully, in the subsequent reasoning process, due to the difference of C and E, even if D is the same, it will also lead to different processes of twice reasoning calculation, so as to cause that even if D is the same, the key-value cache of D cannot be reused, that is, in the subsequent reasoning process, only the segments that have been matched successfully when the first matching fails, that is, the prefix segmentation group, can be reused.

[0085] As a possible implementation manner, the k-th segment in the plurality of segments included in the i-th round prompt word is matched with the k-th segment in the historical segmentation group, k is an integer starting from 1; if the matching is successful, k+1 is updated to k, and the matching is continued; if the matching fails, the segments in the i-th round prompt word that are matched successfully are taken as the prefix segmentation group, and the segments in the plurality of segments included in the i-th round prompt word other than the segments included in the prefix segmentation group are taken as the remaining segmentation group.

[0086] Continue with the foregoing example, k=1, the 1st segment in the i-th round prompt word is matched with the 1st historical segment in the historical segmentation group, after the matching is successful, k=2, the 2nd segment in the i-th round prompt word is matched with the 2nd historical segment in the historical segmentation group, after the matching is successful, k=3, the 3rd segment in the i-th round prompt word is matched with the 3rd historical segment in the historical segmentation group, after the matching fails, the prefix segmentation group “today” “weather” “how” is obtained, and the remaining segmentation group “today” “31℃” “cloudy to sunny” “I” “wear what” “compare” “suitable” is obtained.

[0087] Thus, by one-to-one corresponding matching of the i-th round of prompt words and the segmented words in the historical segmented word group according to the segmented word arrangement order, the prefix segmented word group and the remaining segmented word group are obtained. Moreover, the matching is ended after the first matching fails, avoiding the occupation of computing resources by subsequent useless matching, improving the video memory space in the graphics processor, further increasing the number of key-value caches, and improving the inference speed.

[0088] S503: If the key-value cache corresponding to the prefix segmented word group is not obtained from the graphics processor, the key-value cache corresponding to the prefix segmented word group is obtained from the central processor.

[0089] As known from the foregoing, in the embodiments of the present application, the graphics processor and the central processor jointly store the key-value cache to further increase the number of reusable key-value caches. Moreover, compared with other storage spaces, the interaction speed between the central processor and the graphics processor is faster to ensure the inference speed. Thus, while ensuring the inference speed, the number of reusable key-value caches is increased, and the range of application scenarios is expanded.

[0090] Therefore, after determining the prefix segmented word group, the key-value cache corresponding to the prefix segmented word group is first obtained from the graphics processor. If the obtaining is successful, S504 is executed. If the obtaining fails, since there is still a part of the key-value cache stored in the central processor, the key-value cache corresponding to the prefix segmented word group can be obtained from the central processor.

[0091] In addition, if the key-value cache corresponding to the prefix segmented word group is not obtained from the graphics processor, and the key-value cache corresponding to the prefix segmented word group is not obtained from the central processor, it indicates that neither the graphics processor nor the central processor stores the key-value cache corresponding to the prefix segmented word group. At this time, the prefix segmented word group and the remaining segmented word group can be sent to the inference engine, that is, the i-th round of prompt words is sent to the inference engine, so that the inference engine re-performs inference calculation, that is, the inference engine obtains the response content for the i-th round of prompt words according to the prefix segmented word group and the remaining segmented word group. Thus, even if the central processor and the graphics processor do not store the key-value cache corresponding to the prefix segmented word group, the normal inference calculation for the i-th round of prompt words can be ensured, and the user experience is improved.

[0092] S504: The key-value cache of the prefix segmented word group and the remaining segmented word group are sent to the inference engine, so that the inference engine obtains the response content for the i-th round of prompt words according to the key-value cache of the prefix segmented word group and the remaining segmented word group.

[0093] Generally, the inference engine is internally installed with a model providing corresponding services, such as a large language model based on a generative pre-trained transformer (GPT) and the like, which can provide services such as conversation services, code completion services, and the like.

[0094] In the related art, the key-value cache generated in each inference process is released immediately after the inference ends. Therefore, the i-th round of prompt words is directly sent to the inference engine, so that the inference engine re-performs inference calculation based on the multiple tokens included in the i-th round of prompt words to obtain response content for the i-th round of prompt words. It can be understood that the inference engine cooperates with the graphics processor to improve the inference speed. Specifically, the inference engine distributes multiple inferences for the same round of prompt words to the graphics processor to perform inference calculation through the graphics processor to obtain tokens obtained by the inference calculation, and store the final result after the inference calculation process based on the video memory of the graphics processor. The graphics processor sends the calculated tokens to the inference engine.

[0095] In the embodiments of the present application, the key-value cache generated in each inference process is not released after the inference ends, but can be directly reused for the key-value cache generated by the previous i-1 rounds of prompt words, i.e., the key-value cache of the prefix token group. The key-value cache of the prefix token group and the remaining token group are sent to the inference engine, so that the inference engine does not need to perform inference calculation again, but directly reuses the key-value cache of the prefix token group and only performs inference calculation on the remaining token group that cannot be reused, thereby improving the inference speed.

[0096] As can be seen from the above technical solution, if the video memory of the graphics processor cannot store the key-value cache, the key-value cache is no longer directly released, but a part of the key-value cache is stored in the central processor, so that the key-value cache is stored by the graphics processor and the central processor together, to further improve the number of reusable key-value caches. Moreover, compared with other storage spaces, the central processor and the graphics processor have faster interaction speed to ensure the inference speed. Therefore, while ensuring the inference speed, the number of reusable key-value caches is improved and the range of application scenarios is expanded. The process of reusing the key-value cache is described below.

[0097] When the i-th round of the prompt word is obtained, according to the plurality of words included in the i-th round of the prompt word, the history word group generated by the previous i-1 rounds of the prompt word is matched according to the word arrangement order, and the words with the same word arrangement order are matched to obtain the prefix word group and the remaining word group, that is, the reusable words and the non-reusable words in the i-th round of the prompt word are determined by matching. Then, the key value cache of the prefix word group is obtained from the graphics processor, if the key value cache of the prefix word group fails to be obtained, the key value cache of the prefix word group may be moved to the central processor for storage, and then the key value cache corresponding to the prefix word group is obtained from the central processor. Finally, the key value cache of the prefix word group and the remaining word group are sent to the inference engine, so that the inference engine performs inference according to the key value cache of the prefix word group and the remaining word group to obtain the response content for the i-th round of the prompt word. Therefore, the inference engine does not need to recalculate the key value cache of the prefix word group in the calculation process, but directly reuses the key value cache of the prefix word group, and only calculates the key value cache of the remaining word group, thereby improving the inference speed.

[0098] The embodiments of the present application do not specifically limit the way in which the graphics processor and the central processor cooperate to store the key value cache, which will be described below by way of example in two ways.

[0099] The first way is to execute the dynamic unloading strategy in advance.

[0100] A1: Obtain the key value cache generated by the i-1th round of the prompt word.

[0101] The key value cache generated by the i-1th round of the prompt word refers to the key value cache generated in the inference calculation process based on the plurality of words included in the i-1th round of the prompt word.

[0102] A2: Store the key value cache generated by the i-1th round of the prompt word to the graphics processor.

[0103] Since the key value cache generated by the i-2th round of the prompt word is stored, sufficient storage space is reserved in the graphics processor, so that the key value cache generated by the i-1th round of the prompt word can be directly stored to the graphics processor.

[0104] It should be noted that steps A1 and A2 are not necessary steps for executing the dynamic unloading strategy in advance, and the dynamic unloading strategy can be executed from step A3 at any time before the i-th round of the prompt word is obtained.

[0105] A3: Obtain the remaining video memory space of the graphics processor.

[0106] The following will be described by way of example after the key value cache generated by the i-1th round of the prompt word is stored, and sufficient storage space is reserved in the graphics processor.

[0107] A portion of the graphics processor's storage space is typically used for inference calculations, while the remaining storage space, i.e., the remaining video memory, is used to store key-value caches. Therefore, it is necessary to obtain the remaining storage space of the graphics processor after storing the key-value cache generated by the (i-1)th round of prompts, i.e., to obtain the remaining video memory space of the graphics processor.

[0108] This application does not specifically limit the method for determining the remaining video memory space. For example, the remaining video memory space of the graphics processor can be determined based on the video memory space occupied by the key-value cache already stored in the graphics processor, the video memory space required by the graphics processor to perform inference calculations, and the total video memory space of the graphics processor. Specifically, the video memory space occupied by the stored key-value cache can be determined based on the model deployed by the inference engine, which determines the hidden dimension of the attention layer of the model, the number of attention layers of the decoder, and the number of attention heads of the attention layer, thereby determining the video memory space required for one word segmentation. Then, the video memory space occupied by the stored key-value cache can be determined based on the number of stored words and the video memory space required for one word segmentation.

[0109] A4: If the remaining video memory space of the graphics processor is less than the preset storage space, then determine the first target key value cache from the key value cache stored in the graphics processor, store the first target key value cache in the central processing unit, and delete the first target key value cache from the graphics processor.

[0110] If the remaining video memory space of the graphics processor is less than the preset storage space, it means that the remaining video memory space of the graphics processor is insufficient to directly store the key value cache generated by the i-th round of prompt words. Therefore, it is necessary to release a part of the key value cache from the graphics processor, namely the first target key value cache, store the first target key value cache in the central processing unit, and delete the first target key value cache from the graphics processor. The first target key value cache and the remaining video memory space are greater than or equal to the preset storage space.

[0111] The embodiments of this application do not limit the size of the preset storage space. Those skilled in the art can set it according to actual needs to reserve a cache that can directly store the key value generated by the i-th round of prompts, and ensure that the graphics processor has reorganized video memory to support inference calculations.

[0112] This application does not specifically limit the method for determining the first target key-value cache. For example, the key-value cache with the longest storage time can be determined from the multiple key-value caches included in the graphics processor and used as the first target key-value cache. Alternatively, key-value caches with a storage time exceeding a preset time threshold can be determined from the multiple key-value caches included in the graphics processor. These key-value caches are less likely to be reused, so they can be used as the first target key-value cache and moved to the central processing unit for storage.

[0113] A5: If the remaining memory space of the graphics processor is greater than or equal to the predicted storage space, the key-value cache does not need to be released from the graphics processor.

[0114] If the remaining memory space of the graphics processor is greater than or equal to the predicted storage space, it means that the remaining memory space of the graphics processor can support the direct storage of the next round of prompt words, that is, the key-value cache generated by the i-th round of prompt words, so that a part of the key-value cache does not need to be released from the graphics processor.

[0115] Thus, each time the graphics processor stores the key-value cache, the remaining memory space of the graphics processor is detected. If the remaining memory space is greater than or equal to the preset storage space, that is, the graphics processor can still support the storage of the key-value cache corresponding to the next round of prompt words, no further processing is needed. If the remaining memory space is less than the predicted storage space, that is, the graphics processor cannot support the storage of the key-value cache corresponding to the next round of prompt words, a part of the key-value cache in the graphics processor needs to be moved to the central processor for storage, so as to reserve sufficient memory in the graphics processor for storing the key-value cache corresponding to the next round of prompt words. Thus, the function of dynamically offloading the key-value cache according to the remaining memory space of the graphics processor is realized, and only the remaining memory space of the graphics processor needs to be detected after the key-value cache generated by one round of prompt words is stored, without the need for real-time detection of the remaining memory space of the graphics processor, thereby reducing the detection frequency. In addition, by detecting the remaining memory space of the graphics processor and then determining the key-value cache to be moved to the central processor, unnecessary exchange of the key-value cache between the central processor and the graphics processor can be avoided.

[0116] Method two: Real-time execution of the dynamic offloading strategy.

[0117] B1: Obtain the remaining memory space of the graphics processor and the to-be-stored key-value cache generated by the i-th round of prompt words.

[0118] The to-be-stored key-value cache is determined according to the key-value cache of one segmented word and the number of segmented words to be stored in the remaining segmented word group, that is, the key-value cache of one segmented word is multiplied by the number of segmented words to be stored in the remaining segmented word group to obtain the to-be-stored key-value cache.

[0119] It should be noted that the number of segmented words to be stored in the remaining segmented word group can be the number of all segmented words included in the remaining segmented word group, that is, the to-be-stored key-value cache generated by the i-th round of prompt words is stored at one time. The number of segmented words to be stored in the remaining segmented word group can be the number of corresponding segmented words stored at one time in the remaining segmented word group, that is, the to-be-stored key-value cache generated by the i-th round of prompt words is stored in multiple times, which is not limited in the present application and can be set according to actual needs by those skilled in the art.

[0120] B2: If the remaining video memory space is less than the key value cache to be stored, then determine the second target key value cache from the key value cache stored in the graphics processor, store the second target key value cache in the central processing unit, delete the second target key value cache from the graphics processor, and store the key value cache to be stored in the graphics processor.

[0121] If the remaining video memory space is less than the key value cache to be stored, it means that the current remaining storage space of the graphics processor is insufficient to support the key value cache to be stored generated by the i-th round of prompts. At this time, it is necessary to move a part of the key value cache in the graphics processor to the central processing unit for storage, so as to expand the storage space of the graphics processor and reuse this part of the key value cache in the future.

[0122] This portion of the key-value cache is the second target key-value cache. The second target key-value cache is stored in the central processing unit (CPU) and then deleted from the graphics processing unit (GPU). At this point, the sum of the second target key-value cache released from the GPU and the remaining video memory space of the GPU is greater than or equal to the key-value cache to be stored, thus the key-value cache to be stored is stored in the GPU.

[0123] This application does not specifically limit the method for determining the second target key-value cache. For example, the key-value cache with the longest storage time can be determined from the multiple key-value caches included in the graphics processor and used as the second target key-value cache. Alternatively, key-value caches with a storage time exceeding a preset time threshold can be determined from the multiple key-value caches included in the graphics processor. These key-value caches are less likely to be reused, so they can be used as the second target key-value cache and moved to the central processing unit for storage.

[0124] B3: If the remaining video memory space is greater than or equal to the key-value cache to be stored, then the key-value cache to be stored will be stored in the graphics processor.

[0125] If the remaining video memory space is greater than or equal to the key value cache to be stored, it means that the current storage space of the graphics processor is sufficient to support the storage of the key value cache generated by the i-th round of prompts. Therefore, there is no need to move part of the key value cache to the central processing unit for storage. Instead, the key value cache to be stored is directly stored in the graphics processor.

[0126] Thus, each time before the graphic processor is about to store the to-be-stored key-value cache, the remaining memory space of the graphic processor is detected. If the remaining memory space is greater than or equal to the to-be-stored key-value cache, that is, the graphic processor can still support storage of the to-be-stored key-value cache, the to-be-stored key-value cache is directly stored in the graphic processor. If the remaining memory space is less than the to-be-stored key-value cache, that is, the graphic processor cannot support storage of the to-be-stored key-value cache, a part of the key-value cache in the graphic processor needs to be moved to the central processor for storage, so that the to-be-stored key-value cache can be stored in the graphic processor. Thus, the function of dynamically unloading the key-value cache according to the remaining memory space of the graphic processor is realized, and the remaining memory space is detected each time of storage, so as to fully utilize the storage space of the graphic processor and further increase the storage amount of the key-value cache of the word segmentation. In addition, by detecting the remaining memory space of the graphic processor and then determining the key-value cache to be moved to the central processor, unnecessary exchange of the key-value cache between the central processor and the graphic processor can be avoided.

[0127] As a possible implementation manner, when the remaining memory space of the graphic processor is insufficient, a part of the key-value cache in the graphic processor can be released to the central processor. With the increase of time, the central processor will also be unable to continue to store the key-value cache from the graphic processor.

[0128] Based on this, the embodiment of the present application provides a kind of based on key-value cache's storage time determination third target key-value cache, and third target key-value cache is released from central processor. Specific description is made below.

[0129] C1: obtain the remaining memory space of the central processor.

[0130] The remaining memory space refers to the remaining storage space of the central processor.

[0131] C2: if the remaining memory space is less than the second target key-value cache, determine the third target key-value cache from the central processor, delete the third target key-value cache from the central processor, and store the second target key-value cache to the central processor.

[0132] The storage time of the third target key-value cache is greater than the preset time threshold, that is, the storage time of the third target key-value cache is longer, and the possibility of its reuse is lower, so it can be released as the key-value cache to expand the storage space of the central processor. For example, the third target key-value cache can be the key-value cache with the longest storage time in the central processor. For another example, the storage time of the third target key-value cache exceeds any key-value cache of the preset time threshold, which is not limited by the present application.

[0133] The preset time threshold is not limited by the embodiment of the present application, and can be limited by the person skilled in the art according to actual needs.

[0134] It should be noted that the sum of the third target key-value cache and the remaining memory space is greater than or equal to the second target key-value cache, so that after the third target key-value cache is released from the central processor, the storage space of the central processor can store the second target key-value cache.

[0135] C3: If the remaining memory space is greater than or equal to the second target key-value cache, store the second target key-value cache to the central processor.

[0136] If the remaining memory space is greater than or equal to the second target key-value cache, it means that the remaining memory space of the central processor is sufficient to store the second target key-value cache, so there is no need to release a part of the key-value cache from the central processor, and the second target key-value cache can be stored in the central processor, thereby avoiding unnecessary key-value cache interaction between the central processor and the graphics processor.

[0137] Therefore, after the graphics processor determines the second target key-value cache to be stored in the central processor, if the remaining memory space of the central processor is sufficient to store the second target key-value cache, the second target key-value cache is stored, and if the remaining memory space of the central processor is insufficient to store the second target key-value cache, a part of the key-value cache with a longer storage time is released from the central processor to ensure that the second target key-value cache is stored in the central processor, thereby ensuring the normal function of the central processor while avoiding unnecessary key-value cache interaction between the central processor and the graphics processor.

[0138] In related technologies, in order to improve the speed of inference calculation, multiple inference engines are used to complete the corresponding services, but the multi-turn interaction with the association relationship, such as the multi-turn interaction from the same user, has a large amount of redundant prefix context, so when the multi-turn interaction with the association relationship is routed to the same inference engine, the reuse of the key-value cache can be realized.

[0139] Based on this, the multi-turn prompt words with the association relationship can be routed to the same inference engine through the session management mechanism to realize the reuse of the key-value cache. For details, see D1-D5.

[0140] D1: Obtain an initial prompt word including a session identifier.

[0141] The session identifier is used to identify the association relationship, that is, through the session identifier, the multi-turn prompt words with the association relationship can be determined, such as determining the multi-turn interaction from the same user based on the session identifier.

[0142] D2: If it is determined according to the session identifier that the initial prompt word has i-1 round prompt words with the association relationship, obtain the i round prompt word based on the initial prompt word.

[0143] If other prompt words having an association relationship with the initial prompt word can be determined according to the session identifier, it indicates that the initial prompt word is not the first appearing prompt word, and the initial prompt word can be reused in other prompt words appearing before. For example, the initial prompt word can be directly used as the i th round prompt word with other prompt words as the first i-1 round prompt words. For another example, in a multi-round dialogue request scenario, the initial prompt word can be a query statement input by the user, and the i th round prompt word can be obtained based on the i th round query statement, the first i-1 round query statements and the corresponding reply statements. For another example, in a code completion scenario, the initial prompt word can be the code before the cursor, and the i th round prompt word can be generated based on the code before the cursor.

[0144] D3: Perform the foregoing S502-S503, and send the key-value cache of the prefix token group and the remaining token group to a target inference engine in the plurality of inference engines.

[0145] After obtaining the i th round prompt word, the foregoing S502-S503 are performed to obtain the key-value cache of the prefix token group and the remaining token group. The key-value cache of the prefix token group and the remaining token group are sent to a target inference engine in the plurality of inference engines.

[0146] Among them, the multi-round prompt words of the same session identifier are executed by the same inference engine, so as to route the multi-round prompt words having an association relationship to the same inference engine, to realize reuse of the key-value cache.

[0147] The target inference engine is an inference engine in the plurality of inference engines for processing the multi-round prompt words having an association relationship with the i th round prompt word, that is, the multi-round prompt words having an association relationship are distributed to the same inference engine based on the session identifier, so that the same inference engine can better reuse the key-value cache of the prefix token group for the multi-round prompt words having an association relationship.

[0148] Further:

[0149] D4: If it is determined according to the session identifier that the initial prompt word does not have prompt words having an association relationship, the first round prompt word is obtained based on the initial prompt word.

[0150] If it is determined according to the session identifier that the initial prompt word does not have other prompt words (such as the first i-1 round prompt words) having an association relationship, it indicates that the initial prompt word is the first obtained prompt word, and the initial prompt word cannot be reused in other prompt words. The first round prompt word is obtained based on the initial prompt word.

[0151] D5: For the first round prompt word, the plurality of tokens included in the first round prompt word are sent to any inference engine in the plurality of inference engines.

[0152] The first round of prompt words does not have a reusable prefix word segmentation group, so the multiple word segments included in the first round of prompt words can be directly sent to any one of the multiple inference engines, so that the multiple rounds of prompt words that have an association relationship with the first round of prompt words are all sent to the inference engine based on the session identifier for inference calculation.

[0153] Therefore, the multiple rounds of dialogues that have an association relationship are sent to the same inference engine for inference calculation, such as routing multiple rounds of dialogue requests from the same user to the same inference engine based on the session identifier, so that the key-value cache of the prefix word segmentation group is more conveniently reused.

[0154] As a possible implementation, the storage of data can be achieved by creating a triple. The triple is respectively a session identifier, a word segment generated in a session, and a key-value cache corresponding to each word segment. For example, for the first round of prompt words, the metadata group of the first round of prompt words, i.e., the triple, includes the session identifier carried by the initial prompt word for obtaining the first round of prompt words, multiple word segments generated by the first round of prompt words, and the key-value cache corresponding to each word segment.

[0155] Subsequently, the existing metadata is updated to obtain an updated metadata group. Taking the i-th round of prompt words as an example, the metadata group of the (i-1)-th round of prompt words is updated to obtain the metadata group of the i-th round of prompt words. The metadata group of the i-th round of prompt words includes a session identifier, multiple word segments generated by the i-th round of prompt words, and the key-value cache corresponding to each word segment. That is, the session identifier remains unchanged, and the key-value cache of the word segment included in the (i-1)-th round of prompt words is newly added based on the remaining word segment group included in the i-th round of prompt words.

[0156] Further, the time can also be added in the metadata group, such as the metadata group of the first round of prompt words including a session identifier, a time of obtaining the prompt word in this round, multiple word segments generated by the first round of prompt words, and the key-value cache corresponding to each word segment. For another example, the metadata group of the i-th round of prompt words includes a session identifier, a time of obtaining the prompt word in this round, multiple word segments generated by the i-th round of prompt words, and the key-value cache corresponding to each word segment. Thus, according to the time of obtaining the prompt word in each round, the storage time of the key-value cache is determined, so that the target key-value cache (such as the first target key-value cache, the second target key-value cache, etc.) can be determined based on the time stored in the metadata group in the subsequent process. It can be understood that in the process of updating the metadata group, the time of obtaining the prompt word in this round also needs to be updated, such as the time stored in the metadata group of the i-th round of prompt words is the time of obtaining the i-th round of prompt words, and the time stored in the metadata group of the (i-1)-th round of prompt words is the time of obtaining the (i-1)-th round of prompt words.

[0157] Thus, by constructing the metadata group and updating the metadata group, the prefix tokenization group and the key-value cache of the prefix tokenization group are quickly obtained based on the session identifier in the metadata group, and even the target key-value cache is determined based on the time of obtaining the prompt word in the current round in the metadata group, so as to realize the release of the target key-value cache which is less likely to be reused again.

[0158] As a possible implementation manner, if the (i-1)th round prompt word is modified before obtaining the ith round prompt word, such as deleting some round of dialog or deleting code content, so that the (i-1)th round prompt word is deleted, the stored key-value cache can be modified based on the key-value cache corresponding to the token included in the modified (i-1)th round prompt word, so that the stored key-value cache is consistent with the token included in the (i-1)th round prompt word, thereby reducing the storage space of the graphics processor or the central processing unit.

[0159] For convenience of description, the reuse of the key-value cache provided by the embodiments of the present application is described below by taking a multi-round dialog request scenario as an example, and specific reference is made to E1-E5.

[0160] Referring to FIG. 6, this figure is a schematic diagram of a multi-round dialog request scenario provided by an embodiment of the present application.

[0161] In FIG. 6, after obtaining the first round prompt word, the plurality of tokens included in the first round prompt word are sent to the inference engine, and the response content 1 is obtained by performing inference calculation based on the inference engine, that is, the response content obtained for the first round prompt word. The first round prompt word and the response content 1 constitute a round of dialog. Similarly, the response content 2 can be obtained based on the second round prompt word to obtain the second round of dialog. The response content i is obtained based on the ith round prompt word to obtain the ith round of dialog. The response content i is taken as an example for description below.

[0162] E1: obtaining the query statement of the ith round of dialog, the (i-1)th round prompt word, and the response content for the (i-1)th round prompt word.

[0163] The query statement of the ith round of dialog is the query statement input by the user for the ith time, and each round of dialog includes a query statement and a response content for replying to the query statement.

[0164] The (i-1)th round prompt word is a prompt word constructed for the query statement of the (i-1)th round of dialog, and is obtained by splicing the (i-2)th round prompt word, the response content for the (i-2)th round prompt word, and the query statement of the (i-1)th round of dialog.

[0165] The response content for the (i-1)th round prompt word is also a reply statement for the query statement of the (i-1)th round.

[0166] E2: concatenate the i-th round prompt word, the response content to the i-th round prompt word, and the query statement of the i-th round of dialogue to obtain the i-th round prompt word.

[0167] For example, the query statement of the first round of dialogue is represented as u1, the first round prompt word is represented as (u1), and the response content to the first round prompt word is represented as a1. The query statement of the second round of dialogue is represented as u2, the second round prompt word is represented as (u1, a1, u2), and the response content to the second round prompt word is represented as a2. By analogy, the query statement of the i-th round of dialogue is represented as ui, the i-th round prompt word is represented as (u1, a1, u2, …, ui), and the response content to the i-th round prompt word is represented as ai.

[0168] E3: According to the plurality of segmented words included in the i-th round prompt word and the historical segmented word group, the segmented words with the same segmented word arrangement order are matched according to the segmented word arrangement order, to obtain a prefix segmented word group and a remaining segmented word group.

[0169] If the user does not delete the dialogue in the multi-round dialogue, the statement formed by the prefix segmented word group is generally the i-1th round prompt word. If the user deletes a round of dialogue in the multi-round dialogue, taking the user deleting the jth round of dialogue as an example, j is less than or equal to i-1, and the jth round of dialogue includes the query statement of the jth round of dialogue and the response content to the jth round prompt word.

[0170] In response to receiving the deletion instruction for the jth round of dialogue in the first i-1 rounds of dialogue, the segmented words included in the jth round of dialogue are deleted from the i-1th round prompt word, and if the segmented words included in the jth round of dialogue are stored in the graphics processor, the key-value cache corresponding to the segmented words included in the jth round of dialogue is deleted from the graphics processor or the central processing unit; if the segmented words included in the jth round of dialogue are stored in the central processing unit, the key-value cache corresponding to the segmented words included in the jth round of dialogue is deleted from the central processing unit.

[0171] Among them, in response to, which is used to represent the condition or state on which the operation is dependent, when the dependent condition or state is met, the executed one or more operations can be real-time or have a set delay: in the absence of special instructions, there is no restriction on the execution order of the multiple operations executed.

[0172] For example, the user has two rounds of dialogue, the second round prompt word is represented as (u1, a1, u2), and the user actively deletes the second round of dialogue. At this time, the second round prompt word is updated to (u1, a1), and the key-value cache corresponding to the segmented words included in u2 and a2 can be deleted from the graphics processor or the central processing unit to further release the storage space of the graphics processor or the central processing unit. After obtaining the query statement of the new round of dialogue, the third round prompt word can be represented as (u1, a1, u3).

[0173] Thus, the deletion of the corresponding dialogue based on the deletion instruction is more in line with the current user's behavior pattern in the multi-round dialogue request scenario, thereby deleting the corresponding key-value cache from the graphics processor or central processor, i.e., updating the stored key-value cache according to the deletion instruction, to further release the storage space of the graphics processor or central processor, improve the reuse quantity of the key-value cache, improve the inference speed, and thus improve the user experience.

[0174] As a possible implementation, in the multi-round dialogue request scenario, if the deletion instruction is not obtained before the i-th round of prompt word is obtained, it indicates that the i-th round of prompt word and the (i-1)-th round of prompt word only differ in the response content to the (i-1)-th round of prompt word and the query content of the i-th round of dialogue. Based on this, it is not necessary to match according to the historical segmented word group, but directly take the segmented words included in the (i-1)-th round of prompt word and the segmented words included in the response content to the (i-1)-th round of prompt word as the prefix segmented word group, and then obtain the remaining segmented word group according to the prefix segmented word group and the i-th round of prompt word.

[0175] If the deletion instruction is obtained before the i-th round of prompt word is obtained, it indicates that the i-th round of prompt word and the (i-1)-th round of prompt word not only differ in the response content to the (i-1)-th round of prompt word and the query content of the i-th round of dialogue, but also differ in the segmented words related to the deleted dialogue. Based on this, the foregoing S502 can be performed, i.e., the step of performing matching on the segmented words with the same arrangement order according to the plurality of segmented words included in the i-th round of prompt word and the historical segmented word group, to obtain the prefix segmented word group and the remaining segmented word group.

[0176] Thus, if the deletion instruction is obtained before the i-th round of prompt word is obtained, after the i-th round of prompt word is obtained, matching is performed on the segmented words with the same arrangement order according to the plurality of segmented words included in the i-th round of prompt word and the historical segmented word group, to obtain the prefix segmented word group and the remaining segmented word group. If the deletion instruction is not obtained before the i-th round of prompt word is obtained, after the i-th round of prompt word is obtained, it is not necessary to perform matching, but directly take the segmented words included in the (i-1)-th round of prompt word and the segmented words included in the response content to the (i-1)-th round of prompt word as the prefix segmented word group, and then obtain the remaining segmented word group according to the prefix segmented word group and the i-th round of prompt word. Thus, the number of matching is reduced for the multi-round dialogue request scenario, the speed of determining the prefix segmented word group is improved, and thus the inference speed and the user experience are improved.

[0177] E4: If the key-value cache corresponding to the prefix segmented word group fails to be obtained from the graphics processor, the key-value cache corresponding to the prefix segmented word group is obtained from the central processor.

[0178] E5: The key-value cache of the prefix segmented word group and the remaining segmented word group are sent to the inference engine, so that the inference engine obtains the response content to the i-th round of prompt word according to the key-value cache of the prefix segmented word group and the remaining segmented word group.

[0179] Inference engines typically employ two key steps in their inference computation process: a prefill phase and a decoder phase. The prefill phase involves providing the model with contextual information or hints before it begins generating text. The decoder phase, on the other hand, generates text character by character based on the intermediate results from the prefill phase, continuing until a termination condition is met. Key-value caching is generally reused during the decoder phase. Large language models are primarily composed of multiple decoders, each including a forward propagation layer and a self-attention layer, with the self-attention layer being the core of the transformer architecture.

[0180] Assuming the i-th round prompt word is (u1, a1, u2, a2, ..., ui-1, ai-1, ui), taking the Very Large Language Model (vLLM) as the model used as the inference engine as an example, in related technologies, all the word segments included in the i-th round prompt word are calculated during the inference calculation process. However, in the embodiment of this application, a portion of the word segment key-value cache (i.e., the key-value cache of the prefix word segment group) can be reused, and only the newly appearing word segments (i.e., the remaining word segments) need to be calculated. Then the calculation of the self-attention layer can be expressed as formula (1):

[0181] Among them, O ui This represents the response to the i-th prompt; Q ui This represents the multiple words included in the remaining word group in the i-th round of prompts. Taking a remaining word group containing n words as an example, it can be represented as Q. ui =(q1,q2,…,qn);K u1,a1；...；ui The key in the key-value cache of the i-th round prompt word is the representation of each position in the i-th round prompt word, such as K = (k1, k2, ..., kn). In the embodiments of this application, the key can be obtained from the key-value cache of the prefix segment group and calculated only for the segments included in the remaining segment group. V represents the transpose of the representation of each position in the i-th round of prompts; u1,a1；...；ui This represents the value corresponding to the key in the key-value cache of the i-th round prompt word, that is, the weight corresponding to each position in the i-th round prompt word, which can be represented as V = (v1, v2, ..., vn). In this embodiment, this value can be obtained from the key-value cache of the prefix segmentation group and calculated only for the segments included in the remaining segmentation groups; M ui This represents the mask matrix, used to ensure that subsequent word segmentation can obtain information from previous word segmentation; d is K. u1,a1；...；ui The corresponding key dimension.

[0182] It can be understood that all the data (such as query statements) collected in the specific embodiments of the present application are collected with the consent and authorization of the object to which the data belongs (such as a user, an institution or an enterprise). When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0183] The key-value cache reuse method provided by the embodiments of the present application can mainly involve artificial intelligence technology.

[0184] In order to further understand the technical solutions provided by the embodiments of the present application, the execution subject of the key-value cache reuse method provided by the embodiments of the present application is taken as a server as an example, and the key-value cache reuse method is introduced in the whole.

[0185] Currently, a large number of large language models emerge, such as GPT, pre-training language model (Large Language Model Meta AI, Llama), etc., which make it possible to handle many natural language processing tasks that were previously difficult to handle. Therefore, LLM inference service plays a crucial role in a wide range of fields and becomes the main workload of cloud service. Although the large language model inference process does not require as many resources as the training process, it still needs an extended computing platform with accelerators such as graphics processors to provide efficient processing and fast response. The key-value cache reuse method provided by the embodiments of the present application can be used as an extension of the large language model and deployed in the inference service of the large language model to accelerate the multi-round interaction between the user and the large language model, thereby reducing the interaction delay with the user, improving the product experience of the user, and increasing the throughput of the graphics processor that can serve the request, reducing the cost of the graphics processor inference service. Application scenarios include but are not limited to multi-round dialogue request scenarios, code completion scenarios and long text inquiry scenarios.

[0186] The multi-round dialogue request scenario will be described first.

[0187] Referring to FIG. 7, which is a schematic diagram of an application scenario of a key-value cache reuse method provided by an embodiment of the present application. Referring to FIG. 8, which is a schematic diagram of an application scenario of a key-value cache reuse method provided by an embodiment of the present application.

[0188] The architecture of the key-value cache reuse method includes a key-value cache manager 701, a storage manager 702, an inference engine 703, a graphics processor 704 and a central processing unit 705. The key-value cache manager 701 and the storage manager 702 can be different functional units in the same entity hardware, as shown in FIG. 7, or different entity hardware, as shown in FIG. 8.

[0189] The key-value cache manager 701 is configured to manage the key-value caches from different session requests to realize reuse of the prefix token group key-value cache.

[0190] The storage manager 702 is configured to detect the remaining memory space of the graphics processor and the remaining memory space of the central processor, so that the graphics processor and the central processor have sufficient storage space to store the key-value cache.

[0191] The inference engine 703 is configured to generate corresponding response content for the session request.

[0192] The graphics processor 704 is configured to store the key-value cache and accelerate the inference calculation.

[0193] The central processor 705 is configured to store the key-value cache.

[0194] The reuse method of the key-value cache is described below.

[0195] S1: Obtain a query statement carrying a session identifier.

[0196] S2: If it is determined according to the session identifier that the query statement does not exist a plurality of rounds of prompt words having an association relationship, obtain a first round of prompt words based on the query statement.

[0197] If it is determined according to the session identifier that the query statement does not exist a plurality of rounds of prompt words having an association relationship, it is indicated that the query statement based on the session identifier appears for the first time, and a first round of prompt words is obtained based on the query statement, such as directly taking the query statement as the first round of prompt words.

[0198] S3: For the first round of prompt words, a plurality of tokens included in the first round of prompt words are sent to any inference engine of a plurality of inference engines.

[0199] The first round of prompt words does not exist a prefix token group that can be reused, and therefore the plurality of tokens included in the first round of prompt words can be directly sent to any inference engine of the plurality of inference engines, so that a plurality of rounds of prompt words having an association relationship with the first round of prompt words are all sent to the inference engine based on the session identifier for inference calculation.

[0200] S4: If it is determined according to the session identifier that the query statement exists a plurality of rounds of prompt words having an association relationship, obtain an i-th round of prompt words based on the query statement.

[0201] If it is determined according to the session identifier that the query statement exists a plurality of rounds of prompt words having an association relationship, it is indicated that the initial prompt word is a prompt word appearing for the i-th time based on the session identifier, and an i-th round of prompt words is obtained based on the query statement, such as splicing the (i-1)-th round of prompt words, response content for the (i-1)-th round of prompt words, and the query statement of the i-th round of dialogue to obtain the i-th round of prompt words.

[0202] S5: According to the plurality of segmented words included in the i th round of prompt words and the historical segmented word group, the segmented words with the same segmented word arrangement order are matched according to the segmented word arrangement order, to obtain a prefix segmented word group and a remaining segmented word group.

[0203] For example, the k th segmented word in the plurality of segmented words included in the i th round of prompt words is matched with the k th segmented word in the historical segmented word group, k being an integer starting from 1. If the matching is successful, k+1 is updated to k, and the step of matching the k th segmented word in the plurality of segmented words included in the i th round of prompt words with the k th segmented word in the historical segmented word group is performed again; if the matching fails, the segmented word in the i th round of prompt words that matches successfully is taken as the prefix segmented word group, and the segmented word in the plurality of segmented words included in the i th round of prompt words other than the segmented word included in the prefix segmented word group is taken as the remaining segmented word group.

[0204] S6: If the key-value cache corresponding to the prefix segmented word group is obtained from the graphics processor, the key-value cache of the prefix segmented word group and the remaining segmented word group are sent to the target inference engine, so that the target inference engine obtains the response content for the i th round of prompt words according to the key-value cache of the prefix segmented word group and the remaining segmented word group.

[0205] S7: If the key-value cache corresponding to the prefix segmented word group fails to be obtained from the graphics processor, the key-value cache corresponding to the prefix segmented word group is obtained from the central processor, and the key-value cache of the prefix segmented word group and the remaining segmented word group are sent to the target inference engine, so that the target inference engine obtains the response content for the i th round of prompt words according to the key-value cache of the prefix segmented word group and the remaining segmented word group.

[0206] S8: If the key-value caches corresponding to the plurality of prefix segmented words fail to be obtained from the central processor and the graphics processor, the prefix segmented word group and the remaining segmented word group are sent to the target inference engine, so that the target inference engine obtains the response content for the i th round of prompt words according to the prefix segmented word group and the remaining segmented word group.

[0207] Wherein, the plurality of rounds of prompt words with the same session identifier are executed by the same inference engine, and the target inference engine is an inference engine in the plurality of inference engines for processing the plurality of rounds of prompt words having an association relationship with the i th round of prompt words, that is, the plurality of rounds of prompt words having an association relationship are distributed into the same inference engine based on the session identifier, so that the same inference engine better reuses the key-value cache of the prefix segmented word group for the plurality of rounds of prompt words having an association relationship.

[0208] It should be noted that in the process of reasoning of the target inference engine, it will call the graphics processor to accelerate the reasoning calculation process, and the graphics processor is used to calculate the key-value cache of each segmented word and store the key-value cache in the graphics processor or the central processor, which will be described below.

[0209] S9: Obtain the remaining memory space of the graphics processor and the to-be-stored key-value cache generated by the i-th round of prompt words.

[0210] The to-be-stored key-value cache is determined according to the key-value cache of one segmented word and the number of segmented words to be stored in the remaining segmented word group. For example, the key-value cache of one segmented word is 20k, and one segmented word is stored each time, so the key-value cache of one segmented word is 20k each time, that is, the to-be-stored key-value cache is 20k.

[0211] S10: If the remaining memory space is less than the to-be-stored key-value cache, determine the second target key-value cache from the key-value cache stored in the graphics processor, store the second target key-value cache to the central processor, delete the second target key-value cache from the graphics processor, and store the to-be-stored key-value cache to the graphics processor.

[0212] The sum of the second target key-value cache and the remaining memory space is greater than or equal to the to-be-stored key-value cache, so as to reserve sufficient storage space in the graphics processor.

[0213] The process of storing the second target key-value cache to the central processor is as follows:

[0214] Obtain the remaining memory space of the central processor; if the remaining memory space is less than the second target key-value cache, determine the third target key-value cache from the central processor, delete the third target key-value cache from the central processor, and store the second target key-value cache to the central processor; if the remaining memory space is greater than or equal to the second target key-value cache, store the second target key-value cache to the central processor.

[0215] The sum of the third target key-value cache and the remaining memory space is greater than or equal to the second target key-value cache, and the storage time of the third target key-value cache is greater than a preset time threshold, so as to reserve sufficient space in the central processor to store the second target key-value cache, and the third target key-value cache released from the central processor is the key-value cache with a longer storage time.

[0216] S11: If the remaining memory space is greater than or equal to the to-be-stored key-value cache, store the to-be-stored key-value cache to the graphics processor.

[0217] In addition, when storing the key-value cache, the key-value cache can be stored based on a quadruple. The quadruple includes a session identifier, a time of obtaining the i-th round of session identifier, a plurality of segmented words generated by the i-th round of prompt words, and a key-value cache corresponding to each segmented word.

[0218] It should be noted that if the deletion instruction for the jth round of dialogue in the first i-1 rounds of dialogue is received before the ith round of prompt word is obtained, the jth round of dialogue included in the i-1th round of prompt word is deleted, and the key value cache corresponding to the jth round of dialogue included in the i-1th round of prompt word is deleted from the graphics processor or the central processing unit, the jth round of dialogue includes the query statement of the jth round of dialogue and the response content for the jth round of prompt word, and j is less than or equal to i-1.

[0219] In order to evaluate the effectiveness of the method provided by the embodiments of the present application, four existing methods are compared. The four existing methods are: (1) TensorRT-LLM (TensorRT-Large Language Model, TRTLLM) is an inference system of a large language model provided by NVIDIA; (2) vLLM is a high-throughput large language model inference system with efficient storage management paged attention; (3) Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills (Chunked Prefill) mixes the tokenization in the decoding stage and the tokenization in the prefilling stage to improve the utilization of the graphics processor; (4) Prefix Cache is an optional feature of vLLM, which can reuse the key value cache between requests through hash.

[0220] There are two test scenarios. In the multi-round dialogue request scenario, the model in the inference engine is ChatGLM3-6B (a dialogue pre-training model), and the prompt word is constructed from ShareGPT (a tool for sharing GPT dialogue). In the code completion scenario, the model in the inference engine is StarCoderBase-7B (a dialogue pre-training model), and the prompt word is constructed from the Stack of BigCode (a technology for processing large-scale and highly complex programming code). The comparison results are shown in Table 1.

[0221] Table 1

[0222] Table 1 shows the overall performance of the method provided in the embodiments of the present application and other four methods. As can be seen from Table 1, the method provided in the embodiments of the present application achieves 37% and 190% throughput improvement in the multi-turn dialogue request scenario and the code completion scenario, respectively, compared with the best method PrefixCache, which shows that the method provided in the embodiments of the present application has great advantages. The advantage of the method provided in the embodiments of the present application in the code completion scenario is much greater than that in the multi-turn dialogue, and the main reason is that code completion contains more repeated calculations and requires more storage to place the key-value cache.

[0223] In order to prove the effectiveness of the method provided in the embodiments of the present application, an ablation experiment is also performed, which ablates the method provided in the embodiments of the present application into two methods: (1) releasing the corresponding key-value cache immediately after the inference engine finishes each inference calculation, which can be regarded as removing the reuse of the key-value cache of the prefix segmentation group and the dynamic offloading strategy from the method provided in the embodiments of the present application, thereby becoming a vLLM method. (2) Reusing the key-value cache of the prefix segmentation group, and deleting a part of the key-value cache when the remaining memory space of the graphics processor is insufficient, which can be regarded as removing the dynamic offloading strategy from the method provided in the embodiments of the present application, and is denoted as a CrossKV method.

[0224] Referring to FIG. 9, which is a result diagram of end-to-end delay decomposition provided in the embodiments of the present application. In FIG. 9, the results of evaluating the end-to-end delay decomposition of the method provided in the embodiments of the present application, the vLLM method and the CrossKV method are shown, and it can be found that the method provided in the embodiments of the present application achieves 37% / 66% and 47% / 68% end-to-end delay reduction in the multi-turn dialogue request scenario and the code completion scenario, respectively, compared with the CrossKV method and the vLLM method, and the main delay reduction contribution comes from the decoding stage, which also proves the effectiveness of the method provided in the embodiments of the present application.

[0225] In order to prove that the method provided in the embodiments of the present application can be applied to various scenarios of multi-turn interaction, experiments are performed on the three factors of dialogue turn number, input length and output length, and the results are shown in FIG. 10.

[0226] Referring to FIG. 10, which is an experimental result diagram of a multi-turn interaction scenario provided in the embodiments of the present application. In FIG. 10, it can be found that the method provided in the embodiments of the present application exhibits the optimal performance under the three factors, and as the number of turns increases, the advantage of the method provided in the embodiments of the present application becomes more obvious, because a larger number of turns exists more reusable information. Secondly, as the input length or the output length increases, the advantage of the PrefixCache method compared with the vLLM method gradually disappears, because the storage of the graphics processor is very limited and cannot store enough key-value caches in the face of longer context length.

[0227] For the key-value cache reuse method described above, the application also provides a corresponding key-value cache reuse device to make the key-value cache reuse method described above applicable and implemented in practice.

[0228] Referring to FIG. 11, which is a structural schematic diagram of a key-value cache reuse device provided by an embodiment of the application. As shown in FIG. 11, the key-value cache reuse device 1100 comprises an acquisition unit 1101, a matching unit 1102, a reuse unit 1103 and a sending unit 1104.

[0229] The acquisition unit 1101 is configured to acquire an i th round of prompt words, i being an integer greater than 1.

[0230] The matching unit 1102 is configured to match, according to a plurality of segmented words included in the i th round of prompt words and a historical segmented word group, segmented words with the same segmented word arrangement order to obtain a prefix segmented word group and a remaining segmented word group, the historical segmented word group being segmented words with a segmented word arrangement order obtained based on the first i-1 rounds of prompt words, the first i-1 rounds of prompt words having a correlation with the i th round of prompt words, the prefix segmented word group including segmented words in the i th round of prompt words that match successfully for the first time, and the remaining segmented word group including segmented words in the i th round of prompt words other than the prefix segmented word group.

[0231] The reuse unit 1103 is configured to acquire, if the prefix segmented word group fails to obtain the corresponding key-value cache from the graphic processor, the prefix segmented word group from the central processor.

[0232] The sending unit 1104 is configured to send the key-value cache of the prefix segmented word group and the remaining segmented word group to an inference engine, so that the inference engine obtains response content for the i th round of prompt words according to the key-value cache of the prefix segmented word group and the remaining segmented word group.

[0233] As can be seen from the above technical solution, the device provided by the embodiment of the application comprises an acquisition unit, a matching unit, a reuse unit and a sending unit. If the video memory of the graphic processor can no longer store key-value caches, the key-value caches are no longer directly released, but a part of the key-value caches are stored in the central processor, so that the key-value caches are stored by the graphic processor and the central processor together, to further increase the number of reusable key-value caches. Moreover, compared with other storage spaces, the central processor and the graphic processor have a faster interaction speed to ensure the inference speed. Thus, while ensuring the inference speed, the number of reusable key-value caches is increased, and the range of application scenarios is expanded. The process of reusing key-value caches is described below.

[0234] The i-th round of prompt words is acquired by an acquisition unit. A matching unit matches the i-th round of prompt words according to the historical segmented word groups generated by the previous i-1 rounds of prompt words. The segmented words with the same arrangement order are matched according to the arrangement order of the segmented words, and the prefix segmented word group and the remaining segmented word group are obtained. That is, the reusable segmented words and the non-reusable segmented words in the i-th round of prompt words are determined by matching. Then, a reuse unit acquires the key-value cache of the prefix segmented word group from the graphics processor. If the acquisition fails, the key-value cache of the prefix segmented word group may be moved to the central processor for storage. The key-value cache corresponding to the prefix segmented word group is acquired from the central processor. Finally, a sending unit sends the key-value cache of the prefix segmented word group and the remaining segmented word group to the inference engine, so that the inference engine performs inference according to the key-value cache of the prefix segmented word group and the remaining segmented word group, and obtains the response content for the i-th round of prompt words. Thus, the inference engine does not need to recalculate the key-value cache of the prefix segmented word group in the calculation process, but directly reuses the key-value cache of the prefix segmented word group, and only calculates the key-value cache of the remaining segmented word group, thereby improving the inference speed.

[0235] As a possible implementation manner, the apparatus further includes a storage unit, configured to, before the acquisition of the i-th round of prompt words:

[0236] acquire the remaining memory space of the graphics processor;

[0237] if the remaining memory space of the graphics processor is less than the preset storage space, determine a first target key-value cache from the key-value caches stored in the graphics processor, store the first target key-value cache into the central processor, and delete the first target key-value cache from the graphics processor, wherein the sum of the first target key-value cache and the remaining memory space is greater than or equal to the preset storage space.

[0238] As a possible implementation manner, the acquisition unit 1101 is specifically configured to acquire the remaining memory space of the graphics processor and the to-be-stored key-value cache generated by the i-th round of prompt words, wherein the to-be-stored key-value cache is determined according to the key-value cache of one segmented word and the number of segmented words to be stored in the remaining segmented word group;

[0239] The apparatus further includes a storage unit, configured to:

[0240] if the remaining memory space is less than the to-be-stored key-value cache, determine a second target key-value cache from the key-value caches stored in the graphics processor, store the second target key-value cache into the central processor, delete the second target key-value cache from the graphics processor, and store the to-be-stored key-value cache into the graphics processor, wherein the sum of the second target key-value cache and the remaining memory space is greater than or equal to the to-be-stored key-value cache;

[0241] If the remaining memory space is greater than or equal to the to-be-stored key-value cache, the to-be-stored key-value cache is stored to the central processor.

[0242] As a possible implementation manner, the apparatus further includes a storage unit, configured to:

[0243] acquire a remaining memory space of the central processor;

[0244] If the remaining memory space is less than the second target key-value cache, a third target key-value cache is determined from the central processor, the third target key-value cache is deleted from the central processor, the second target key-value cache is stored to the central processor, a sum of the third target key-value cache and the remaining memory space is greater than or equal to the second target key-value cache, and a storage time of the third target key-value cache is greater than a preset time threshold.

[0245] If the remaining memory space is greater than or equal to the second target key-value cache, the second target key-value cache is stored to the central processor.

[0246] As a possible implementation manner, the sending unit 1101 is configured to, if the key-value cache corresponding to the prefix token group fails to be acquired from the central processor and the graphics processor, send the prefix token group and the remaining token group to the inference engine, so that the inference engine obtains response content for the i th round of prompt words according to the prefix token group and the remaining token group.

[0247] As a possible implementation manner, the acquiring unit 1101 is specifically configured to:

[0248] acquire an initial prompt word including a session identifier, the session identifier being used to identify the association relationship;

[0249] if it is determined according to the session identifier that the initial prompt word exists i-1 rounds of prompt words having the association relationship, acquire the i th round of prompt words based on the initial prompt word;

[0250] The sending unit 1104 is specifically configured to:

[0251] for the i th round of prompt words, send the key-value cache of the prefix token group and the remaining token group to a target inference engine in the plurality of inference engines, the target inference engine being an inference engine in the plurality of inference engines and being used to process prompt words having the association relationship with the i th round of prompt words.

[0252] As a possible implementation manner, the apparatus further includes a storage unit, configured to:

[0253] If it is determined according to the session identifier that the initial prompt word does not exist prompt words having the association relationship, a first round prompt word is obtained based on the initial prompt word;

[0254] For the first round prompt word, a metadata group of the first round prompt word is stored, the metadata group of the first round prompt word including the session identifier, a plurality of words generated by the first round prompt word, and key-value caches corresponding to each of the words;

[0255] For the i th round prompt word, the metadata group of the (i-1) th round prompt word is updated to obtain the metadata group of the i th round prompt word, the metadata group of the i th round prompt word including the session identifier, a plurality of words generated by the i th round prompt word, and key-value caches corresponding to each of the words.

[0256] As a possible implementation manner, the matching unit 1102 is specifically configured to:

[0257] The k th word in the plurality of words included in the i th round prompt word is matched with the k th word in the historical word group, k being an integer starting from 1;

[0258] If the matching is successful, k+1 is updated as k, and the step of matching the k th word in the plurality of words included in the i th round prompt word with the k th word in the historical word group is executed;

[0259] If the matching fails, the words in the i th round prompt word that match successfully are taken as the prefix word group, and the words in the i th round prompt word other than the prefix word group are taken as the remaining word group.

[0260] As a possible implementation manner, the obtaining unit 1101 is specifically configured to:

[0261] Obtaining a query statement of the i th round dialogue, the (i-1) th round prompt word, and response content for the (i-1) th round prompt word;

[0262] Splicing the (i-1) th round prompt word, the response content for the (i-1) th round prompt word, and the query statement of the i th round dialogue to obtain the i th round prompt word.

[0263] As a possible implementation manner, the device further includes a deleting unit configured to, in response to receiving a deletion instruction for the j th round dialogue in the first i-1 round dialogues, delete the words included in the j th round dialogue from the (i-1) th round prompt word, and delete the key-value caches corresponding to the words included in the j th round dialogue from the graphic processor, the j th round dialogue including a query statement of the j th round dialogue and response content for the j th round prompt word, j being less than or equal to i-1.

[0264] As a possible implementation, the matching unit 1102 is specifically configured to:

[0265] If the deletion instruction is not acquired before the i-th round of the prompt word is acquired, the prefix word group is obtained by taking the word segmentation included in the (i-1)-th round of the prompt word and the word segmentation included in the response content obtained for the (i-1)-th round of the prompt word as the prefix word group, and obtaining the remaining word group according to the prefix word group and the i-th round of the prompt word.

[0266] If the deletion instruction is acquired before the i-th round of the prompt word is acquired, the prefix word group and the remaining word group are obtained by matching the word segmentation with the same word segmentation arrangement order according to the multiple word segmentations included in the i-th round of the prompt word and the historical word group.

[0267] The embodiments of the present application also provide a computer device, which can be a server or a terminal device. The computer device provided by the embodiments of the present application will be introduced from the perspective of hardware materialization. FIG. 12 shows a structural schematic diagram of a server, and FIG. 13 shows a structural schematic diagram of a terminal device.

[0268] Referring to FIG. 12, which is a structural schematic diagram of a server provided by the embodiments of the present application, the server 1400 can have great differences due to different configurations or performances, and can include one or more processors 1422, such as a central processing unit (CPU), a memory 1432, one or more storage media 1430 (for example, one or more mass storage devices) of an application program 1442 or data 1444. The memory 1432 and the storage medium 1430 can be temporary storage or persistent storage. The program stored in the storage medium 1430 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Further, the processor 1422 can be configured to communicate with the storage medium 1430 and execute a series of instruction operations in the storage medium 1430 on the server 1400.

[0269] The server 1400 can also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM , and the like.

[0270] The steps performed by the server in the above embodiments can be based on the server structure shown in FIG. 12.

[0271] The processor 1422 is configured to perform the following steps:

[0272] Obtain an i-th round of prompt words, i being an integer greater than 1;

[0273] According to the plurality of words included in the i-th round of prompt words and the historical word group, match the words with the same word arrangement order according to the word arrangement order, to obtain a prefix word group and a remaining word group, the historical word group being the words with the word arrangement order obtained based on the first i-1 rounds of prompt words, the first i-1 rounds of prompt words having a correlation with the i-th round of prompt words, the prefix word group including the words in the i-th round of prompt words that match successfully when the first matching fails, and the remaining word group including the words in the plurality of words included in the i-th round of prompt words except the words included in the prefix word group;

[0274] If the key-value cache corresponding to the prefix word group fails to be obtained from the graphics processor, the key-value cache corresponding to the prefix word group is obtained from the central processor;

[0275] The key-value cache of the prefix word group and the remaining word group are sent to the inference engine, so that the inference engine obtains the response content for the i-th round of prompt words according to the key-value cache of the prefix word group and the remaining word group.

[0276] Optionally, the processor 1422 can also perform the method steps of any of the specific implementation manners of the key-value cache multiplexing method in the embodiments of the present application.

[0277] Referring to FIG. 13, which is a structural schematic diagram of a terminal device provided by an embodiment of the present application. Taking the terminal device as a smart phone as an example, FIG. 13 shows a block diagram of part of the structure of the smart phone, which includes a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a wireless fidelity (WiFi) module 1570, a processor 1580, and a power supply 1590, and the like. Those skilled in the art can understand that the structure of the smart phone shown in FIG. 13 does not constitute a limitation on the smart phone, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0278] The components of the smart phone will be specifically introduced below in combination with FIG. 13:

[0279] The RF circuit 1510 can be used for receiving and transmitting signals in the process of information or communication, in particular, receiving the downlink information from the base station and processing by the processor 1580. In addition, the uplink data is transmitted to the base station.

[0280] The memory 1520 can be used to store software programs and modules, and the processor 1580 can realize various functions of the smart phone and data processing by running the software programs and modules stored in the memory 1520.

[0281] The input unit 1530 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the smart phone. Specifically, the input unit 1530 can include a touch panel 1531 and other input devices 1532. The touch panel 1531, also known as a touch screen, can collect user touch operations on or near it and drive the corresponding connection device according to the pre-set program. In addition to the touch panel 1531, the input unit 1530 can also include other input devices 1532. Specifically, the other input devices 1532 can include one or more of, but not limited to, a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), trackballs, mice, joysticks, etc.

[0282] The display unit 1540 can be used to display information input by the user or information provided to the user and various menus of the smart phone. The display unit 1540 can include a display panel 1541, which can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0283] The smart phone can also include at least one sensor 1550, such as a light sensor, a motion sensor, and other sensors. As for the gyroscope, barometer, hygrometer, thermometer, infrared sensor, and other sensors that the smart phone can also be configured, they are not described here.

[0284] The audio circuit 1560, the speaker 1561, and the microphone 1562 can provide an audio interface between the user and the smart phone. The audio circuit 1560 can convert the received audio data into an electrical signal and transmit it to the speaker 1561, which converts it into a sound signal output. On the other hand, the microphone 1562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1560 and converted into audio data. After the audio data is output to the processor 1580 for processing, it is transmitted to another smart phone via the RF circuit 1510, or the audio data is output to the memory 1520 for further processing.

[0285] The processor 1580 is a control center of the smartphone, which connects various parts of the smartphone through various interfaces and lines, and performs various functions of the smartphone and processes data by running or executing software programs and / or modules stored in the memory 1520 and calling data stored in the memory 1520. Optionally, the processor 1580 can include one or more processing units.

[0286] In the embodiment of the present application, the memory 1520 included in the smartphone can store a computer program and transmit the computer program to the processor.

[0287] The processor 1580 included in the smartphone can execute the key-value cache multiplexing method provided in the above embodiment according to the instructions in the computer program.

[0288] The embodiment of the present application also provides a computer readable storage medium for storing a computer program, the computer program being used to execute the key-value cache multiplexing method provided in the above embodiment.

[0289] The embodiment of the present application also provides a computer program product or a computer program, the computer program product or the computer program including computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the key-value cache multiplexing method provided in various optional implementation manners of the above aspect.

[0290] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium. When the program is executed, the steps of the above method embodiments are executed. The storage medium can be at least one of the following media: read-only memory (English: Read-Only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, and various media that can store computer programs.

[0291] In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or a memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0292] It should be noted that each of the embodiments of the present specification is described in a progressive manner, and the same or similar parts between each embodiment can be mutually referred to, and each embodiment focuses on the difference from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts can be referred to the part of the description of the method embodiments. The above-described device and system embodiments are only illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to the actual needs. Those skilled in the art can understand and implement it without creative labor.

[0293] The above describes only one specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. On the basis of the implementation manners provided by the above aspects of the present application, further combinations can be made to provide more implementation manners. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for reusing key-value caches, the method being performed by a computer device, and the method comprising: obtaining an i-th round of prompt words, i being an integer greater than 1; matching, according to a plurality of word pieces included in the i-th round of prompt words and historical word piece groups, word pieces having a same word piece arrangement order, to obtain a prefix word piece group and a remaining word piece group, the historical word piece groups being word pieces having the word piece arrangement order obtained based on i-1 previous rounds of prompt words, the i-1 previous rounds of prompt words having a correlation with the i-th round of prompt words, the prefix word piece group including word pieces in the i-th round of prompt words that are successfully matched for the first time, and the remaining word piece group including word pieces in the i-th round of prompt words other than the prefix word piece group; obtaining, if a key-value cache corresponding to the prefix word piece group fails to be obtained from a graphics processing unit, the key-value cache corresponding to the prefix word piece group from a central processing unit; and sending the key-value cache of the prefix word piece group and the remaining word piece group to an inference engine, so that the inference engine obtains response content for the i-th round of prompt words according to the key-value cache of the prefix word piece group and the remaining word piece group.

2. The method of claim 1, before the obtaining the i-th round of prompt words, the method further comprising: obtaining a remaining video memory space of the graphics processing unit; and if the remaining video memory space of the graphics processing unit is less than a preset storage space, determining a first target key-value cache from key-value caches stored in the graphics processing unit, storing the first target key-value cache to the central processing unit, and deleting the first target key-value cache from the graphics processing unit, a sum of the first target key-value cache and the remaining video memory space being greater than or equal to the preset storage space.

3. The method of claim 1, the method further comprising: obtaining a remaining video memory space of the graphics processing unit and a to-be-stored key-value cache generated by the i-th round of prompt words, the to-be-stored key-value cache being determined according to a key-value cache of one word piece and a number of word pieces to be stored in the remaining word piece group; if the remaining video memory space is less than the to-be-stored key-value cache, determining a second target key-value cache from key-value caches stored in the graphics processing unit, storing the second target key-value cache to the central processing unit, deleting the second target key-value cache from the graphics processing unit, and storing the to-be-stored key-value cache to the graphics processing unit, a sum of the second target key-value cache and the remaining video memory space being greater than or equal to the to-be-stored key-value cache; and if the remaining video memory space is greater than or equal to the to-be-stored key-value cache, storing the to-be-stored key-value cache to the graphics processing unit.

4. The method of claim 3, the storing the second target key-value cache to the central processing unit comprising: obtaining a remaining memory space of the central processing unit. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ If the remaining memory space is less than the second target key-value cache, determining a third target key-value cache from the central processor, deleting the third target key-value cache from the central processor, storing the second target key-value cache to the central processor, the sum of the third target key-value cache and the remaining memory space is greater than or equal to the second target key-value cache, and the storage time of the third target key-value cache is greater than a preset time threshold; If the remaining memory space is greater than or equal to the second target key-value cache, storing the second target key-value cache to the central processor.

5. The method of claim 1, further comprising: If the key-value cache corresponding to the prefix token group fails to be obtained from the central processor and the graphics processor, sending the prefix token group and the remaining token group to the inference engine, so that the inference engine obtains the response content for the i th round of prompt words according to the prefix token group and the remaining token group.

6. The method of claim 1, wherein the i th round of prompt words is obtained, comprising: Obtaining an initial prompt word including a session identifier, the session identifier being used to identify the association relationship; If it is determined according to the session identifier that the initial prompt word exists with the first i-1 round of prompt words having the association relationship, obtaining the i th round of prompt words based on the initial prompt word; The key-value cache of the prefix token group and the remaining token group are sent to the inference engine, comprising: For the i th round of prompt words, the key-value cache of the prefix token group and the remaining token group are sent to a target inference engine in the plurality of inference engines, the target inference engine being an inference engine in the plurality of inference engines for processing prompt words having the association relationship with the i th round of prompt words.

7. The method of claim 6, further comprising: If it is determined according to the session identifier that the initial prompt word does not exist with the prompt words having the association relationship, obtaining the first round of prompt words based on the initial prompt word; For the first round of prompt words, storing a metadata group of the first round of prompt words, the metadata group of the first round of prompt words including the session identifier, a plurality of tokens generated by the first round of prompt words, and the key-value cache corresponding to each token; For the i th round of prompt words, updating the metadata group of the i-1 round of prompt words to obtain the metadata group of the i th round of prompt words, the metadata group of the i th round of prompt words including the session identifier, a plurality of tokens generated by the i th round of prompt words, and the key-value cache corresponding to each token.

8. The method of claim 1, wherein the plurality of tokens included in the i th round of prompt words and the historical token group are matched according to the token arrangement order of the tokens with the same token arrangement order to obtain the prefix token group and the remaining token group, comprising: Matching the k th token in the plurality of tokens included in the i th round of prompt words with the k th token in the historical token group, k being an integer starting from 1; If the matching is successful, k+1 is updated as k, and the step of matching the kth word in the plurality of words included in the ith prompt word with the kth word in the historical word group is performed; If the matching fails, the words in the ith prompt word that match successfully are taken as the prefix word group, and the words in the ith prompt word other than the prefix word group are taken as the remaining word group.

9. The method of claim 1, wherein the obtaining the ith prompt word comprises: obtaining a query statement of an ith dialogue, an (i-1)th prompt word, and response content corresponding to the (i-1)th prompt word; concatenating the (i-1)th prompt word, the response content corresponding to the (i-1)th prompt word, and the query statement of the ith dialogue to obtain the ith prompt word.

10. The method of claim 9, further comprising: in response to receiving a deletion instruction corresponding to a jth dialogue in the first i-1 dialogues, deleting words included in the jth dialogue from the (i-1)th prompt word, and deleting key-value caches corresponding to the words included in the jth dialogue from the graphic processor, the jth dialogue including a query statement of the jth dialogue and response content corresponding to the jth prompt word, j being less than or equal to i-1.

11. The method of claim 10, further comprising: if the deletion instruction is not obtained before the ith prompt word is obtained, taking words included in the (i-1)th prompt word and words included in response content corresponding to the (i-1)th prompt word as the prefix word group, and obtaining the remaining word group from the prefix word group and the ith prompt word; if the deletion instruction is obtained before the ith prompt word is obtained, performing the step of matching words having the same word arrangement order from the plurality of words included in the ith prompt word and the historical word group according to the word arrangement order to obtain the prefix word group and the remaining word group.

12. An apparatus for reusing a key-value cache, the apparatus comprising: an obtaining unit, a matching unit, a multiplexing unit, and a sending unit; the obtaining unit is configured to obtain an ith prompt word, i being an integer greater than 1; the matching unit is configured to match words having the same word arrangement order from the plurality of words included in the ith prompt word and a historical word group according to the word arrangement order to obtain a prefix word group and a remaining word group, the historical word group being words having the word arrangement order obtained based on (i-1)th prompt words, the (i-1)th prompt words having a correlation with the ith prompt word, the prefix word group including words in the ith prompt word that match successfully for the first time, and the remaining word group including words in the ith prompt word other than the prefix word group; the multiplexing unit is configured to obtain key-value caches corresponding to the prefix word group from a central processing unit if the key-value caches corresponding to the prefix word group fail to be obtained from a graphic processor. The sending unit is configured to send the key value cache of the prefix token group and the remaining token group to an inference engine, so that the inference engine obtains response content for the i-th round of prompt words according to the key value cache of the prefix token group and the remaining token group. 13.A computer device, comprising a processor and a memory: The memory is configured to store a computer program and transmit the computer program to the processor. The processor is configured to execute the method according to any one of claims 1-11 according to the computer program. 14.A computer readable storage medium configured to store a computer program, the computer program being configured to execute the method according to any one of claims 1-11. 15.A computer program product comprising a computer program which, when executed on a computer device, causes the computer device to execute the method according to any one of claims 1-11.

Images