Large model KV cache optimization method and device based on keyword discrimination
By judging the importance of word segmentation in the input text in a large language model and optimizing KV cache, the problems of information loss and compression neglect in the prior art are solved, and efficient KV cache compression and model inference efficiency are achieved.
Patent Information
- Application Number
- CN202411396111.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-02-14
AI Technical Summary
In the prior art, the method of evicting KV cache may lead to the loss of important information, while the method of generating step compressing KV cache ignores the compression of the input sequence KV cache, cannot effectively realize memory reduction and overhead calculation, and lacks in long context environments. The detailed evaluation of the generated context affects the accurate extraction of key information and the operation efficiency of the large model.
By judging the importance of each word participle in the input text of the large language model, KV cache retention of key word participles is achieved, thereby optimizing KV cache and improving the efficiency of KV cache compression and model reasoning.
It realizes the cleaning and removal of redundant caches, improves the efficiency of KV cache compression and model inference, and ensures the accurate extraction of key information and the efficient operation of large models.
Smart Images
Figure CN119443097A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of large language models, and in particular to a large model KV (Key-Value) cache optimization method and device based on keyword discrimination. Background Art
[0002] Large Language Models (LLMs) are affected by the key-value cache in the attention calculation when effectively processing long context inputs. During the model inference process, the attention of past key-value KVs needs to be calculated. As the input length increases, the decoding speed will increase linearly. The large KV cache created during the prompt increases the hardware requirements and limits the scalability of the model. In related technologies, in order to solve the huge overhead brought by the KV cache, the KV cache can be evicted by using different algorithms, such as Stream LLM, Heavy-Hitter Oracle, and Scissor Hands.
[0003] However, in the related art, the method of expelling KV cache may lead to the loss of important information, while the method of compressing KV cache in the generation step ignores the compression of the input sequence KV cache, and cannot effectively achieve memory reduction and overhead calculation. It lacks a detailed evaluation of the generated context in a long context environment, which affects the accurate extraction of key information and the operating efficiency of large models, and needs to be solved urgently. Summary of the invention
[0004] The present application provides a large model KV cache optimization method and device based on keyword discrimination to solve the problems in the related art that the method of expelling the KV cache may cause the loss of important information, and the method of compressing the KV cache in the generation step ignores the compression of the input sequence KV cache, and cannot effectively achieve memory reduction and overhead calculation. There is a lack of detailed evaluation of the generated context in a long context environment, which affects the accurate extraction of key information and the operating efficiency of the large model.
[0005] The first aspect of the present application provides a large model KV cache optimization method based on keyword discrimination, comprising the following steps: obtaining input text data of a large language model, segmenting the input text data to obtain a segmentation set; obtaining a current interactive word of the large language model based on the segmentation set, and extracting a target text to be optimized corresponding to the current interactive word from the segmentation set; based on each segmentation in the target text to be optimized, determining whether there is at least one segmentation that satisfies a preset keyword condition; if at least one segmentation exists in the target text to be optimized that satisfies the preset keyword condition, obtaining a target keyword queue based on all segmentations that satisfy the preset keyword condition, and traversing each interactive word in the segmentation set to optimize the KV cache of the input text data in the large language model.
[0006] Optionally, in one embodiment of the present application, the determining whether there is at least one segmentation word that satisfies a preset keyword condition includes: detecting whether there is at least one segmentation word in the target text to be optimized that satisfies a preset word position condition; when it is detected that there is at least one segmentation word in the target text to be optimized that satisfies the preset word position condition, determining that the segmentation word that satisfies the preset word position condition satisfies the preset keyword condition, otherwise, performing a word frequency detection on the target text to be optimized.
[0007] Optionally, in one embodiment of the present application, the word frequency detection on the target text to be optimized includes: detecting whether there is at least one word segmentation greater than or equal to a preset word frequency threshold in the target text to be optimized; when it is detected that there is at least one word segmentation greater than or equal to the preset word frequency threshold in the target text to be optimized, determining that the word segmentation greater than or equal to the preset word frequency threshold satisfies the preset keyword condition, otherwise, performing part-of-speech detection on the target text to be optimized.
[0008] Optionally, in one embodiment of the present application, the part-of-speech detection on the target text to be optimized includes: obtaining the actual part-of-speech of the current word in the target text to be optimized, and detecting whether the actual part-of-speech meets the preset part-of-speech condition; when it is detected that the actual part-of-speech meets the preset part-of-speech condition, determining that the current word meets the preset keyword condition, otherwise, determining that the current word does not meet the preset keyword condition.
[0009] The second aspect of the present application provides a large model KV cache optimization device based on keyword discrimination, including: an acquisition module, used to acquire input text data of a large language model, segment the input text data, and obtain a segmentation set; an extraction module, used to obtain the current interactive word of the large language model based on the segmentation set, and extract the target text to be optimized corresponding to the current interactive word from the segmentation set; a judgment module, used to judge whether there is at least one segmentation that satisfies a preset keyword condition based on each segmentation in the target text to be optimized; an optimization module, used to obtain a target keyword queue according to all segmentations that satisfy the preset keyword condition when there is at least one segmentation that satisfies the preset keyword condition in the target text to be optimized, and traverse each interactive word in the segmentation set to optimize the KV cache of the input text data in the large language model.
[0010] Optionally, in one embodiment of the present application, the judgment module includes: a detection unit, used to detect whether there is at least one segmentation word in the target text to be optimized that satisfies a preset word position condition; a judgment unit, used to determine whether the segmentation word that satisfies the preset word position condition satisfies the preset keyword condition when it is detected that there is at least one segmentation word in the target text to be optimized that satisfies the preset word position condition; otherwise, perform word frequency detection on the target text to be optimized.
[0011] Optionally, in one embodiment of the present application, the determination unit includes: a detection subunit, used to perform word frequency detection on the target text to be optimized, and detect whether there is at least one word segmentation greater than or equal to a preset word frequency threshold in the target text to be optimized; a determination subunit, used to, when detecting that there is at least one word segmentation greater than or equal to the preset word frequency threshold in the target text to be optimized, determine that the word segmentation greater than or equal to the preset word frequency threshold satisfies the preset keyword condition; otherwise, perform part-of-speech detection on the target text to be optimized.
[0012] Optionally, in one embodiment of the present application, the determination subunit is specifically used to: obtain the actual part of speech of the current word in the target text to be optimized, and detect whether the actual part of speech meets the preset part of speech condition; when it is detected that the actual part of speech meets the preset part of speech condition, determine that the current word meets the preset keyword condition, otherwise, determine that the current word does not meet the preset keyword condition.
[0013] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large model KV cache optimization method based on keyword discrimination as described in the above embodiment.
[0014] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the large model KV cache optimization method based on keyword discrimination as described above.
[0015] The fifth aspect of the present application provides a computer program, which, when executed, implements the large model KV cache optimization method based on keyword discrimination as described above.
[0016] The embodiment of the present application can judge the importance of each word segment in the input text of the large language model to achieve KV cache retention of key word segments, thereby realizing the cleaning and removal of redundant caches, improving the efficiency of KV cache compression and model reasoning, and being more practical. Thus, the problem in the related art that the method of expelling the KV cache may lead to the loss of important information, and the method of compressing the KV cache in the generation step ignores the compression of the input sequence KV cache, cannot effectively achieve memory reduction and overhead calculation, lacks a detailed evaluation of the generated context in a long context environment, affects the accurate extraction of key information, and affects the operating efficiency of the large model.
[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0019] Figure 1 A flowchart of a large model KV cache optimization method based on keyword discrimination provided according to an embodiment of the present application;
[0020] Figure 2 A logical diagram of keyword identification according to an embodiment of the present application;
[0021] Figure 3 It is a structural diagram of a large model KV cache optimization device based on keyword discrimination according to an embodiment of the present application;
[0022] Figure 4 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0023] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0024] The following describes the large model KV cache optimization method and device based on keyword discrimination of the embodiment of the present application with reference to the accompanying drawings. In view of the related art mentioned in the above background technology, the method of expelling the KV cache may lead to the loss of important information, and the method of compressing the KV cache in the generation step ignores the compression of the input sequence KV cache, and cannot effectively achieve memory reduction and overhead calculation, lacks a detailed evaluation of the generated context in the long context environment, affects the accurate extraction of key information, and affects the operating efficiency of the large model. The present application provides a large model KV cache optimization method based on keyword discrimination. In this method, the importance of each word segment in the input text of the large language model can be judged to achieve KV cache retention of key word segments, thereby achieving the cleaning and removal of redundant caches, improving the efficiency of KV cache compression and model reasoning, and being more practical. Thus, the problem that the method of expelling the KV cache in the related art may lead to the loss of important information, and the method of compressing the KV cache in the generation step ignores the compression of the input sequence KV cache, and cannot effectively achieve memory reduction and overhead calculation, lacks a detailed evaluation of the generated context in the long context environment, affects the accurate extraction of key information, and affects the operating efficiency of the large model is solved.
[0025] Specifically, Figure 1 A flowchart of a large model KV cache optimization method based on keyword discrimination provided in an embodiment of the present application.
[0026] like Figure 1 As shown, the large model KV cache optimization method based on keyword discrimination includes the following steps:
[0027] In step S101, the input text data of the large language model is obtained, and the input text data is segmented to obtain a segmentation set.
[0028] It can be understood that, in the embodiment of the present application, tokenizing the input text data can be performed by tokenizing a given sentence S (input text data) using a tokenization tool, using S = {s1, s2, ..., s T}express.
[0029] In step S102, the current interactive word of the large language model is obtained based on the word segmentation set, and the target text to be optimized corresponding to the current interactive word is extracted from the word segmentation set.
[0030] It can be understood that in the embodiment of the present application, the current interactive word can be each word in the word set of the input text data, and the word s in the set S can be i is the current interaction word at the current moment, and s i All previous word segmentations are used as target texts to be optimized.
[0031] In step S103, based on each segmented word in the target text to be optimized, it is determined whether there is at least one segmented word that meets the preset keyword condition.
[0032] It should be noted that the preset keyword conditions can be set by those skilled in the art according to actual conditions and are not specifically limited here.
[0033] It can be understood that in an embodiment of the present application, for the target text to be optimized of the current interactive word, each word segment in the target text to be optimized can be judged according to preset keyword conditions to confirm the importance of the word segment, thereby further confirming the key word segment that needs KV cache.
[0034] Optionally, in one embodiment of the present application, determining whether there is at least one segmentation word that satisfies a preset keyword condition includes: detecting whether there is at least one segmentation word that satisfies a preset word position condition in the target text to be optimized; when it is detected that there is at least one segmentation word that satisfies a preset word position condition in the target text to be optimized, determining that the segmentation word that satisfies the preset word position condition satisfies the preset keyword condition, otherwise, performing word frequency detection on the target text to be optimized.
[0035] It should be noted that the preset word position condition can be set by those skilled in the art according to actual conditions and is not specifically limited here.
[0036] In the actual execution process, the m words closest to the current interactive word and the n words at the front of the sequence in the current target text to be optimized can be determined to meet the preset word position conditions. At this time, the segmented words that meet the preset word position conditions are determined to be the segmented words that can meet the preset keyword conditions, and the remaining segmented words in the target text to be optimized are further subjected to word frequency detection.
[0037] Optionally, in one embodiment of the present application, word frequency detection is performed on the target text to be optimized, including: detecting whether there is at least one word segmentation greater than or equal to a preset word frequency threshold in the target text to be optimized; when it is detected that there is at least one word segmentation greater than or equal to the preset word frequency threshold in the target text to be optimized, determining that the word segmentation greater than or equal to the preset word frequency threshold satisfies the preset keyword condition, otherwise, performing part of speech detection on the target text to be optimized.
[0038] It should be noted that the preset word frequency threshold can be set by those skilled in the art according to actual conditions and is not specifically limited here.
[0039] In the actual execution process, the higher the frequency of a certain word in the entire training data, the higher the possibility that the word is a key word. The word frequency of the remaining words in the target text to be optimized is obtained, and the word frequency of each word is compared with the preset word frequency threshold. The word that is greater than or equal to the preset word frequency threshold is determined to be the word that can meet the preset keyword condition, and the remaining words in the target text to be optimized are further subjected to part-of-speech detection.
[0040] Optionally, in one embodiment of the present application, part-of-speech detection is performed on the target text to be optimized, including: obtaining the actual part-of-speech of the current word in the target text to be optimized, and detecting whether the actual part-of-speech meets the preset part-of-speech condition; when it is detected that the actual part-of-speech meets the preset part-of-speech condition, determining that the current word meets the preset keyword condition, otherwise, determining that the current word does not meet the preset keyword condition.
[0041] It should be noted that the preset part-of-speech condition can be set by those skilled in the art according to actual conditions and is not specifically limited here.
[0042] In the actual execution process, when the actual part of speech of a certain participle is any one of noun, verb, adjective and adverb, it is determined that the participle meets the preset keyword condition, and the key participle determination of the target text to be optimized is completed.
[0043] For example, Figure 2 , which is a logical diagram of keyword discrimination in an embodiment of the present application. Specifically, an empty keyword queue φ={} can be initialized, and for each word s in the target text to be optimized, i , keyword identification is performed in the following order:
[0044] Judgment 1: If s i Satisfying the word position strategy, directly add to φ, update φ, and enter the next word, that is, φ←φ∪s i , otherwise proceed to judgment 2.
[0045] Judgment 2: If Judgment 1 is not satisfied, then if s i In line with the word frequency strategy, directly add it to φ, update φ, and enter the next word, that is, φ←φ∪s i , otherwise proceed to judgment 3.
[0046] Judgment 2: When Judgment 1 and 2 are not satisfied, if s i In line with the part-of-speech strategy, directly add it to φ, update φ, and enter the next word, that is, φ←φ∪s i, otherwise go directly to the next word.
[0047] In step S104, if there is at least one word segmentation in the target text to be optimized that meets the preset keyword condition, a target keyword queue is obtained based on all word segments that meet the preset keyword condition, and each interactive word in the word segmentation set is traversed to optimize the KV cache of the input text data in the large language model.
[0048] It can be understood that in the embodiment of the present application, given a large language model containing L layers of Transformers, in the pre-filling stage, the long text S after word segmentation is inputted as {s1, s2, ..., s T}, for each layer of Transformer, only the KV cache of the keywords retained by the target keyword queue φ is saved in memory. In the decoding stage, the current interactive words only interact with the KV cache after the optimized cache, thereby significantly compressing the video memory usage of the large model during the inference process, reducing the memory access compression during the inference process, improving communication efficiency, and realizing large model inference acceleration.
[0049] According to the large model KV cache optimization method based on keyword discrimination proposed in the embodiment of the present application, the importance of each word segment in the input text of the large language model can be judged to achieve KV cache retention of key word segments, thereby realizing the cleaning and removal of redundant caches, improving the efficiency of KV cache compression and model reasoning, and being more practical. Thus, the problem in the related art that the method of expelling KV cache may lead to the loss of important information, and the method of compressing KV cache in the generation step ignores the compression of the input sequence KV cache, cannot effectively achieve memory reduction and overhead calculation, lacks a detailed evaluation of the generated context in a long context environment, affects the accurate extraction of key information, and affects the operating efficiency of the large model.
[0050] Next, a large model KV cache optimization device based on keyword discrimination proposed according to an embodiment of the present application is described with reference to the accompanying drawings.
[0051] Figure 3 It is a structural diagram of a large model KV cache optimization device based on keyword discrimination in an embodiment of the present application.
[0052] like Figure 3 As shown, the large model KV cache optimization device 10 based on keyword discrimination includes: an acquisition module 100, an extraction module 200, a judgment module 300 and an optimization module 400.
[0053] The acquisition module 100 is used to acquire input text data of the large language model, segment the input text data, and obtain a segmentation set.
[0054] The extraction module 200 is used to obtain the current interactive word of the large language model based on the word segmentation set, and extract the target text to be optimized corresponding to the current interactive word in the word segmentation set.
[0055] The judgment module 300 is used to judge whether there is at least one word that meets the preset keyword condition based on each word in the target text to be optimized.
[0056] The optimization module 400 is used to obtain a target keyword queue based on all the segmentations that meet the preset keyword conditions when there is at least one segmentation in the target text to be optimized that meets the preset keyword conditions, and traverse each interactive word in the segmentation set to optimize the KV cache of the input text data in the large language model.
[0057] Optionally, in one embodiment of the present application, the judgment module 300 includes: a judgment unit and a detection unit.
[0058] The detection unit is used to detect whether there is at least one word segmentation in the target text to be optimized that meets the preset word position condition.
[0059] The determination unit is used to determine that the segmented words satisfying the preset word position condition meet the preset keyword condition when it is detected that there is at least one segmented word satisfying the preset word position condition in the target text to be optimized; otherwise, perform word frequency detection on the target text to be optimized.
[0060] Optionally, in one embodiment of the present application, the determination unit includes: a detection subunit and a determination subunit.
[0061] The detection subunit is used to perform word frequency detection on the target text to be optimized, and detect whether there is at least one word in the target text to be optimized that is greater than or equal to a preset word frequency threshold.
[0062] The determination subunit is used to determine that the word segmentation greater than or equal to the preset word frequency threshold satisfies the preset keyword condition when it is detected that there is at least one word segmentation greater than or equal to the preset word frequency threshold in the target text to be optimized; otherwise, part of speech detection is performed on the target text to be optimized.
[0063] Optionally, in one embodiment of the present application, the determination subunit is specifically used to: obtain the actual part of speech of the current word segment in the target text to be optimized, and detect whether the actual part of speech meets the preset part of speech condition; when it is detected that the actual part of speech meets the preset part of speech condition, determine that the current word segment meets the preset keyword condition, otherwise, determine that the current word segment does not meet the preset keyword condition.
[0064] It should be noted that the aforementioned explanation of the embodiment of the large model KV cache optimization method based on keyword discrimination is also applicable to the large model KV cache optimization device based on keyword discrimination in this embodiment, and will not be repeated here.
[0065] According to the large model KV cache optimization device based on keyword discrimination proposed in the embodiment of the present application, the importance of each word segment in the input text of the large language model can be judged to achieve KV cache retention of key word segments, thereby realizing the cleaning and removal of redundant caches, improving the efficiency of KV cache compression and model reasoning, and being more practical. Thus, the problem in the related art that the method of expelling KV cache may lead to the loss of important information, and the method of compressing KV cache in the generation step ignores the compression of the input sequence KV cache, cannot effectively achieve memory reduction and overhead calculation, lacks a detailed evaluation of the generated context in a long context environment, affects the accurate extraction of key information, and affects the operating efficiency of the large model.
[0066] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0067] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .
[0068] When the processor 402 executes the program, the large model KV cache optimization method based on keyword identification provided in the above embodiment is implemented.
[0069] Furthermore, the electronic device further comprises:
[0070] The communication interface 403 is used for communication between the memory 401 and the processor 402 .
[0071] The memory 401 is used to store computer programs that can be executed on the processor 402 .
[0072] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0073] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0074] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.
[0075] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0076] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the large model KV cache optimization method based on keyword discrimination as described above is implemented.
[0077] This embodiment also provides a computer program, which, when executed, implements the large model KV cache optimization method based on keyword discrimination as described above.
[0078] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0079] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0080] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0081] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.
[0082] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0083] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0084] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0085] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A large model KV cache optimization method based on keyword discrimination, characterized in that: The following steps are involved: Obtaining input text data of a large language model, and performing word segmentation on the input text data to obtain a word segmentation set; Obtaining the current interactive word of the large language model based on the word segmentation set, and extracting the target text to be optimized corresponding to the current interactive word from the word segmentation set; Based on each segmentation in the target text to be optimized, determining whether there is at least one segmentation that meets a preset keyword condition; If there is at least one word segmentation in the target text to be optimized that satisfies the preset keyword condition, a target keyword queue is obtained based on all word segments that satisfy the preset keyword condition, and each interactive word in the word segmentation set is traversed to optimize the KV cache of the input text data in the large language model.
2. The method according to claim 1, characterized in that The step of determining whether there is at least one word segmentation that meets the preset keyword condition includes: Detecting whether there is at least one word segmentation satisfying a preset word position condition in the target text to be optimized; When it is detected that at least one segmented word in the target text to be optimized satisfies the preset word position condition, it is determined that the segmented word satisfying the preset word position condition satisfies the preset keyword condition; otherwise, a word frequency detection is performed on the target text to be optimized.
3. The method according to claim 2, characterized in that The performing word frequency detection on the target text to be optimized includes: Detecting whether there is at least one word in the target text to be optimized whose word frequency is greater than or equal to a preset word frequency threshold; When it is detected that there is at least one word segmentation greater than or equal to the preset word frequency threshold in the target text to be optimized, it is determined that the word segmentation greater than or equal to the preset word frequency threshold satisfies the preset keyword condition, otherwise, part of speech detection is performed on the target text to be optimized.
4. The method according to claim 3, characterized in that The performing part-of-speech detection on the target text to be optimized includes: Obtaining the actual part of speech of the current word in the target text to be optimized, and detecting whether the actual part of speech meets the preset part of speech condition; In the case where it is detected that the actual part of speech meets the preset part of speech condition, it is determined that the current participle meets the preset keyword condition; otherwise, it is determined that the current participle does not meet the preset keyword condition.
5. A large model KV cache optimization device based on keyword identification, characterized in that: include: An acquisition module, used for acquiring input text data of a large language model, and performing word segmentation on the input text data to obtain a word segmentation set; An extraction module, configured to obtain a current interactive word of the large language model based on the word segmentation set, and extract a target text to be optimized corresponding to the current interactive word from the word segmentation set; A judgment module, used for judging whether there is at least one word that satisfies a preset keyword condition based on each word in the target text to be optimized; The optimization module is used to obtain a target keyword queue according to all the segmentations that meet the preset keyword condition when there is at least one segmentation that meets the preset keyword condition in the target text to be optimized, and traverse each interactive word in the segmentation set to optimize the KV cache of the input text data in the large language model.
6. The device according to claim 5, characterized in that The judging module comprises: A detection unit, used to detect whether there is at least one word segmentation satisfying a preset word position condition in the target text to be optimized; A determination unit is used to determine that the segmented words satisfying the preset word position condition satisfy the preset keyword condition when it is detected that there is at least one segmented word satisfying the preset word position condition in the target text to be optimized; otherwise, perform word frequency detection on the target text to be optimized.
7. The device according to claim 6, characterized in that The determination unit comprises: A detection subunit, configured to perform word frequency detection on the target text to be optimized, and detect whether there is at least one word segmentation greater than or equal to a preset word frequency threshold in the target text to be optimized; The determination subunit is used to determine that the word segmentation greater than or equal to the preset word frequency threshold satisfies the preset keyword condition when it is detected that there is at least one word segmentation greater than or equal to the preset word frequency threshold in the target text to be optimized; otherwise, perform part of speech detection on the target text to be optimized.
8. The device according to claim 7, characterized in that The determination subunit is specifically used for: Obtaining the actual part of speech of the current word in the target text to be optimized, and detecting whether the actual part of speech meets the preset part of speech condition; In the case where it is detected that the actual part of speech meets the preset part of speech condition, it is determined that the current participle meets the preset keyword condition; otherwise, it is determined that the current participle does not meet the preset keyword condition.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large model KV cache optimization method based on keyword discrimination as described in any one of claims 1 to 4.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the large model KV cache optimization method based on keyword discrimination as described in any one of claims 1-4.
11. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the large model KV cache optimization method based on keyword discrimination as described in any one of claims 1-4.