A method, apparatus, device, and medium for context compression of large language models

By hierarchically compressing and prioritizing the retention of core information in the context input sequence of large language models, the problem of information loss in large language models with extremely long context inputs is solved, thereby improving the accuracy and efficiency of reasoning.

CN123115333APending Publication Date: 2026-11-17XINGZHI INTELLIGENT (BEIJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611329138.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-31
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

When large language models process extremely long dialogue histories or complex documents, the context input sequence exceeds the window length limit, leading to information loss, logical breaks, missing instructions, and inference errors.

Method used

The context input sequence is split into multiple units, classified into core information and secondary information according to the importance of the current task, and compressed under the word quota limit, prioritizing the retention of core information and reasonably allocating word quotas for secondary information.

Benefits of technology

It avoids logical breaks and reasoning errors caused by information loss, improves the accuracy and consistency of output, makes efficient use of context resources, reduces computational overhead and latency, and enhances task adaptability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115333A_ABST
    Figure CN123115333A_ABST
Patent Text Reader

Abstract

This application discloses a method, apparatus, device, and medium for context compression of a large language model, relating to the field of data processing technology. The method includes: acquiring the context input sequence of the current inference of the large language model and splitting it into multiple units; classifying the context input sequence into core information and secondary information based on the importance of each unit in the context input sequence to the current task; if the total number of lexical units in the core information and secondary information exceeds the current total lexical unit quota, allocating a lexical unit quota not exceeding the maximum number of input lexical units for the current inference of the large language model to the core information, and allocating a lexical unit quota not exceeding the remaining lexical unit quota to the secondary information if there is a remaining lexical unit quota in the current total lexical unit quota; compressing the context input sequence according to the lexical unit quotas of the core information and secondary information to obtain the final context input sequence. Using this method, abnormal output of the large language model due to missing core information in the context can be avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device and medium for large language model context compression. Background Technology

[0002] Large language models (such as GPT-5 and Tongyi 1000 Questions) perform well in natural language processing tasks, but their context window length is usually limited (e.g., limited to a total number of lexical units between 4K and 128K). When processing extremely long dialogue histories, multi-turn task planning, or complex documents, if the total number of lexical units in the context input sequence (i.e., the model context) exceeds the context window length limit of the large language model, the large language model will forcibly truncate the excess part, resulting in the loss of historical context information. This leads to problems such as logical breaks, missing instructions, and inference errors in the output of the large language model, which is called the "context overflow truncation" problem. Summary of the Invention

[0003] This application provides a method, apparatus, device, and medium for compressing the context of a large language model, in order to solve or at least partially solve the defects or deficiencies in related technologies.

[0004] Firstly, this application provides a method for compressing the context of a large language model, the method comprising: Obtain the context input sequence of the current inference of the large language model, and split the context input sequence into multiple units; The current task is determined based on the current user query text in the context input sequence, and the context input sequence is classified into core information and secondary information according to the importance of each unit in the context input sequence to the current task. Determine whether the sum of the number of lexical units of the core information and the secondary information exceeds the current total lexical unit quota, wherein the current total lexical unit quota is the maximum number of input lexical units for this inference by the large language model; If the total number of lexical units in the core information and the secondary information exceeds the current total lexical unit quota, then a lexical unit quota not exceeding the maximum number of input lexical units in this inference by the large language model is allocated to the core information. If there is a remaining lexical unit quota in the current total lexical unit quota, then a lexical unit quota not exceeding the remaining lexical unit quota is allocated to the secondary information. The secondary information includes key information and supplementary information, wherein the lexical unit quota allocated to the key information does not exceed a preset proportion of the remaining lexical unit quota. The context input sequence is compressed based on the lexical quotas allocated to the core information and the secondary information to obtain the final context input sequence for this inference by the large language model.

[0005] Secondly, this application provides a large language model context compression device, the large language model context compression device comprising: The input acquisition module is used to acquire the context input sequence of the large language model in this inference and to split the context input sequence into multiple units. The information classification module is used to determine the current task based on the current user question text in the context input sequence, and classify the context input sequence into core information and secondary information according to the importance of each unit in the context input sequence to the current task. The total number judgment module is used to determine whether the sum of the number of lexical units of the core information and the secondary information exceeds the current total lexical unit quota, wherein the current total lexical unit quota is the maximum number of input lexical units for this inference of the large language model; The quota allocation module is used to allocate a quota of no more than the maximum number of input lexical units in the current inference of the large language model to the core information if the sum of the number of lexical units in the core information and the secondary information exceeds the current total lexical unit quota; and to allocate a quota of no more than the remaining lexical unit quota to the secondary information when there is a remaining lexical unit quota in the current total lexical unit quota; the secondary information includes key information and supplementary information, wherein the lexical unit quota allocated to the key information does not exceed a preset proportion of the remaining lexical unit quota; The sequence compression module is used to compress the context input sequence according to the lexical quotas allocated to the core information and the secondary information, so as to obtain the final context input sequence of the large language model for this inference.

[0006] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the large language model context compression method described in the first aspect above.

[0007] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the large language model context compression method described in the first aspect.

[0008] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, apparatus, device, and medium for context compression of a large language model. The method involves splitting the context input sequence of the current inference of the large language model into multiple units. Based on the importance of each unit in the context input sequence to the current task, the context input sequence is classified into core information and secondary information. When the total number of lexical units in the core information and secondary information exceeds the current total lexical unit quota (i.e., the maximum number of input lexical units in the current inference of the large language model), a lexical unit quota not exceeding the maximum number of input lexical units in the current inference of the large language model is allocated to the core information. If there is a remaining lexical unit quota in the current total lexical unit quota, a lexical unit quota not exceeding the remaining lexical unit quota is allocated to the secondary information. The secondary information includes key information and supplementary information, wherein the lexical unit quota allocated to the key information does not exceed a preset proportion of the remaining lexical unit quota. The context input sequence is then compressed based on the lexical unit quotas of the core information and secondary information to obtain the final context input sequence. This application first classifies the context input sequence according to the importance of each unit in the context input sequence to the current task. Then, when the number of lexical units in the context input sequence exceeds the maximum number of input lexical units for the current inference of the large language model, the core information in the context input sequence that is most important and relevant to completing the inference task is retained first. After reserving sufficient lexical unit quota for the core information, if there is any remaining lexical unit quota, it is then allocated to the secondary information in the context input sequence. The context input sequence is compressed according to the lexical unit quota of different levels of information, rather than forcibly truncating the context input sequence. This avoids problems such as logical breaks, missing instructions, and inference errors in the output of the large language model due to the lack of core information in the context input sequence. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a large language model context compression method provided in an embodiment of this application; Figure 2 A schematic diagram of the functional modules of a large language model context compression device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0010] In one exemplary embodiment, such as Figure 1 As shown, a method for context compression of large language models is provided. This method is executed by a computer device, specifically a terminal or server, or both. The terminal can be, but is not limited to, various desktop computers, laptops, tablets, etc. The server can be a standalone server, a server cluster consisting of multiple servers, or a cloud server.

[0011] In the embodiments of this application, such as Figure 1 As shown, the large language model context compression method includes the following steps 101 to 105. Wherein: Step 101: Obtain the context input sequence of the large language model for this inference, and split the context input sequence into multiple units.

[0012] The contextual input sequence for this inference by the large language model includes system instructions, historical dialogue records (including user question texts and model response texts from previous rounds), documents involved in the historical dialogue records, and the current user question text.

[0013] In this step, the context input sequence for this inference by the large language model can be split into multiple units. In one optional embodiment, the splitting method can be based on sentences as units, i.e., splitting into multiple sentence units. In another optional embodiment, the splitting method can be based on continuous text segments that embody the minimum complete intent as units, i.e., splitting into multiple semantic units.

[0014] Step 102: Determine the current task based on the current user query text in the context input sequence, and classify the context input sequence into core information and secondary information according to the importance of each unit in the context input sequence to the current task.

[0015] The current task refers to the specific goal that the user wants the large language model to achieve this time. If the semantics of the current user question text in the context input sequence are complete and clear, the current task can be directly identified from the current user question text. Otherwise (for example, if the user's current question is a follow-up to the previous question), the current task can be determined based on the semantics of the current user question text and the semantics of the user question text in the historical dialogue record.

[0016] In this step, all units in the context input sequence can be categorized into core information and secondary information based on their importance to the current task. Core information refers to the content in the context input sequence necessary to complete the current task, such as explicit user instructions, the main body of the problem, logical premises, or key constraints. Secondary information refers to the content in the context input sequence other than the core information.

[0017] Optionally, secondary information may include key information and supplementary information. Key information is less important than core information and plays an important supporting role in the completion of the current task. Supplementary information is less important than key information and is considered additional information.

[0018] In an optional embodiment, the importance of each unit in the context input sequence to the current task can be determined by a pre-constructed domain-specific rule base. In this embodiment, the domain-specific rule base of the current task's domain is selected to determine the importance of each unit in the context input sequence to the current task. This domain-specific rule base stores the importance between specific tasks within the domain and each preset information unit within the domain in a structured data format, forming a structured format of "specific task - preset information unit - importance". In this embodiment, semantic matching or keyword matching can be performed between the current task and the "specific task" field in the domain-specific rule base of the current task's domain to obtain the "specific task" matched by the current task. Semantic matching or keyword matching can be performed between each unit in the context input sequence and the "preset information unit" field in the domain-specific rule base of the current task's domain to obtain the "preset information unit" matched by each unit in the context input sequence. Then, the importance corresponding to the "specific task" matched by the current task and the "preset information unit" matched by the unit in the context input sequence is queried from the domain-specific rule base of the current task's domain.

[0019] Optionally, importance in a specific domain rule base can be in numerical form or in textual form such as "core," "key," or "supplementary." If importance in a specific domain rule base is in numerical form, it can be divided into two levels (corresponding to core and secondary) or three levels (corresponding to core, key, and supplementary).

[0020] For highly specialized and structured closed-domain scenarios (such as pathology and pharmacology, adversarial tactics, risk control and compliance), these scenarios are characterized by limited task types, well-defined terminology, clear information boundaries, and low fault tolerance. If a large language model is used to dynamically determine the importance of each unit in the context input sequence to the current task, it may introduce uncontrollable semantic bias. In such scenarios, this uncertainty is usually unacceptable. Therefore, determining the importance of each unit in the context input sequence to the current task through a domain-specific rule base manually constructed by professionals can avoid semantic bias in specific domain scenarios.

[0021] Step 103: Determine whether the sum of the number of lexical units of core information and secondary information exceeds the current total lexical unit quota, where the current total lexical unit quota is the maximum number of input lexical units for this inference by the large language model.

[0022] A word is a token.

[0023] In this step, it is determined whether the sum of the number of lexical units C of the core information and the number of lexical units D of the secondary information (D = number of lexical units K of the key information + number of lexical units S of the supplementary information) exceeds the current total lexical unit quota T, that is, whether C+D (i.e. C+K+S) is greater than T.

[0024] Here, the current total token quota T represents the maximum number of input tokens for this inference by the large language model. T equals the total number of tokens in the context window of the large language model (or the length of the context window) minus the number of tokens reserved by the large language model for output generation. The value of T varies depending on the large language model. For example, the context window length of the qwen series models ranges from 128K to 1 million tokens; for instance, the qwen2-72b-instruct model supports a context window length of 131,072 tokens.

[0025] Step 104: If the total number of lexical units in the core information and secondary information exceeds the current total lexical unit quota, then allocate a lexical unit quota to the core information that does not exceed the maximum number of input lexical units in this inference of the large language model. If there is a remaining lexical unit quota in the current total lexical unit quota, allocate a lexical unit quota to the secondary information that does not exceed the remaining lexical unit quota. The secondary information includes key information and supplementary information. The lexical unit quota allocated to the key information does not exceed a preset proportion of the remaining lexical unit quota.

[0026] For core information, in one optional embodiment, it is first determined whether the number of lexical units in the core information exceeds the current total lexical unit quota, i.e., whether C is greater than T. If the number of lexical units in the core information does not exceed the current total lexical unit quota, i.e., C≤T, then a lexical unit quota equal to the number of lexical units C of the core information is allocated to the core information, i.e., the lexical unit quota allocated to the core information is C. If the number of lexical units in the core information exceeds the current total lexical unit quota, i.e., C>T, then a lexical unit quota equal to the current total lexical unit quota is allocated to the core information, i.e., the lexical unit quota allocated to the core information is T. In this embodiment, the upper limit of the lexical unit quota for the core information is T.

[0027] For secondary information, which includes key information and supplementary information, in an optional embodiment, the word segment quota allocated to key information does not exceed a preset proportion of the remaining word segment quota R, where R=TC.

[0028] Optionally, the preset ratio is greater than or equal to 0.6 and less than or equal to 0.85. For example, the preset ratio can be 0.7, that is, the upper limit of the keyword quota is 70% of the remaining keyword quota.

[0029] For key information and supplementary information, in an optional embodiment, when there is a remaining word quota R in the current total word quota, word quota is allocated to key information by the following formula (1), and word quota is allocated to supplementary information by the following formula (2); K1=min(K,p×R) formula (1); S1=min(S,R-K1) Formula (2); Where K1 is the word quota allocated to key information, K is the number of words in key information, p is the preset ratio, R is the remaining word quota, S1 is the word quota allocated to supplementary information, and S is the number of words in supplementary information.

[0030] In this embodiment, the upper limit of the word quota for key information is the portion p of the remaining word quota, and the upper limit of the word quota for supplementary information is the remaining portion of the remaining word quota.

[0031] Step 105: Compress the context input sequence according to the word quotas allocated to the core information and secondary information to obtain the final context input sequence for this inference of the large language model.

[0032] The compression method for core information is as follows: based on the allocated word quota C, the first C words in the core information are retained, and words after the first C words are deleted. The compression method for key information is as follows: based on the allocated word quota K1, the first K1 words in the key information are retained, and words after the first K1 words are deleted. The compression method for supplementary information is as follows: based on the allocated word quota S1, the first S1 words in the supplementary information are retained, and words after the first S1 words are deleted. The words in the core information, key information, and supplementary information are arranged in ascending order according to their sequence in the context input sequence.

[0033] In one optional embodiment, for core information: when the word quota allocated to the core information is equal to the number of words C in the core information, i.e., C≤T, all words in the core information are retained; when the word quota allocated to the core information is equal to the current total word quota T, i.e., C>T, the current total word quota is used as a word number constraint to generate a summary of the core information, resulting in compressed core information.

[0034] In this embodiment, if C ≤ T, the word quota for the core information is C, and all words in the core information can be retained. If C > T, the word quota for the core information is T, and not all words in the core information can be retained. To address the situation where not all words in the core information can be retained, this embodiment does not directly extract the first T words from the core information. Instead, it generates a core information summary with no more than T words to preserve the complete semantic information of the core information and avoid affecting the accuracy of the reasoning results.

[0035] For secondary information, in one optional embodiment: when the word quota allocated to the key information is p×R, that is, when the number of words K in the key information is greater than p×R, not all words in the key information can be retained. The first p×R words in the key information are retained to obtain the compressed key information; when the word quota allocated to the supplementary information is R-K1, that is, when the number of words S in the supplementary information is greater than R-K1, not all words in the supplementary information can be retained. The first R-K1 words in the supplementary information are retained to obtain the compressed supplementary information.

[0036] In this embodiment, a sequential truncation strategy is used for key information and supplementary information, rather than random retention. The rationale for sequential truncation is that the system has already performed a structured sorting of the input content during the information classification stage, and content with higher importance will be prioritized and placed first. Therefore, sequential truncation applies to information sequences that have already been sorted by importance, and content that appears earlier does indeed have a higher retention priority, rather than being an arbitrary positional preference.

[0037] In one alternative implementation, information can be sorted in order of importance (e.g., core information, key information, and supplementary information), and then further sorted according to its relevance to the current task. This ensures that the least relevant tail content is discarded first during truncation, thereby improving the rationality of the truncation strategy.

[0038] If the total number of lexical units in the core information, key information, and supplementary information exceeds the current total lexical unit quota, at least one of the core information, key information, and supplementary information will be compressed. Finally, the processed core information (compressed or uncompressed), processed key information (compressed or uncompressed), and processed supplementary information (compressed or uncompressed) will be combined in the order of "core-key-supplementary" to obtain the final context input sequence for this inference by the large language model. The number of lexical units N in the final context input sequence satisfies N≤T. The large language model will perform this inference based on the final context input sequence (i.e., structured, high signal-to-noise ratio context) and generate the response text.

[0039] The “core-key-supplement” combination sequence achieves the following task processing logic: When the large language model processes the final context input sequence, the content at the beginning has a greater impact on its attention allocation (i.e., the “position bias effect”). Placing the core information at the beginning of the final context input sequence can ensure that the large language model prioritizes and makes full use of the information necessary for the task, thereby improving the accuracy of reasoning.

[0040] In the exemplary embodiments of this application, the following beneficial effects are achieved through the layered dynamic compression mechanism: Mitigating catastrophic forgetting: By prioritizing the retention of core information, the risk of content necessary to complete the current task being truncated is significantly reduced; Improve output accuracy and logical consistency: Ensure that large language models can reason based on semantically complete and structured contexts, reducing errors caused by missing information; Efficiently utilize context resources: Through a proportional and adaptive lexical quota allocation strategy, the proportion of task-related information at different levels is reasonably allocated within a limited context window length; Reduce computational overhead and inference latency: Reduce invalid token input, alleviate the computational burden on the model, and improve response speed; Enhanced task adaptability and scalability: The classification criteria and compression ratio can be flexibly adjusted according to specific business scenarios, making it suitable for a variety of complex tasks such as battle planning, multi-turn dialogues, and long document reasoning.

[0041] Based on the same inventive concept, this application also provides a large language model context compression apparatus for implementing the large language model context compression method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the large language model context compression apparatus provided below can be found in the limitations of the large language model context compression method described above, and will not be repeated here.

[0042] In one exemplary embodiment, such as Figure 2 As shown, a large language model context compression device is provided, which includes: The input acquisition module is used to acquire the context input sequence of the large language model for this inference and to break the context input sequence into multiple units. The information classification module is used to determine the current task based on the current user query text in the context input sequence, and classify the context input sequence into core information and secondary information according to the importance of each unit in the context input sequence to the current task. The total count judgment module is used to determine whether the sum of the number of lexical units of core information and secondary information exceeds the current total lexical unit quota, where the current total lexical unit quota is the maximum number of input lexical units for this inference by the large language model; The quota allocation module is used to allocate a quota of no more than the maximum number of input lexical units in the current inference of the large language model to the core information if the total number of lexical units in the core information and the secondary information exceeds the current total quota of lexical units. If there is a remaining quota of lexical units in the current total quota of lexical units, it allocates a quota of no more than the remaining quota of lexical units to the secondary information. The secondary information includes key information and supplementary information. The quota of lexical units allocated to the key information does not exceed a preset proportion of the remaining quota of lexical units. The sequence compression module is used to compress the context input sequence according to the word quotas allocated to the core and secondary information, so as to obtain the final context input sequence of the large language model for this inference.

[0043] Optionally, a lexical quota not exceeding the maximum number of input lexical units in this inference by the large language model may be allocated to the core information, including: Determine whether the number of lexical elements in the core information exceeds the current total lexical element quota; If the number of lexical units in the core information does not exceed the current total lexical unit quota, then the same lexical unit quota as the number of lexical units in the core information will be allocated to the core information. If the number of lexical units in the core information exceeds the current total lexical unit quota, then the same lexical unit quota as the current total lexical unit quota will be allocated to the core information.

[0044] Optionally, the contextual input sequence is compressed based on the lexical quotas allocated to core and secondary information, including: When the number of lexical units allocated to the core information is equal to the number of lexical units in the core information, all lexical units in the core information are retained. When the word quota allocated to the core information is equal to the current total word quota, the current total word quota is used as a word number constraint to generate a summary of the core information, resulting in compressed core information.

[0045] Optionally, when there are remaining word unit quotas in the current total word unit quota, a word unit quota not exceeding the remaining word unit quota may be allocated to secondary information, including: When there are remaining word quotas in the current total word quota, word quotas are allocated to key information using the following formula (1), and word quotas are allocated to supplementary information using the following formula (2); K1=min(K,p×R) formula (1); S1=min(S,R-K1) Formula (2); Where K1 is the word quota allocated to key information, K is the number of words in key information, p is the preset ratio, R is the remaining word quota, S1 is the word quota allocated to supplementary information, and S is the number of words in supplementary information.

[0046] Optionally, the contextual input sequence is compressed based on the lexical quotas allocated to core and secondary information, including: When the word quota allocated to the key information is p×R, the first p×R words in the key information are retained to obtain the compressed key information. When the token quota allocated to the supplementary information is R-K1, the first R-K1 tokens in the supplementary information are retained to obtain the compressed supplementary information.

[0047] Optionally, the preset ratio is greater than or equal to 0.6 and less than or equal to 0.85.

[0048] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0049] The aforementioned computer device may be, for example, a server or a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a large language model context compression method.

[0050] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0051] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0052] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for context compression of large language models, characterized in that, The large language model context compression method includes: Obtain the context input sequence of the current inference of the large language model, and split the context input sequence into multiple units; The current task is determined based on the current user query text in the context input sequence, and the context input sequence is classified into core information and secondary information according to the importance of each unit in the context input sequence to the current task. Determine whether the sum of the number of lexical units of the core information and the secondary information exceeds the current total lexical unit quota, wherein the current total lexical unit quota is the maximum number of input lexical units for this inference by the large language model; If the total number of lexical units in the core information and the secondary information exceeds the current total lexical unit quota, then a lexical unit quota not exceeding the maximum number of input lexical units in this inference by the large language model is allocated to the core information. If there is a remaining lexical unit quota in the current total lexical unit quota, then a lexical unit quota not exceeding the remaining lexical unit quota is allocated to the secondary information. The secondary information includes key information and supplementary information, wherein the lexical unit quota allocated to the key information does not exceed a preset proportion of the remaining lexical unit quota. The context input sequence is compressed based on the lexical quotas allocated to the core information and the secondary information to obtain the final context input sequence for this inference by the large language model.

2. The large language model context compression method according to claim 1, characterized in that, The allocation of a lexical quota to the core information, not exceeding the maximum number of input lexical units in this inference by the large language model, includes: Determine whether the number of lexical units in the core information exceeds the current total lexical unit quota; If the number of lexical units in the core information does not exceed the current total lexical unit quota, then a lexical unit quota equal to the number of lexical units in the core information is allocated to the core information. If the number of lexical units in the core information exceeds the current total lexical unit quota, then the same lexical unit quota as the current total lexical unit quota is allocated to the core information.

3. The large language model context compression method according to claim 2, characterized in that, The compression of the context input sequence based on the lexical quotas allocated to the core information and the secondary information includes: When the lexical quota allocated to the core information is equal to the number of lexical units in the core information, all lexical units in the core information are retained. When the lexical quota allocated to the core information is equal to the current total lexical quota, the current total lexical quota is used as a lexical number constraint to generate a summary of the core information, resulting in compressed core information.

4. The large language model context compression method according to claim 1, characterized in that, When there is a remaining word segment quota in the current total word segment quota, allocating a word segment quota not exceeding the remaining word segment quota to the secondary information includes: When there are remaining word quotas in the current total word quota, word quotas are allocated to the key information using the following formula (1), and word quotas are allocated to the supplementary information using the following formula (2); K1=min(K,p×R) formula (1); S1=min(S,R-K1) Formula (2); Wherein, K1 is the word element quota allocated to the key information, K is the number of words in the key information, p is the preset ratio, R is the remaining word element quota, S1 is the word element quota allocated to the supplementary information, and S is the number of words in the supplementary information.

5. The large language model context compression method according to claim 4, characterized in that, The compression of the context input sequence based on the lexical quotas allocated to the core information and the secondary information includes: When the word quota allocated to the key information is p×R, the first p×R words in the key information are retained to obtain the compressed key information; When the word quota allocated to the supplementary information is R-K1, the first R-K1 words in the supplementary information are retained to obtain the compressed supplementary information.

6. The large language model context compression method according to claim 1, characterized in that, The preset ratio is greater than or equal to 0.6 and less than or equal to 0.

85.

7. A large language model context compression device, characterized in that, The large language model context compression device includes: The input acquisition module is used to acquire the context input sequence of the large language model in this inference and to split the context input sequence into multiple units. The information classification module is used to determine the current task based on the current user question text in the context input sequence, and classify the context input sequence into core information and secondary information according to the importance of each unit in the context input sequence to the current task. The total number judgment module is used to determine whether the sum of the number of lexical units of the core information and the secondary information exceeds the current total lexical unit quota, wherein the current total lexical unit quota is the maximum number of input lexical units for this inference of the large language model; The quota allocation module is used to allocate a quota of no more than the maximum number of input lexical units in the current inference of the large language model to the core information if the sum of the number of lexical units in the core information and the secondary information exceeds the current total lexical unit quota; and to allocate a quota of no more than the remaining lexical unit quota to the secondary information when there is a remaining lexical unit quota in the current total lexical unit quota; the secondary information includes key information and supplementary information, wherein the lexical unit quota allocated to the key information does not exceed a preset proportion of the remaining lexical unit quota; The sequence compression module is used to compress the context input sequence according to the lexical quotas allocated to the core information and the secondary information, so as to obtain the final context input sequence of the large language model for this inference.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the large language model context compression method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the large language model context compression method according to any one of claims 1-6.