Long text memory retention method, system, device and storage medium based on large language model

Abstract

The application relates to a long text memory keeping method, system, device and storage medium based on a large language model. The method comprises the following steps: grading the text content in long text information through a word segmenter in the large language model to obtain various levels of various text contents; respectively calculating the context information weight of the text content at each level based on an attention mechanism to obtain the weight corresponding to each level; storing each text content in a short-term memory space or a long-term memory space according to the weight corresponding to each level; and using a KV Cache to compress the short-term memory space or the long-term memory space, obtaining the text content reserved after compression and outputting the same. The text content in the long text information is graded, and then the weight of the corresponding level is calculated, so that the logical consistency of the context content in multiple dimensions is ensured; the memory space is compressed using the KV Cache, so that the inference time of the large language model can be accelerated.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, system, device, and storage medium for long text memory retention based on a large language model. Background Technology

[0002] In recent years, with the rapid development of natural language processing technology, Large Language Models (LLMs) such as GPT and BERT have made significant progress in many fields. These models have been widely applied to tasks such as machine translation, sentiment analysis, dialogue systems, and generative writing. However, these models still face significant challenges when processing long text information, especially in the memory and retention of long dialogues and information across paragraphs or even chapters. The memory retention of long texts by large language models is mainly applied in fields such as dialogue systems, news summaries, legal text analysis, and scientific paper writing, which require deep semantic understanding and multi-turn reasoning of long texts. Traditional methods face significant information loss problems in these applications, such as the breakage of contextual relationships and the forgetting of important information, resulting in poor coherence in the generated content. At the same time, the lack of effective preservation of the long text context can easily lead to repetitive generation and unclear logic, seriously affecting the performance of large language models in long text generation.

[0003] To compensate for the limitations of large language models in long-term memory, researchers have proposed improvements such as hierarchical attention mechanisms, memory networks, and external storage to enhance the continuity and consistency of models in long text processing. Among these, hierarchical attention mechanisms capture different levels of contextual information at different levels, enabling the model to effectively focus on key parts of the text; memory networks introduce external memory units to record the structure of historical information, helping the model maintain context across multiple paragraphs or chapters; and models with expanded context windows extend the memory range by directly expanding the context window of the attention mechanism to meet the demands of long text processing.

[0004] However, while methods such as hierarchical attention and memory networks can improve the model's ability to process long texts to some extent, their effectiveness drops significantly when the text length exceeds a certain range, making it difficult to maintain contextual coherence in longer texts. Memory networks and external storage methods are limited by the capacity of storage units and cannot effectively accommodate all the key information in long texts, leading to problems such as information loss, repetition, and lack of contextual coherence in the generated content. As the context window expands and the model parameters increase, the computational resource requirements of long text processing models increase dramatically, which not only increases deployment costs but also limits the application of models in scenarios with limited device resources.

[0005] Therefore, traditional long text memory retention methods struggle to maintain global semantic consistency and logical coherence when processing long texts, easily leading to semantic inconsistencies or logical ambiguities in the generated content, thus affecting the performance of large language models in generative tasks. Summary of the Invention

[0006] In order to solve the above-mentioned technical problems, a method, system, device and storage medium for long text memory retention based on a large language model are provided. This can improve the memory retention capability of the large language model in long text processing and achieve high coherence and logical consistency of the generated long text content.

[0007] A long text memory retention method based on a large language model, the method comprising:

[0008] The long text information is input into a large language model. The word segmenter in the large language model is used to classify the text content in the long text information to obtain the various levels of each text content.

[0009] Based on the attention mechanism, the contextual information weights of the text content at each level are calculated to obtain the weights corresponding to each level.

[0010] Each piece of text content is stored in short-term memory or long-term memory space according to the weight corresponding to each level.

[0011] The KV Cache is used to compress the short-term memory space or long-term memory space to obtain the compressed text content retained in the short-term memory space or long-term memory space and output it.

[0012] In one embodiment, the method further includes:

[0013] Load the large language model and the word segmenter, and set initial parameters for the large language model and the word segmenter;

[0014] Obtain historical text, determine short-term memory space and long-term memory space based on the historical text, and set capacity and weight allocation strategies for the short-term memory space and long-term memory space to complete the initialization.

[0015] In one embodiment, the text content in the long text information is classified by the word segmenter in the large language model to obtain various levels of each text content, including:

[0016] The text content in the long text information is extracted using the large language model.

[0017] The text content is segmented using a word segmenter, dividing it into token-level, sentence-level, paragraph-level, chapter-level, and dialogue-level segments.

[0018] In one embodiment, contextual information weights are calculated for each level of text content based on an attention mechanism to obtain the weights corresponding to each level, including:

[0019] In the large language model, a hierarchical memory network structure is adopted, and an attention matrix is ​​generated in combination with an attention mechanism;

[0020] The text content at each level is mapped to the attention matrix, and the attention score corresponding to each level position is calculated.

[0021] The attention scores are averaged and pooled to obtain the weights corresponding to each level based on the position at each level.

[0022] In one embodiment, storing each piece of text content into short-term memory or long-term memory space according to the weight corresponding to each level includes:

[0023] Determine the target weight, compare the weights corresponding to each level with the target weight, and obtain the comparison result;

[0024] The text content corresponding to a comparison result greater than or equal to the target weight is stored in the short-term memory space;

[0025] The text content whose comparison result is less than the target weight is stored in the long-term memory space.

[0026] In one embodiment, the method further includes:

[0027] The text content in the short-term memory space is sorted in chronological order to obtain a text list;

[0028] The text list is used as a prompt to guide the output of the large language model, and the large language model is used to summarize and abstract the content.

[0029] In one embodiment, the short-term memory space or long-term memory space is compressed using a KV Cache, including:

[0030] The target content in the short-term memory space is input as a cue word into the large language model;

[0031] During the pre-filling stage of the large language model, the target content is converted into a KVCache stored in the video memory;

[0032] Each KV Cache with a weight greater than the threshold is used as embedded information of the context text content and retained in the short-term memory space to complete the compression.

[0033] A long text memory retention system based on a large language model, the system comprising:

[0034] The information classification module is used to input long text information into the large language model, and classify the text content in the long text information by the word segmenter in the large language model to obtain the various levels of each text content.

[0035] The weight calculation module is used to calculate the contextual information weights of the text content at each level based on the attention mechanism, so as to obtain the weights corresponding to each level.

[0036] The storage module is used to store each text content into a short-term memory space or a long-term memory space according to the weight corresponding to each level.

[0037] The compression module is used to compress the short-term memory space or long-term memory space using KV Cache, and output the compressed text content retained in the short-term memory space or long-term memory space.

[0038] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0039] The long text information is input into a large language model. The word segmenter in the large language model is used to classify the text content in the long text information to obtain the various levels of each text content.

[0040] Based on the attention mechanism, the contextual information weights of the text content at each level are calculated to obtain the weights corresponding to each level.

[0041] Each piece of text content is stored in short-term memory or long-term memory space according to the weight corresponding to each level.

[0042] The KV Cache is used to compress the short-term memory space or long-term memory space to obtain the compressed text content retained in the short-term memory space or long-term memory space and output it.

[0043] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0044] The long text information is input into a large language model. The word segmenter in the large language model is used to classify the text content in the long text information to obtain the various levels of each text content.

[0045] Based on the attention mechanism, the contextual information weights of the text content at each level are calculated to obtain the weights corresponding to each level.

[0046] Each piece of text content is stored in short-term memory or long-term memory space according to the weight corresponding to each level.

[0047] The KV Cache is used to compress the short-term memory space or long-term memory space to obtain the compressed text content retained in the short-term memory space or long-term memory space and output it.

[0048] The aforementioned method, system, device, and storage medium for long text memory retention based on large language models ensure the logical consistency of contextual content across multiple dimensions by classifying the text content in long text information and calculating the weights of the corresponding levels. This preserves key information at different levels and improves the high coherence and logical consistency of the generated long text content. Furthermore, using KV Cache to compress the memory space can accelerate the inference time of large language models and reduce the memory pressure caused by high concurrency. Attached Figure Description

[0049] Figure 1 This is a diagram illustrating the application environment of a long text memory retention method based on a large language model in one embodiment.

[0050] Figure 2 This is a flowchart illustrating a long text memory retention method based on a large language model in one embodiment.

[0051] Figure 3 This is a schematic diagram of the structure of a long text memory retention method based on a large language model in one embodiment;

[0052] Figure 4 This is a block diagram of a long text memory retention system based on a large language model in one embodiment;

[0053] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The long text memory retention method based on a large language model provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1As shown, the application environment includes computer device 110. Computer device 110 can input long text information into a large language model, and use the word segmenter in the large language model to classify the text content into different levels. Computer device 110 can calculate the contextual information weights of each level of text content based on an attention mechanism, obtaining the weights corresponding to each level. Computer device 110 can store each text content in short-term memory or long-term memory space according to the weights corresponding to each level. Computer device 110 can use a KV cache to compress the short-term memory or long-term memory space, obtaining the compressed text content retained in the short-term memory or long-term memory space and outputting it. Computer device 110 can be, but is not limited to, various personal computers, laptops, smartphones, robots, drones, tablets, etc.

[0056] In one embodiment, such as Figure 2 As shown, a method for long text memory retention based on a large language model is provided, including the following steps:

[0057] Step 202: Input the long text information into the large language model, and use the word segmenter in the large language model to classify the text content in the long text information to obtain the various levels of each text content.

[0058] The long text information can be multi-turn dialogue information input to the computer device. The computer device can receive user ID, dialogue ID, and user input information. The computer device can find the dialogue history corresponding to the user ID and dialogue ID, combine this dialogue history with the current user input, and then divide it into sentences, paragraphs, chapters, and dialogues using rules such as symbols (periods, commas, semicolons, etc.) and identifiers.

[0059] Specifically, in one embodiment, a long text memory retention method based on a large language model may further include a content classification process, which includes: extracting text content from long text information through a large language model; and using a word segmenter to segment the text content into token-level, sentence-level, paragraph-level, chapter-level, and dialogue-level categories.

[0060] Computer devices can classify speech history and user input into five levels—token level, sentence level, paragraph level, chapter level, and dialogue level—using a word segmenter in a large language model.

[0061] In one embodiment, a long text memory retention method based on a large language model may further include a model initialization and preprocessing process, specifically including: loading the large language model and the word segmenter, and setting initial parameters for the large language model and the word segmenter; obtaining historical text, determining short-term memory space and long-term memory space based on the historical text, and setting capacity and weight allocation strategies for the short-term memory space and long-term memory space to complete the initialization.

[0062] The computer equipment certificate allows loading pre-trained large language models and corresponding word segmenters, and configuring initial parameters. Specifically, it can divide historical text into two memory storage modules: a short-term memory storage module and a long-term memory storage module. These two modules are initialized, and the capacity and weight allocation strategy of the hierarchical memory space are set.

[0063] In this embodiment, the computer device can put the original dialogue history into the long-term memory storage module according to the user ID and dialogue ID; then the short-term memory module is initialized, and the maximum budget for short-term memory is initially set; during operation, the content of the long-term memory storage module of specific user ID and dialogue ID is extracted and sequentially transferred to the short-term memory storage module. If the extracted content exceeds the maximum budget, the large language model is used to summarize the excess part to ensure that the data is relatively complete.

[0064] Step 204: Calculate the contextual information weights for each level of text content based on the attention mechanism to obtain the corresponding weights for each level.

[0065] In the process of long text processing, computer devices can decompose the contextual text content into five levels: token level, sentence level, paragraph level, chapter level, and dialogue level. Based on the attention mechanism, the weight of contextual information is calculated, and the weights of token level, sentence level, paragraph level, chapter level, and dialogue level content are reasonably allocated. High-weight content is stored in short-term memory space for generation stage retrieval, while other content is stored in long-term memory space for subsequent retrieval.

[0066] In one embodiment, a long text memory retention method based on a large language model may further include a weight calculation process, specifically including: in the large language model, a hierarchical memory network structure is adopted, and an attention matrix is ​​generated by combining an attention mechanism; the text content of each level is mapped to the attention matrix, and the attention score corresponding to each level position is calculated; the attention score is averaged and pooled, and the weight corresponding to each level is obtained by combining the position of each level.

[0067] In large language models, a hierarchical memory network structure can be employed, combined with an adaptive context window mechanism. Based on the attention scores of the input text content and contextual content at different levels, the most important segments are selected, and these segments are retrieved from the memory space and stored in temporary memory units. Through this hierarchical storage mechanism, the model can capture key information at different levels, achieving efficient memory retention of tokens, sentences, paragraphs, chapters, and dialogue content within long texts. Then, the content in the memory units can be dynamically updated based on the context requirements generated by the model and the availability of storage space. When new information is added, the system automatically evaluates the currently stored content, deleting low-weight information to free up storage space, thereby optimizing system performance.

[0068] The attention matrix contains key information about contextual weights. The computer can map five levels of content onto the attention matrix, calculate the importance of the attention at the current position, and average the values ​​to obtain an attention score for each token position. For sentence, paragraph, chapter, and dialogue-level text content, the attention scores can be further averaged and accumulated to the corresponding token positions.

[0069] Step 206: Store each text content in short-term memory space or long-term memory space according to the weight corresponding to each level.

[0070] Among them, computer devices can adopt a hierarchical memory structure, dividing the memory storage space into more granular short-term and long-term memory storage spaces. Based on the importance and relevance of text content at different levels (token level, sentence level, paragraph level, chapter level, dialogue level), different text positions are given weights, and more critical texts with larger comprehensive weights are selected from large to small to maintain contextual information across paragraphs and chapters.

[0071] In one embodiment, a long text memory retention method based on a large language model may further include the process of storing information in a memory space. The specific process includes: determining a target weight, comparing the weights corresponding to each level with the target weight to obtain a comparison result; storing the text content corresponding to the comparison result being greater than or equal to the target weight in a short-term memory space; and storing the text content corresponding to the comparison result being less than the target weight in a long-term memory space.

[0072] The computer device can divide the attention score of each token position according to the highest dialogue level, select the K1 dialogues with the highest scores and put them into the short-term memory storage module; among the remaining content, select the K2 chapter content with the highest scores and put it into the short-term memory storage module; then select the remaining K3 paragraph-level content and put it into the short-term memory storage module; finally, for the remaining sentence-level content, select K4 and put them in the short-term memory storage module, so as to ensure that the logic of the dialogue is smooth and coherent, and that no important content is missing.

[0073] In one embodiment, the computer device can sort the text content in the short-term memory space in chronological order to obtain a text list; use the text list as a prompt to guide the output of the large language model; and use the large language model to summarize and abstract.

[0074] That is, when paragraph, chapter, and dialogue modules are too long, a large language model is used to summarize and abstract them, thereby greatly reducing the computational length of paragraph, chapter, and dialogue content, which can reduce resource consumption and computation time.

[0075] The content in the short-term memory module is ordered chronologically by dialogue time and used as final cues to guide the output of the large language model. Each time a new dialogue arrives, the attention score is used to determine which content needs to be retained and updated. The large language model can also summarize content exceeding the maximum capacity of the short-term memory module, ensuring that relatively coherent content is preserved.

[0076] In this embodiment, the text is compressed and extracted by calculating context weights, and the key information most relevant to the current content is selected, thereby optimizing storage space and improving generation efficiency. Furthermore, the secondary relevant text information is summarized, which can ensure the coherence of the context.

[0077] Step 208: Use KV Cache to compress the short-term memory space or long-term memory space to obtain the compressed text content retained in the short-term memory space or long-term memory space and output it.

[0078] In one embodiment, a long text memory retention method based on a large language model may further include a compression process, specifically including: inputting the target content in the short-term memory space as cue words into the large language model; converting the target content into a KV Cache stored in the video memory during the pre-filling stage of the large language model; and retaining each KV Cache with a weight greater than a threshold as embedding information of the context text content in the short-term memory space to complete the compression.

[0079] In the pre-filling stage of a large language model, by calculating attention weights at different levels, relatively important key-value caches can be dynamically selected for retention, which can reduce the inference time of the large language model.

[0080] Specifically, in high-concurrency scenarios, the actual GPU memory used for inference increases with rising concurrency. The main reason for this increase is the continuous growth of the key-value cache (KV cache) of the large language model during the inference phase. The content in the short-term memory module is input as prompt words into the large language model, and during the pre-filling phase of the large language model, it becomes stored in the KV cache of the GPU memory. If its size is not controlled, it will affect the entire service.

[0081] Specifically, in the large language model pre-filling stage, after obtaining the attention score, the M KVCaches with the highest scores are obtained, and these KVCaches are retained as the most critical embedded information of the context text content. By setting the hierarchical dynamic retention mode according to the layers of the large language model, the memory size can be reduced in the large language model pre-filling stage.

[0082] By employing the KV Cache compression algorithm to further reduce storage space in the pre-filling stage of large language model inference, and by rationally allocating computing resources in long context window expansion and high-load scenarios, the performance and computational efficiency of the model when processing long text can be improved.

[0083] In one embodiment, a long text memory retention method based on a large language model is provided, which generates high-quality output that conforms to logical consistency through optimized short-term dynamic memory storage settings, a summary assistance module, and a KV Cache compression module.

[0084] Computer devices can update memory content based on user feedback or interaction results to maintain relevance to future dialogue needs; they can also periodically clear low-weight content in long-term memory to free up storage space; and they can adjust memory retention parameters according to usage scenarios and task requirements. Through an adaptive memory update mechanism, it can dynamically retrieve needed memories, delete or update content in memory units, ensuring that memory content always matches the needs of the current generation task.

[0085] In one embodiment, such as Figure 3As shown, a long text memory retention method based on a large language model is provided. First, the large language model is deployed as a service to determine the format of the user's current input text. The user's input text is then separated into five levels: token, sentence, paragraph, chapter, and dialogue, according to rules. An attention mechanism is used to calculate the weights of contextual information, obtaining the attention weights between the user's current input text and historical dialogue text. The text at each of the five levels is mapped to a weight, and the average relevance of the text is calculated. The most relevant text information from the five levels is then fused and placed into short-term memory (STM) for subsequent calculations. Furthermore, a portion of the less relevant paragraph, chapter, and dialogue level information is summarized using the large language model and sequentially embedded into STM. Under high load, a KVcache compression scheme is used to control the context length during the pre-filling stage, thereby ensuring the normal operation of the inference service.

[0086] In this embodiment, the five-level hierarchical extraction and storage ensures the logical consistency of contextual content across multiple dimensions. The long text memory retention module ensures high-precision memory retention while reducing the actual storage space and computation time during inference. Segment-level, chapter-level, and dialogue-level summary summaries reduce excessive computation time and resource consumption, enhancing contextual memory capabilities while ensuring accuracy. KV Cache accelerates the inference time of large language models and reduces the GPU memory pressure caused by high concurrency. By introducing a multi-turn dialogue dynamic memory retention mechanism, the model's contextual memory capability and logical consistency for long text content are enhanced.

[0087] This application provides a long text memory retention method based on a large language model. It combines five common text content levels—token level, sentence level, paragraph level, chapter level, and dialogue level—by calculating the input text content and the content weight scores of the five levels at different layers of the large language model to comprehensively determine the key information that needs to be retained, achieving multi-level and multi-content-level dynamic key content retention. By combining information from five common dimensions, it comprehensively retains key information at different levels, greatly improving the continuity and consistency of the output. Through a long short-term memory retention module and a long text summarization auxiliary module, it dynamically and selectively selects key information and summarizes non-key information. By dynamically adjusting the window length at each layer, it achieves more accurate content responses while maintaining a consistent total length.

[0088] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0089] In one embodiment, such as Figure 4 As shown, a long text memory retention system based on a large language model is provided, including: an information classification module 410, a weight calculation module 420, a storage module 430, and a compression module 440, wherein:

[0090] The information classification module 410 is used to obtain the input long text information and input it into the large language model. The word segmenter in the large language model classifies the text content in the long text information to obtain the various levels of each text content.

[0091] The weight calculation module 420 is used to calculate the contextual information weight of the text content at each level based on the attention mechanism, so as to obtain the weight corresponding to each level.

[0092] Storage module 430 is used to store each text content into short-term memory space or long-term memory space according to the weight corresponding to each level.

[0093] Compression module 440 is used to compress short-term memory space or long-term memory space using KV Cache, and output the compressed text content retained in short-term memory space or long-term memory space.

[0094] In one embodiment, the information classification module 410 is also used to load a large language model and a word segmenter, and set initial parameters for the large language model and the word segmenter; obtain historical text, determine short-term memory space and long-term memory space based on the historical text, and set capacity and weight allocation strategies for the short-term memory space and long-term memory space to complete the initialization.

[0095] In one embodiment, the information classification module 410 is also used to extract the text content from the long text information through a large language model; and to use a word segmenter to perform word segmentation on the text content, dividing the text content into token level, sentence level, paragraph level, chapter level, and dialogue level.

[0096] In one embodiment, the weight calculation module 420 is also used in a large language model to generate an attention matrix by adopting a hierarchical memory network structure and combining it with an attention mechanism; to map the text content of each level to the attention matrix and calculate the attention score corresponding to each level position; to perform average pooling on the attention score and combine it with each level position to obtain the weight corresponding to each level.

[0097] In one embodiment, the storage module 430 is further configured to determine the target weight, compare the weights corresponding to each level with the target weight, and obtain a comparison result; store the text content corresponding to the comparison result being greater than or equal to the target weight in the short-term memory space; and store the text content corresponding to the comparison result being less than the target weight in the long-term memory space.

[0098] In one embodiment, the compression module 440 is further configured to sort the text content in the short-term memory space in chronological order to obtain a text list; use the text list as a prompt to guide the output content of the large language model; and use the large language model to perform a summary.

[0099] In one embodiment, the compression module 440 is further configured to input the target content in the short-term memory space as prompt words into the large language model; in the pre-filling stage of the large language model, the target content is converted into a KVCache stored in the video memory; and each KVCache with a weight greater than a threshold is used as the embedding information of the context text content and retained in the short-term memory space to complete the compression.

[0100] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a long text memory retention method based on a large language model. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0101] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0102] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement steps of a long text memory retention method based on a large language model.

[0103] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement the steps of a long text memory retention method based on a large language model.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0105] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for long text memory retention based on a large language model, characterized in that, The method includes: The long text information is input into a large language model. The word segmenter in the large language model is used to classify the text content in the long text information to obtain the various levels of each text content. The text content at each level is weighted based on an attention mechanism to obtain the weights corresponding to each level. This includes: in the large language model, a hierarchical memory network structure is used to generate an attention matrix in combination with the attention mechanism; the text content at each level is mapped to the attention matrix, and the attention score corresponding to each level position is calculated; the attention score is averaged and pooled, and the weights corresponding to each level are obtained in combination with the level position. Storing each piece of text content into short-term memory or long-term memory space according to the weight corresponding to each level includes: determining a target weight, comparing the weight corresponding to each level with the target weight to obtain a comparison result; storing the text content corresponding to the target weight whose comparison result is greater than or equal to the target weight in the short-term memory space; and storing the text content corresponding to the target weight whose comparison result is less than the target weight in the long-term memory space. The compression process of the short-term memory space or long-term memory space using KV Cache to obtain and output the compressed text content retained in the short-term memory space or long-term memory space includes: inputting the target content in the short-term memory space as prompt words into the large language model; converting the target content into a KV Cache stored in the video memory during the pre-filling stage of the large language model; and retaining each KV Cache with a weight greater than a threshold as embedding information of the context text content in the short-term memory space to complete the compression.

2. The method for long text memory retention based on a large language model according to claim 1, characterized in that, The method further includes: Load the large language model and the word segmenter, and set initial parameters for the large language model and the word segmenter; Obtain historical text, determine short-term memory space and long-term memory space based on the historical text, and set capacity and weight allocation strategies for the short-term memory space and long-term memory space to complete the initialization.

3. The method for long text memory retention based on a large language model according to claim 1, characterized in that, The text content in the long text information is classified by the word segmenter in the large language model to obtain the various levels of each text content, including: The text content in the long text information is extracted using the large language model. The text content is segmented using a word segmenter, dividing it into token-level, sentence-level, paragraph-level, chapter-level, and dialogue-level segments.

4. The method for long text memory retention based on a large language model according to claim 1, characterized in that, The method further includes: The text content in the short-term memory space is sorted in chronological order to obtain a text list; The text list is used as a prompt to guide the output of the large language model, and the large language model is used to summarize and abstract the content.

5. A long text memory retention system based on a large language model, characterized in that, The system includes: The information classification module is used to input long text information into the large language model, and classify the text content in the long text information by the word segmenter in the large language model to obtain the various levels of each text content. The weight calculation module is used to calculate the contextual information weights of the text content at each level based on the attention mechanism, so as to obtain the weights corresponding to each level. The module includes: in the large language model, a hierarchical memory network structure is adopted, and an attention matrix is ​​generated by combining the attention mechanism; the text content at each level is mapped to the attention matrix, and the attention score corresponding to each level position is calculated; the attention score is subjected to average pooling, and the weights corresponding to each level are obtained by combining the position of each level. A storage module is used to store each piece of text content into a short-term memory space or a long-term memory space according to the weights corresponding to each level, including: determining a target weight; comparing the weights corresponding to each level with the target weight to obtain a comparison result; storing the text content corresponding to the target weight whose comparison result is greater than or equal to the target weight in the short-term memory space; and storing the text content corresponding to the target weight whose comparison result is less than the target weight in the long-term memory space. The compression module is used to compress the short-term memory space or long-term memory space using KV Cache to obtain and output the compressed text content retained in the short-term memory space or long-term memory space. This includes: inputting the target content in the short-term memory space as prompt words into the large language model; converting the target content into a KV Cache stored in the video memory during the pre-filling stage of the large language model; and retaining each KV Cache with a weight greater than a threshold as embedding information of the context text content in the short-term memory space, thus completing the compression.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images