Large language model dialogue system for Token-level resident memory and use method

Abstract

The invention discloses a big language model dialogue system for Token-level resident memory and a use method. The big language model dialogue system comprises an inference engine module, a big language model module, an independent memory model module and an external storage module, relates to the technical field of big language model memory enhancement, after a current dialogue Token sequence is received, whether historical dialogue information is needed or not is judged through a big language model module, and if yes, the historical dialogue information is called from an external storage module through an independent memory model module, so that information displayed by finally responding to the Token sequence is more complete, and user experience is improved. The problems that a model cannot completely retain and utilize long-distance historical information, so that questions cannot be answered and information is forgotten are solved; an external storage module is arranged, and model parameters of an independent memory model module and Token-level data of all historical conversations are resided through a memory mapping technology; and zero occupation of the GPU video memory is realized.

Description

Technical Field

[0001] This invention relates to the field of large language model memory enhancement technology, and in particular to a large language model dialogue system and its usage method with token-level resident memory. Background Technology

[0002] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on large datasets. The core idea is to learn patterns and structures of natural language through large-scale unsupervised training, mimicking human language cognition and generation processes to some extent. With the widespread application of large language models, the following problems currently exist: Traditional large language models are limited by a fixed context window, typically ranging from 4K to 32K tokens. When processing long dialogues, the model cannot fully retain and utilize historical information from distant sources, leading to irrelevant answers and information forgetting, severely impacting user experience. To expand the context, existing solutions often use vector knowledge bases to store historical dialogues. However, this design requires loading vector indexes and cached data into GPU memory. Storing historical information from multiple rounds of dialogue requires significant GPU memory. In high-concurrency scenarios, GPU memory requirements increase linearly, resulting in extremely high deployment costs. Summary of the Invention

[0003] To address the technical problems existing in the background art, this invention proposes a large language model dialogue system with token-level resident memory and its usage method.

[0004] The present invention proposes a large language model dialogue system with token-level resident memory.

[0005] Preferably, in the large language model module, when initiating a token-level targeted query, the large language model generates a query request for a specific token based on the current dialogue token sequence, and then sends the query request for the specific token to the independent memory model module, thus initiating a token-level targeted query.

[0006] Preferably, in the independent memory model module, in response to the token-level targeted query, after reading and quantizing the relevant token-level historical dialogue information from the external storage module, the vectorized historical dialogue information containing the specific token is transmitted to the large language model module, as follows: It accepts query requests for specific tokens and retrieves token-level historical conversation information containing those specific tokens from external storage modules (such as PCIe-SSDs). As an explanation, token-level historical dialogue information containing a specific token is stored in the external storage module using tokens as the basic unit; Token-level historical dialogue information containing specific tokens is converted into vector representations so that the large language model module can directly understand and process it; The independent memory model module transmits the vectorized historical dialogue information containing specific tokens to the large language model module.

[0007] The usage of a large language model dialogue system with token-level residency memory includes the following steps: S1. The inference engine module receives the current dialogue content input by the user and sends the current dialogue content to the large language model module in the form of the current dialogue token sequence. S2. The large language model module receives the current dialogue token sequence and determines whether historical dialogue information is needed. S3. When historical dialogue information is needed, a token-level targeted query is initiated to the independent memory model module; the independent memory model module and all the historical dialogue data it stores reside in the external storage module and are accessed through memory mapping technology. S4. The independent memory model module reads and quantizes the relevant token-level historical dialogue information from the external storage module according to the token-level targeted query, and then transmits the vectorized historical dialogue information containing the specific token to the large language model module. S5. The large language model module receives the historical dialogue information containing a specific token returned by the independent memory model module, integrates the historical dialogue information containing the specific token with the current dialogue token sequence, generates the final response token sequence, and transmits the final response token sequence to the inference engine module, which then outputs the final response token sequence.

[0008] The token-level resident memory large language model dialogue system and its usage method proposed in this invention have the following beneficial technical effects: Upon receiving the current dialogue token sequence, this application determines whether historical dialogue information is needed through the large language model module. If so, it retrieves it from the external storage module through the independent memory model module, making the information displayed in the final response token sequence more complete. This alleviates the problem of the model being unable to fully retain and utilize historical information from afar, leading to irrelevant answers and information forgetting. This application sets up an external storage module and uses memory mapping technology to reside the model parameters of the independent memory model module and all historical dialogue token-level data. The memory mapping technology maps the model parameters of the independent memory model module and all historical dialogue token-level data to the address space that the operating system can directly address, i.e., the CPU's directly addressable space. This allows the program to access them directly like ordinary memory without performing cumbersome loading or removing operations from GPU memory. Since the model parameters of the independent memory model module and all historical dialogue token-level data are never loaded into GPU memory, this application alleviates the problem of the model being unable to fully retain and utilize historical information from afar, leading to irrelevant answers and information forgetting, while achieving zero GPU memory usage. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the system of the present invention; Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0010] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0011] like Figure 1 The large language model dialogue system with token-level resident memory shown includes an inference engine module, a large language model module, an independent memory model module, and an external storage module; The inference engine module is used to receive the current dialogue content input by the user and send the current dialogue content to the large language model module in the form of the current dialogue token sequence. The inference engine module is used to output the final response token sequence transmitted by the large language model module. The large language model module is connected to the inference engine module. The large language model module includes a large language model and a judgment and scheduling unit. The large language model module is used to perform the following operations: receive the current dialogue token sequence through the large language model; through the judgment and scheduling unit, during the response generation process of the large language model, determine whether historical dialogue information is needed, and if historical dialogue information is needed, initiate a token-level targeted query to the independent memory model module; receive historical dialogue information containing specific tokens returned by the independent memory model module through the large language model, integrate the historical dialogue information containing specific tokens with the current dialogue token sequence to generate the final response token sequence, and transmit the final response token sequence to the inference engine module, which then outputs the final response token sequence. An independent memory model module, connected to the large language model module, is used to respond to the token-level targeted query, read from the external storage module and quantize the relevant token-level historical dialogue information, and then transmit the vectorized historical dialogue information containing the specific token to the large language model module. An external storage module, using PCIe-SSD, is used to store the model parameters and token-level data of all historical dialogues of the independent memory model module through memory mapping technology, so that the system's GPU memory usage is zero during operation.

[0012] When the system of this application starts, the external storage module maps the model parameters of the stored independent memory model module to the memory address space through memory mapping technology, thereby completing the loading and readiness of the model; The system described in this application can be deployed on a hardware platform that includes a GPU, a PCIe 4.0 or higher SSD, and a Linux operating environment; For clarification, in PCIe-SSD, SSD stands for Solid State Drive, and PCIe stands for Peripheral Component Interconnect, a high-speed serial computer expansion bus standard on the motherboard. It is the main channel in modern computers connecting high-performance peripherals such as graphics cards, network cards, and storage devices to the CPU and memory; PCIe-SSD refers to the existing technology of directly connecting the flash memory storage units of SSDs to the system via the PCIe bus. As an illustration, the system of this application stores the model parameters of the independent memory model module and the token-level data of all historical dialogues on a PCIe-SSD. Through memory mapping technology, the model parameters of the independent memory model module and the token-level data of all historical dialogues are mapped to the CPU's directly addressable space, i.e., the mapping region. When memory retrieval is required, the CPU directly accesses and manipulates the data located in the mapping region, completes vectorized calculations, and returns the results to the large language model module on the GPU. Since the model parameters of the independent memory model module and the token-level data of all historical dialogues are never loaded into the GPU memory, this system achieves partial zero GPU memory usage. Resident means that the model parameters of the independent memory model module and the token-level data of all historical dialogues are not only stored on the external storage module, but the system also uses memory mapping technology to map the model parameters of the independent memory model module and the token-level data of all historical dialogues to the address space that the operating system can directly address, that is, the CPU's directly addressable space. This allows the program to access them directly like accessing ordinary memory, without having to perform cumbersome operations of loading into or removing from video memory.

[0013] In an optional embodiment, when a token-level targeted query is initiated in the large language model module, the large language model generates a query request for a specific token based on the current dialogue token sequence, and then sends the query request for the specific token to the independent memory model module, thus initiating a token-level targeted query.

[0014] In an optional embodiment, the independent memory model module, in response to the token-level targeted query, reads and quantizes the relevant token-level historical dialogue information from the external storage module, and then transmits the vectorized historical dialogue information containing the specific token to the large language model module, as follows: It accepts query requests for specific tokens and retrieves token-level historical conversation information containing those specific tokens from external storage modules (such as PCIe-SSDs). As an explanation, token-level historical dialogue information containing a specific token is stored in the external storage module using tokens as the basic unit; Token-level historical dialogue information containing specific tokens is converted into vector representations so that the large language model module can directly understand and process it; The independent memory model module transmits the vectorized historical dialogue information containing specific tokens to the large language model module.

[0015] like Figure 2The usage method of the large language model dialogue system with token-level residency memory shown includes the following steps: S1. The inference engine module receives the current dialogue content input by the user and sends the current dialogue content to the large language model module in the form of the current dialogue token sequence. S2. The large language model module receives the current dialogue token sequence and determines whether historical dialogue information is needed. S3. When historical dialogue information is needed, a token-level targeted query is initiated to the independent memory model module; the independent memory model module and all the historical dialogue data it stores reside in the external storage module and are accessed through memory mapping technology. S4. The independent memory model module reads and quantizes the relevant token-level historical dialogue information from the external storage module according to the token-level targeted query, and then transmits the vectorized historical dialogue information containing the specific token to the large language model module. S5. The large language model module receives the historical dialogue information containing a specific token returned by the independent memory model module, integrates the historical dialogue information containing the specific token with the current dialogue token sequence, generates the final response token sequence, and transmits the final response token sequence to the inference engine module, which then outputs the final response token sequence.

[0016] Upon receiving the current dialogue token sequence, this application determines whether historical dialogue information is needed through the large language model module. If so, it retrieves it from the external storage module through the independent memory model module, making the information displayed in the final response token sequence more complete. This alleviates the problem of the model being unable to fully retain and utilize historical information from afar, leading to irrelevant answers and information forgetting. This application sets up an external storage module and uses memory mapping technology to reside the model parameters of the independent memory model module and all historical dialogue token-level data. The memory mapping technology maps the model parameters of the independent memory model module and all historical dialogue token-level data to the address space that the operating system can directly address, i.e., the CPU's directly addressable space. This allows the program to access them directly like ordinary memory without performing cumbersome loading or removing operations from GPU memory. Since the model parameters of the independent memory model module and all historical dialogue token-level data are never loaded into GPU memory, this application alleviates the problem of the model being unable to fully retain and utilize historical information from afar, leading to irrelevant answers and information forgetting, while achieving zero GPU memory usage.

[0017] For clarification, "acquisition" in this application refers to obtaining the required content or data using existing technical means.

[0018] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0019] In the embodiments provided by this invention, it should be understood that the disclosed system or method can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for instance, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.

[0020] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0021] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.

[0022] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the basic characteristics of the present invention.

[0023] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A large language model dialogue system with token-level resident memory, characterized in that, It includes an inference engine module, a large language model module, an independent memory model module, and an external storage module; The inference engine module is used to receive the current dialogue content input by the user and send the current dialogue content to the large language model module in the form of the current dialogue token sequence. The inference engine module is used to output the final response token sequence transmitted by the large language model module. The large language model module includes a large language model and a decision-scheduling unit. The large language model module is used to perform the following operations: receive the current dialogue token sequence through the large language model; through the decision-scheduling unit, during the response generation process of the large language model, determine whether historical dialogue information is needed, and if historical dialogue information is needed, initiate a token-level targeted query to the independent memory model module; through the large language model, receive the historical dialogue information containing specific tokens returned by the independent memory model module, integrate the historical dialogue information containing specific tokens with the current dialogue token sequence to generate the final response token sequence, and transmit the final response token sequence to the inference engine module, which then outputs the final response token sequence. The independent memory model module is used to respond to the token-level targeted query. After reading and quantizing the relevant token-level historical dialogue information from the external storage module, it transmits the vectorized historical dialogue information containing the specific token to the large language model module. The external storage module, using PCIe-SSD, is used to store the model parameters and token-level data of all historical dialogues of the independent memory model module through memory mapping technology.

2. The large language model dialogue system and its usage method with token-level resident memory as described in claim 1, characterized in that, In the large language model module, when a token-level targeted query is initiated, the large language model generates a query request for a specific token based on the current dialogue token sequence, and then sends the query request for the specific token to the independent memory model module.

3. The large language model dialogue system and its usage method with token-level resident memory as described in claim 2, characterized in that, In the independent memory model module, in response to the token-level targeted query, it reads and quantizes the relevant token-level historical dialogue information from the external storage module, and then transmits the vectorized historical dialogue information containing the specific token to the large language model module, as follows: Accept query requests for a specific token and retrieve token-level historical dialogue information containing that specific token from the external storage module. Convert token-level historical dialogue information containing a specific token into a vector representation; The independent memory model module transmits the vectorized historical dialogue information containing specific tokens to the large language model module.

4. A method for using a large language model dialogue system with token-level residency memory, for using the large language model dialogue system with token-level residency memory as described in any one of claims 1 to 3, characterized in that, Includes the following steps: Includes the following steps: S1. The inference engine module receives the current dialogue content input by the user and sends the current dialogue content to the large language model module in the form of the current dialogue token sequence. S2. The large language model module receives the current dialogue token sequence and determines whether historical dialogue information is needed. S3. When historical dialogue information is needed, a token-level targeted query is initiated to the independent memory model module; the independent memory model module and all the historical dialogue data it stores reside in the external storage module and are accessed through memory mapping technology. S4. The independent memory model module reads and quantizes the relevant token-level historical dialogue information from the external storage module according to the token-level targeted query, and then transmits the vectorized historical dialogue information containing the specific token to the large language model module. S5. The large language model module receives the historical dialogue information containing a specific token returned by the independent memory model module, integrates the historical dialogue information containing the specific token with the current dialogue token sequence, generates the final response token sequence, and transmits the final response token sequence to the inference engine module, which then outputs the final response token sequence.

Images