Token compression method and device for large model dialogue

Abstract

The application provides a Token compression method and device for large model dialogue, which can effectively reduce the Token (word element) number of text input into the large model in multi-round dialogue, while also retaining the key information of the early part of the historical dialogue text. The method is applied to a Token compression management agent, which comprises the following steps: after obtaining the question text of a user input terminal, determining the number of word elements in the initial text (containing historical dialogue text and question text) to be sent to the large model. If the number of word elements in the initial text does not meet the threshold requirement, the state is switched to the word element compression state, and the number of word elements in the compressed intermediate dialogue text (part of the historical dialogue text except the first k rounds of dialogue text, k>0) is obtained by interacting with the large model, and the generated summary text is retained. The first k rounds of dialogue text, the summary text and the question text are spliced to obtain a target text. If the number of word elements in the target text meets the threshold requirement, it is sent to the first large model.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to a token compression method and apparatus for large-scale model dialogue. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, especially the increasingly widespread application of large models, their performance in fields such as natural language processing, code generation and understanding, and intelligent question answering has significantly improved the efficiency of software development and user experience. Due to the limitations of large model architecture design, computational complexity, and storage and computing resources, they typically set a maximum token number threshold for input text; a token is the basic unit of text.

[0003] Dialogue applications based on large models generally require dialogue systems with multi-turn dialogue capabilities. On one hand, this multi-turn dialogue capability makes large models more natural in human-to-human conversations, aligning more closely with human communication habits using natural language. On the other hand, the multi-turn dialogue format allows large models to understand context, capture keywords and nuances in the dialogue, enhance their ability to recognize user intent, correct misunderstandings, and extract key points from the dialogue. However, because dialogue systems typically combine all questions and answers from each turn as input for the new turn, the accumulated number of input tokens increases significantly, leading to historical data inflation. In long dialogues, the accumulated token count may exceed the large model's maximum token count threshold, causing the large model to be unable to effectively process subsequent dialogues, severely impacting the performance of the dialogue system and the user experience.

[0004] Therefore, there is an urgent need for a token compression method for multi-turn dialogues to avoid the accumulated token count exceeding the maximum token count threshold of a large model during long dialogues, thereby optimizing the operating efficiency of the dialogue system. Summary of the Invention

[0005] This application provides a token compression method and apparatus for large model dialogue, which can effectively reduce the number of tokens in the text input to the large model in multi-turn dialogue, while retaining key information from the early parts of the historical dialogue text.

[0006] Firstly, a token compression method for large-scale model dialogue is provided, applied to a token compression management agent, including:

[0007] After obtaining the question text from the user's input terminal, determine the number of tokens in the initial text to be sent to the first large model. The initial text includes historical dialogue text and question text.

[0008] If the number of tokens in the initial text is greater than or equal to the first preset threshold, the state is switched to token compression state. By interacting with the second large model, the summary text of the intermediate dialogue text is obtained. The intermediate dialogue text is the part of the historical dialogue text excluding the first k rounds of dialogue text, where k is an integer greater than or equal to 0. The summary text is generated by the second large model by compressing the number of tokens in the intermediate dialogue text while preserving the semantics of the intermediate dialogue text.

[0009] The text of the first k rounds of dialogue, the summary text, and the question text are concatenated to obtain the target text after token compression of the initial text;

[0010] If the number of tokens in the target text is less than the first preset threshold, send the target text to the first large model and switch the state to non-token compression state.

[0011] In one feasible design, the historical dialogue text includes q rounds of dialogue text, where q is an integer greater than or equal to 0. If the number of tokens in the initial text is greater than or equal to a first preset threshold, a summary text of the intermediate dialogue text is obtained through interaction with the second large model, including:

[0012] If 1≤q≤k and the number of tokens in the initial text is greater than or equal to the first preset threshold, after determining the value of k as q-1, the summary text of the intermediate dialogue text is obtained by interacting with the second largest model based on the latest k.

[0013] After concatenating the text of the first k rounds of dialogue, the summary text, and the question text to obtain the target text after token compression of the initial text, it also includes:

[0014] If the number of tokens in the target text is greater than or equal to the first preset threshold, the value of k is reduced by 1, and then the new intermediate dialogue text is obtained by interacting with the second large model again based on the latest k. The dialogue texts from the previous k rounds, the new summary text, and the question text are concatenated to obtain a new target text after token compression of the initial text. If the number of tokens in the new target text is less than the first preset threshold, the new target text is sent to the first large model. If the number of tokens in the new target text is greater than or equal to the first preset threshold, the value of k is reduced by 1 again, and the initial text is token compressed based on the latest k to obtain a new target text again. This process continues until the number of tokens in the new target text is less than the first preset threshold, at which point the new target text is sent to the first large model.

[0015] In one feasible design, a summary text of the intermediate dialogue is obtained by interacting with a second large model, including:

[0016] Generate a first prompt word and send the first prompt word to the second large model. The first prompt word includes a first instruction information and intermediate dialogue text. The first instruction information instructs the second large model to generate intermediate summary text by compressing the number of tokens in the intermediate dialogue text while preserving the semantics of the intermediate dialogue text. The number of tokens in the intermediate summary text is less than a second preset threshold.

[0017] After receiving the intermediate summary text sent by the second largest model, the system obtains user feedback on the intermediate summary text through interaction with the user.

[0018] If the feedback indicates that the user approves the intermediate summary text, the intermediate summary text will be determined as the summary text of the intermediate dialogue text;

[0019] If the feedback indicates that the user has suggested modifications to the intermediate summary text, a second prompt word is generated and sent to the second large model. The second prompt word includes the second instruction information, the modification suggestion, and the intermediate summary text. The second instruction information instructs the second large model to adjust the intermediate summary text according to the modification suggestion, and the number of tokens in the modified intermediate summary text is less than the second preset threshold.

[0020] After receiving the adjusted intermediate summary text sent by the second largest model, the system interacts with the user to obtain new feedback from the user regarding the adjusted intermediate summary text.

[0021] If the new feedback indicates that the user approves of the adjusted intermediate summary text, the adjusted intermediate summary text will be determined as the summary text of the intermediate dialogue text.

[0022] If new feedback indicates that the user has made new suggestions for modifying the adjusted intermediate summary text, the system will interact with the second model again to adjust the previously generated intermediate summary text according to the new suggestions, until the user feedback indicates that the user approves the latest adjusted intermediate summary text. The latest adjusted intermediate summary text will then be determined as the summary text of the intermediate dialogue text.

[0023] In one feasible design, a token compression management agent is used to store the historical dialogue text between the user and the first-level model, and the method also includes:

[0024] If the feedback indicates that the user needs to regenerate the intermediate summary text, the first prompt word is regenerated based on the saved historical dialogue text and sent to the second large model;

[0025] After receiving the new intermediate summary text sent by the second major model, the system generates an intermediate summary text approved by the user through interaction with the user, and determines the user-approved intermediate summary text as the summary text of the intermediate dialogue text.

[0026] In one feasible design, a token compression management agent is used to store the intermediate summary texts generated by the second-largest model, the user feedback corresponding to each intermediate summary text, and the final summary text. If the second-largest model generates multiple intermediate summary texts, the method further includes the following steps before concatenating the first k rounds of dialogue text, summary text, and question text:

[0027] After obtaining each intermediate summary text, the user feedback corresponding to each intermediate summary text, and the final summary text, the text selected by the user from each intermediate summary text and the final summary text is redefined as the summary text of the intermediate dialogue text through interaction with the user.

[0028] In a feasible design, the method also includes:

[0029] If q = 0 and the number of tokens in the initial text is greater than or equal to the first preset threshold, the summary text of the initial text is obtained by interacting with the second large model;

[0030] The summary text of the initial text is identified as the target text and sent to the first large model.

[0031] In one feasible design, the dialogue text from the first k rounds, the summary text, and the question text are concatenated to obtain the target text after token compression of the initial text; if the number of tokens in the target text is less than a first preset threshold, the target text is sent to the first large model, including:

[0032] Generate a third instruction message, which includes the context and the user's latest question.

[0033] The third prompt word is obtained by concatenating the third instruction information, the text of the first k rounds of dialogue, the summary text, and the question text.

[0034] If the number of tokens in the third prompt word is less than the first preset threshold, the third prompt word is sent to the first large model. The third prompt word is the target text after token compression of the initial text.

[0035] In a viable design, the token compression management agent is deployed on the terminal, server, or front-end page.

[0036] Secondly, a token compression device for large-scale dialogue models is provided, applied to a token compression management agent, including:

[0037] The text acquisition module is used to determine the number of tokens in the initial text to be sent to the first large model after acquiring the question text from the user's input terminal. The initial text includes historical dialogue text and question text.

[0038] The Token compression module is used to switch the state to Token compression state if the number of tokens in the initial text is greater than or equal to the first preset threshold. By interacting with the second large model, it obtains the summary text of the intermediate dialogue text. The intermediate dialogue text is the part of the historical dialogue text excluding the dialogue text of the first k rounds, where k is an integer greater than or equal to 0. The summary text is generated by the second large model by compressing the number of tokens in the intermediate dialogue text while preserving the semantics of the intermediate dialogue text.

[0039] The Token compression module is also used to concatenate the text of the first k rounds of dialogue, the summary text, and the question text to obtain the target text after Token compression of the initial text.

[0040] The text sending module is used to send the target text to the first large model if the number of tokens in the target text is less than a first preset threshold, and switch the state to non-token compression state.

[0041] In one feasible design, the historical dialogue text includes q rounds of dialogue text, where q is an integer greater than or equal to 0. The token compression module is also used to determine the value of k as q-1 if 1≤q≤k and the number of tokens in the initial text is greater than or equal to the first preset threshold. Based on the latest k, the module interacts with the second largest model to obtain the summary text of the intermediate dialogue text.

[0042] The Token compression module is also used to concatenate the text of the first k rounds of dialogue, the summary text, and the question text to obtain the target text after Token compression of the initial text.

[0043] The text sending module is also used to send the target text to the first large model if the number of tokens in the target text is less than a first preset threshold.

[0044] The token compression module is also used to, after concatenating the first k rounds of dialogue text, summary text, and question text to obtain the target text after token compression of the initial text, if the number of tokens in the target text is greater than or equal to a first preset threshold, decrement the value of k by 1, and then, based on the latest k, interact with the second large model to obtain the summary text of the new intermediate dialogue text. Concatenate the first k rounds of dialogue text, the new summary text, and question text to obtain the new target text after token compression of the initial text. The text sending module is also used to, if the number of tokens in the new target text is less than the first preset threshold, send the new target text to the first large model. The token compression module is also used to, if the number of tokens in the new target text is greater than or equal to the first preset threshold, decrement the value of k by 1 again and perform token compression on the initial text based on the latest k to obtain the new target text again, until the number of tokens in the new target text obtained again is less than the first preset threshold. The text sending module is also used to, send the new target text to the first large model.

[0045] This application's research found that during multi-turn dialogues between users and a large model, the text of the first few rounds typically contains a relatively complete overall intent of the user, reflecting the user's topic and the context of initiating the dialogue. Based on this, the token compression management agent in this application's embodiment monitors the number of tokens in the initial text to be sent to the first large model in real time. When the number of tokens in the initial text is greater than or equal to a first preset threshold, the state is switched to token compression mode. The first k rounds of dialogue text are retained without compression, while the portion of the historical dialogue text excluding the first k rounds is compressed to generate summary text. This achieves the goal of compressing the number of tokens in the initial text while retaining the key information of the first k rounds of dialogue text, resulting in target text that meets the token quantity requirement. Compared to compressing the initial text by truncating the early parts of the historical dialogue text, this solution avoids misunderstandings in the large model due to the loss of key information, which could lead to a decline in dialogue quality and user experience, thus ensuring the coherence and accuracy of multi-turn dialogues.

[0046] In summary, the embodiments of this application, while compressing information, emphasize the preservation of complete overall user intent information. Different compression strategies are adopted according to the importance of different stages of a multi-turn dialogue, ensuring that important information from the first k turns of dialogue is not damaged during compression. This achieves the effect of protecting information from all stages of the dialogue history, avoiding the excessive damage to early dialogue information caused by simply truncating early portions of historical dialogue text. Attached Figure Description

[0047] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic flowchart illustrating an exemplary embodiment of a token compression method for large-scale dialogue provided in this application;

[0049] Figure 2 This is a schematic flowchart illustrating another example of a token compression method for large-scale dialogue provided in an exemplary embodiment of this application;

[0050] Figure 3 This is an exemplary flowchart illustrating the generation of summary text provided in an exemplary embodiment of this application;

[0051] Figure 4 This is an exemplary flowchart illustrating how the target text is determined by dynamically adjusting the value of k, as provided in an exemplary embodiment of this application.

[0052] Figure 5 This is a schematic diagram of an example of a token compression device for large-scale dialogue provided in an exemplary embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In recent years, with the rapid development of artificial intelligence technology, especially the increasingly widespread application of large-scale models, its performance in fields such as natural language processing, code generation and understanding, and intelligent question answering has significantly improved the efficiency of software development and user experience. In the field of intelligent question answering, dialogue systems based on large-scale models need to be capable of handling multi-turn dialogues. The multi-turn dialogue process of a dialogue system is as follows:

[0055] First, initialize the history chat list, then enter the chat loop state:

[0056] After the user enters a question, the dialogue system first checks if it's an exit command. If so, it terminates the process directly; otherwise, it adds the question to the history of dialogue and calls the Application Programming Interface (API). The system then checks the API response for success. If successful, it parses the response text output by the API, adds it to the history of dialogue, displays the response to the user, and waits for the user's next input. If it fails, it returns an error message and re-enters the dialogue loop.

[0057] Each time a user enters a question, the historical dialogue text and the latest entered question text need to be concatenated and sent to the large model.

[0058] However, large models face the technical challenge of token limitations when handling multi-turn dialogues. Due to limitations in their architecture, computational complexity, and storage and computing resources, large models typically have a maximum token count threshold. For example, the Generative Pre-trained Transformer-4 (GPT-4) has a maximum token count threshold of 32K. In long dialogues, the number of tokens from the input text may exceed this threshold due to the accumulation of historical dialogue text, preventing the large model from continuing to run.

[0059] To address the aforementioned issues, current dialogue systems generally employ a strategy of truncating historical dialogue text, retaining only the text from the most recent rounds to control the growth of tokens. This study finds that this method is relatively simple to implement and can alleviate the pressure of token quantity limitations to some extent. However, this strategy ignores information such as the user-defined scenario, background, core needs, and basic cognition in the early parts of the historical dialogue text. This information is crucial for understanding the context of multi-turn dialogues and the user's deeper intentions. The early parts of the historical dialogue text form an important foundation for subsequent dialogues; once truncated, the large model may be unable to accurately capture the evolution of the user's intentions, leading to biases in the model's understanding and impacting dialogue quality and user experience.

[0060] This application aims to solve the problem of unlimited token growth, while avoiding the loss of key information in important historical dialogues due to the deletion of early parts of historical dialogue text caused by truncating historical data.

[0061] Therefore, this application proposes a differentiated and weighted token compression method for different stages of historical dialogue text during multi-turn dialogues. Specifically, this application monitors the number of tokens to be sent to the large model during multi-turn dialogues. When the number of tokens to be sent to the large model is greater than or equal to a preset threshold, compression of the historical dialogue text is triggered. Since this application's research found that the dialogue text of the first few rounds in the historical dialogue text contains crucial information, the first k rounds of the historical dialogue text are preserved to protect the crucial information in the first k rounds of the dialogue text. k can be manually set based on experience. The remaining parts of the historical dialogue text are then subjected to token compression, so that the number of tokens in the compressed historical dialogue text is significantly reduced while ensuring that crucial information is not lost. This application suggests that k can be set to 5.

[0062] Based on this, this application designs a Token compression management agent to complete various functions of historical conversation text compression, so as to simplify user operations, improve compression effect, and improve user experience.

[0063] For example, the token compression management agent is deployed on a terminal, server, or front-end page.

[0064] The terminal may include, but is not limited to, mobile phones, tablets, laptops, ultra-mobile personal computers (UMPCs), handheld computers, touch-screen TVs, walkie-talkies, netbooks, point-of-sales (POS) machines, personal digital assistants (PDAs), wearable devices, virtual reality devices, wireless USB flash drives, Bluetooth speakers / headphones / glasses, in-vehicle pre-installed devices, dashcams, security equipment, and other mobile or fixed terminals.

[0065] A server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. A server can also be a server for a distributed system or a server integrated with blockchain technology. In essence, this server can be a server hosted on any website, such as Baidu Server, 360 Server, or Sogou Server.

[0066] When deployed on a terminal, the Token Compression Management Agent can be deployed as a plugin within the application's integrated development environment (IDE). After capturing the user's input of the question text from the terminal's application interface, the agent monitors the number of tokens in the initial text (composed of the question text and historical dialogue text) to be forwarded to the large model. If the number of tokens in the initial text is greater than or equal to a first preset threshold, the initial text is compressed, and the compressed target text is forwarded to the large model via an application API call; otherwise, the initial text is forwarded to the large model. Upon receiving the response text from the large model, the response text is displayed to the user through the application's interface.

[0067] When the Token compression management agent is deployed on the front-end page, it can be deployed as a plugin in the browser page. The Token compression management agent listens for page input events, captures the user's input of the question text into the browser page, and monitors the number of tokens in the initial text to be forwarded to the large model, which consists of the question text and historical dialogue text. If the number of tokens in the initial text is greater than or equal to a first preset threshold, the initial text is compressed, and the compressed target text is forwarded to the large model by calling the browser interface; otherwise, the initial text is forwarded to the large model. After receiving the reply text sent by the large model, the reply text is displayed to the user through the browser page.

[0068] When the Token Compression Management Agent is deployed on a server, it operates as an independent middleware service. User terminals send requests containing query text to the Token Compression Management Agent server via its public port. Upon receiving the request, the Token Compression Management Agent server monitors the number of tokens in the initial text (composed of the query text and historical dialogue text) to be forwarded to the large model. If the number of tokens in the initial text is greater than or equal to a first preset threshold, the initial text is tokenized, and the compressed target text is forwarded to the large model via internal inter-service communication. Otherwise, the initial text is forwarded to the large model. After receiving the reply text from the large model, a response containing the reply text is sent to the user terminal via the public port.

[0069] The first preset threshold can be set according to actual needs, and this application does not limit it. For example, it can be set according to factors such as the number of tokens required by the first major model and the requirements of the dialogue scenario.

[0070] It should be noted that, unless otherwise specified, sending target text or initial text to a large model in this application refers to sending target text or initial text to the server where the large model is deployed.

[0071] In summary, the Token compression management agent can intercept the user's input query text on the terminal, forward the initial text or target text containing the query text to the large model, and after receiving the response text from the large model, display the response text on the terminal or send the response text to the terminal.

[0072] For example, in order to facilitate the dual tasks of managing normal dialogue and token quantity compression, this application designs two system states for the token compression management agent: "non-token compression state" and "token compression state".

[0073] Among them, the non-Token compression state corresponds to the case where the number of Tokens in the initial text is less than the first preset threshold, that is, the state where the number of Tokens is not close to the upper limit of the large model and the initial text does not need to be compressed.

[0074] Typically, at the start of a multi-turn dialogue, the Token Compression Management Agent is in a non-Token Compression state. At this time, the Token Compression Management Agent monitors the number of tokens. In the non-Token Compression state, the Token Compression Management Agent adds the text of each previous dialogue turn to the dialogue history text.

[0075] The token compression state corresponds to a situation where the number of tokens in the initial text is greater than or equal to a first preset threshold. When the token compression management agent detects that the number of tokens in the initial text is greater than or equal to the first preset threshold, it triggers the token compression state.

[0076] The following is combined Figure 1 The workflow of the token compression management agent is illustrated by deploying it on a server:

[0077] After the user enters the first question text, the terminal sends the first question text to the Token Compression Management Agent. At this time, the Token Compression Management Agent is initially in a non-Token Compression state. The Token Compression Management Agent determines that the number of tokens in the first question text is less than a first preset threshold, and then forwards the first question text directly to the large model. After the large model finishes processing, it sends the generated first response text to the Token Compression Management Agent, which then forwards it to the terminal.

[0078] After several rounds of dialogue, the user enters the k-th question text, and the terminal sends the k-th question text to the Token Compression Management Agent. The Token Compression Management Agent determines that the number of tokens in the initial text is less than a first preset threshold, and that the initial text contains the k-th question text and the dialogue text from the previous k-1 rounds. The Token Compression Management Agent forwards the initial text directly to the large model. After the large model finishes processing, it sends the generated k-th response text to the Token Compression Management Agent, which then forwards it to the terminal.

[0079] After several rounds of dialogue, the user inputs the nth question text, and the terminal sends the nth question text to the Token Compression Management Agent. The Token Compression Management Agent, after determining that the number of tokens in the initial text is greater than or equal to a first preset threshold, enters the Token Compression state and performs Token compression on the initial text. The initial text contains the nth question text and the text from the previous n-1 rounds of dialogue. The Token compression of the initial text uses the aforementioned compression method of retaining the text from the first k rounds of dialogue and compressing the text from the k+1th round to the n-1th round (i.e., the intermediate dialogue text). After compressing the number of tokens in the initial text to less than the first preset threshold, the compressed initial text (i.e., the target text) is sent to the large model. The target text contains the text from the previous k rounds of dialogue, the compressed intermediate dialogue text, and the nth question text. After the large model finishes processing, it sends the generated nth reply text to the Token Compression Management Agent, which forwards it to the terminal. The Token Compression Management Agent then ends the Token Compression state and re-enters the non-Token Compression state.

[0080] The user inputs the (n+1)th question text, and the terminal sends the (n+1)th question text to the Token Compression Management Agent. The Token Compression Management Agent determines that the number of tokens in the initial text is less than a first preset threshold. The initial text includes the text of the previous k rounds of dialogue, the intermediate dialogue text after the last compression (i.e., the dialogue text from round (k+1) to round (n-1), the dialogue text of round n, and the (n+1)th question text. The Token Compression Management Agent forwards the initial text directly to the large model. After the large model finishes processing, it sends the generated (n+1)th response text to the Token Compression Management Agent, which then forwards it to the terminal.

[0081] The user inputs the (n+2)th question text, and the terminal sends the (n+2)th question text to the Token Compression Management Agent. After determining that the number of tokens in the initial text is greater than or equal to a first preset threshold, the Token Compression Management Agent enters the Token Compression state and performs Token compression on the initial text. The initial text includes the text of the first k rounds of dialogue, the previously compressed intermediate dialogue text (i.e., the dialogue text from round (k+1) to (n-1), the dialogue text of round n, the dialogue text of round n+1, and the (n+2)th question text. The Token Compression Management Agent performs Token Compression on the initial text (i.e., retaining the first k rounds of dialogue, and using the uncompressed dialogue text from round (k+1) to (n+1) stored by the Token Compression Management Agent to re-compress the dialogue text from round (k+1) to (n+1) with Tokens) until the number of tokens in the initial text is less than the first preset threshold. Then, the target text is obtained, and the Token Compression state ends, entering the non-Token Compression state. The target text includes the text of the first k rounds of dialogue, the currently compressed intermediate dialogue text (i.e., the dialogue text from round (k+1) to (n+1), and the (n+2)th question text. The Token compression management agent sends the target text to the large model. After the large model finishes processing, it sends the generated (n+2)th reply text to the Token compression management agent, which then forwards it to the terminal.

[0082] The method by which the token compression management agent processes subsequent user-inputted question texts can be deduced from the above description, and will not be elaborated further in this application until the end of this multi-round dialogue.

[0083] Unless otherwise specified, each round of dialogue text includes the user's question text and the large model's response text.

[0084] As can be seen, the Token Compression Management Agent has functions such as monitoring the number of Tokens, triggering the Token Compression state, and taking over multiple rounds of dialogue.

[0085] In this embodiment, the Token compression management agent monitors the number of tokens in the initial text to be sent to the first large model and flexibly adjusts the token compression strategy to ensure that the number of tokens in each dialogue text meets the preset threshold requirement before being forwarded to the large model, thereby optimizing dialogue processing efficiency and response speed. Furthermore, during the token compression process, the Token compression management agent retains the first k rounds of dialogue text while compressing the intermediate dialogue text. This ensures that the compressed target text meets the token quantity requirement while also preserving key information such as user-defined scenarios, backgrounds, core needs, and basic understanding from the earlier parts of the historical dialogue text. This prevents the loss of key information from causing misunderstandings in the large model, which in turn leads to a decline in dialogue quality and user experience, ensuring dialogue coherence and accuracy.

[0086] Based on the above implementation of the token compression management agent, such as Figure 2 As shown, this application provides a token compression method for large-scale model dialogue, applied to a token compression management agent, including:

[0087] S110: After obtaining the question text from the user's input terminal, determine the number of Tokens in the initial text to be sent to the first large model.

[0088] The initial text includes historical dialogue text and question text.

[0089] The network framework of the first model should be set according to actual needs, as long as it can realize dialogue function, and this application does not impose any restrictions on it.

[0090] For example, taking the deployment of a token compression management agent on a user terminal as an example, after obtaining the nth question text input by the user terminal, the number of tokens in the initial text is determined. The initial text includes the historical dialogue text from the previous n-1 rounds of dialogue, and the question text is the nth question text.

[0091] At this point, the Token compression management agent remains in its initial state, i.e., non-Token compression state, and continues to monitor the number of tokens in the initial text.

[0092] S120, if the number of tokens in the initial text is greater than or equal to the first preset threshold, the state is switched to token compression state, and the summary text of the intermediate dialogue text is obtained by interacting with the second large model.

[0093] The intermediate dialogue text is the part of the historical dialogue text excluding the first k rounds of dialogue text, where k is an integer greater than or equal to 0. The summary text is generated by the second model by compressing the number of tokens in the intermediate dialogue text while preserving the semantics of the intermediate dialogue text.

[0094] At this point, the Token Compression Management Agent enters Token Compression mode, suspends the multi-round dialogue process, and takes over the management of historical dialogue text. The Token Compression Management Agent will prompt the user that the large model is about to summarize, organize, and compress the historical dialogue text, and will return the results to the user for confirmation or correction.

[0095] For example, if the number of tokens in the initial text is less than a first preset threshold, the initial text is sent to the first large model.

[0096] In one feasible design, the summary text of the intermediate dialogue is obtained by interacting with the second-largest model in the following way:

[0097] Generate a first prompt word and send the first prompt word to the second large model. The first prompt word includes a first instruction information and intermediate dialogue text. The first instruction information instructs the second large model to generate intermediate summary text by compressing the number of tokens in the intermediate dialogue text while preserving the semantics of the intermediate dialogue text. The number of tokens in the intermediate summary text is less than a second preset threshold.

[0098] After receiving the intermediate summary text sent by the second largest model, the system obtains user feedback on the intermediate summary text through interaction with the user.

[0099] If the feedback indicates that the user approves the intermediate summary text, the intermediate summary text will be determined as the summary text of the intermediate dialogue text;

[0100] If the feedback indicates that the user has suggested modifications to the intermediate summary text, a second prompt word is generated and sent to the second large model. The second prompt word includes the second instruction information, the modification suggestion, and the intermediate summary text. The second instruction information instructs the second large model to adjust the intermediate summary text according to the modification suggestion, and the number of tokens in the modified intermediate summary text is less than the second preset threshold.

[0101] After receiving the adjusted intermediate summary text sent by the second largest model, the system interacts with the user to obtain new feedback from the user regarding the adjusted intermediate summary text.

[0102] If the new feedback indicates that the user approves of the adjusted intermediate summary text, the adjusted intermediate summary text will be determined as the summary text of the intermediate dialogue text.

[0103] If new feedback indicates that the user has made new suggestions for modifying the adjusted intermediate summary text, the system will interact with the second model again to adjust the previously generated intermediate summary text according to the new suggestions, until the user feedback indicates that the user approves the latest adjusted intermediate summary text. The latest adjusted intermediate summary text will then be determined as the summary text of the intermediate dialogue text.

[0104] The communication method between the Token compression management agent and the second major model, as well as the interaction method with the user, are related to the deployment method of the Token compression management agent. Please refer to the description in the aforementioned embodiments, which will not be repeated here.

[0105] It should be noted that if the token compression management agent is deployed on the terminal, interaction with the user refers to the agent displaying information to the user or obtaining user input through the application's interface or browser page. If the token compression management agent is deployed on the server, interaction with the user refers to the agent communicating with the terminal to achieve interaction with the user.

[0106] For example, the network framework of the second major model can be set according to actual needs, as long as it can realize the dialogue function; this application does not impose any limitations on this. The network framework of the second major model can be the same as or different from that of the first major model. The second major model and the first major model can be the same major model or they can be different major models.

[0107] For example, continuing with the user-inputted question text as the nth question text, the first instruction is: "You are required to understand and summarize the following detailed original dialogue content, and generate a dialogue summary using concise words. Compress the original dialogue while preserving its semantics, ensuring the number of tokens in the generated summary does not exceed {a second preset threshold}. The original dialogue content is {the original dialogue content}". The intermediate dialogue text consists of the dialogue text from round k+1 to round n-1. The corresponding number is appended at {a second preset threshold}, and the dialogue text from round k+1 to round n-1 is appended at {the original dialogue content} to generate the first prompt word. This first prompt word is sent to the second large model. After receiving the first prompt word, the second large model generates an intermediate summary text and sends it to the token compression management agent. The second preset threshold can be set according to the capabilities of the first large model.

[0108] After obtaining the intermediate digest text, the Token compression management agent confirms the notification text: "The number of tokens is approaching the system limit. We have summarized and reviewed your requirements. Please reply with your suggestions for improvement if there are any omissions or errors. If you agree, please reply with confirmation. The following is a summary: {Intermediate Digest Text}". The compressed intermediate digest text returned by the second major model in the previous step is then appended to the {Intermediate Digest Text} field. The Token compression management agent sends the notification text to the terminal, which displays the notification text to the user through the human-computer interaction interface.

[0109] After receiving user feedback from the human-computer interaction interface, the terminal sends it to the Token compression management agent. If the feedback includes user confirmation, the Token compression management agent determines the intermediate summary text as the summary text of the intermediate dialogue text. If the feedback includes user modification suggestions, the Token compression management agent determines the second instruction message as "You are required to adjust the {intermediate summary text} according to {modification suggestions}, and the number of tokens in the modified intermediate summary text is less than {second preset threshold}." The modification suggestions included in the feedback are appended to {modification suggestions}, and the corresponding number is appended to {second preset threshold} to generate a second prompt word, which is then sent to the second large model. After receiving the second prompt word, the second large model adjusts the intermediate summary text to obtain the adjusted intermediate summary text and sends it to the Token compression management agent. The Token compression management agent determines the notification text as "We have adjusted the summary according to your modification suggestions. Please reply with modification suggestions if there are any omissions or errors. If you agree, please reply with confirmation. The following is the adjusted summary: {intermediate summary text}." The adjusted intermediate summary text returned by the second large model is appended to {intermediate summary text}. The Token compression management agent sends the notification text to the terminal, and the terminal displays the notification text to the user through the human-computer interaction interface.

[0110] The token compression management agent interacts with the user through the terminal to obtain feedback on the adjusted intermediate summary text. If the new feedback contains user confirmation, the adjusted intermediate summary text is determined as the summary text of the intermediate dialogue text. If the new feedback contains new modification suggestions, it interacts with the second-largest model again, which adjusts the previously adjusted intermediate summary text based on the new modifications, resulting in the latest adjusted intermediate summary text. This process continues until the user's feedback contains confirmation, at which point the latest adjusted intermediate summary text is determined as the summary text of the intermediate dialogue text. The above process can be combined with... Figure 3 To understand.

[0111] The example above iteratively refines the generated summary text of the intermediate dialogue, ensuring it is accurate, complete, and concise. Furthermore, the iterative compression optimization process only ends after the user approves the token compression result, further guaranteeing the accuracy and completeness of the compressed summary text. Additionally, the token compression management agent automatically and dynamically generates prompts, eliminating the need for users to manually write prompts to compress the intermediate dialogue text. This simplifies user operations, reduces the workload of prompt engineering, and prevents end-users from getting bogged down in the complex prompt engineering process of compressing historical dialogue text.

[0112] In one feasible design, a token compression management agent is used to store the historical dialogue text between the user and the first-level model, and the method also includes:

[0113] If the feedback indicates that the user needs to regenerate the intermediate summary text, the first prompt word is regenerated based on the saved historical dialogue text and sent to the second large model;

[0114] After receiving the new intermediate summary text sent by the second major model, the system generates an intermediate summary text approved by the user through interaction with the user, and determines the user-approved intermediate summary text as the summary text of the intermediate dialogue text.

[0115] For example, if the feedback includes information indicating that the user should regenerate the summary text, the Token compression management agent determines the dialogue text from round k+1 to round n-1 in the stored historical dialogue text, then regenerates the first prompt word and sends it to the second large model. The second large model regenerates the intermediate summary text and sends it to the terminal. After obtaining the new intermediate summary text, the Token compression management agent continues to iteratively revise the intermediate summary text as shown in the aforementioned embodiment to obtain the intermediate summary text approved by the user.

[0116] In the example above, the Token compression management agent saves all original historical dialogue text records. When the user needs to regenerate the summary text, the second-largest model can regenerate a new intermediate summary text based on the saved original historical dialogue text. The final intermediate summary text approved by the user is then determined as the summary text. In other words, the Token compression management agent discards the previously generated intermediate summary text and reverts to the dialogue state when it first entered the Token compression state (i.e., correcting and restoring to the previous valid dialogue node of the sustainable dialogue), regenerating the summary text based on the original historical dialogue text. When the user is dissatisfied with the current summary, a completely new summary can be directly regenerated based on the original historical text (rather than iteratively correcting erroneous summaries), avoiding multiple correction cycles due to accumulated deviations. This ensures that each correction is based on a complete and lossless dialogue context, fundamentally preventing information loss or semantic distortion, and achieving efficient error correction with a single rollback to a trusted state. This mechanism can significantly reduce the number of interactions with the user and improve summary accuracy.

[0117] In one feasible design, a token compression management agent is used to store the intermediate summary texts generated by the second-largest model, the user feedback corresponding to each intermediate summary text, and the final summary text. If the second-largest model generates multiple intermediate summary texts, the method further includes the following steps before concatenating the first k rounds of dialogue text, summary text, and question text:

[0118] After obtaining each intermediate summary text, the user feedback corresponding to each intermediate summary text, and the final summary text, the text selected by the user from each intermediate summary text and the final summary text is redefined as the summary text of the intermediate dialogue text through interaction with the user.

[0119] In the example above, the Token compression management agent saves the compression results of all intermediate dialogue text and the corresponding user feedback, so as to provide users with a variety of compressed dialogue versions, allowing users to choose a preferred version.

[0120] S130: Concatenate the text of the first k rounds of dialogue, the summary text, and the question text to obtain the target text after token compression of the initial text.

[0121] In one feasible design, the text of the first k rounds of dialogue, the summary text, and the question text are concatenated to obtain the target text after token compression of the initial text; if the number of tokens in the target text is less than a first preset threshold, the target text is sent to the first large model:

[0122] Generate a third instruction message, which includes the context and the user's latest question.

[0123] The third prompt word is obtained by concatenating the third instruction information, the text of the first k rounds of dialogue, the summary text, and the question text.

[0124] If the number of tokens in the third prompt word is less than the first preset threshold, the third prompt word is sent to the first large model. The third prompt word is the target text after token compression of the initial text.

[0125] For example, the generated third prompt message is: "You, as (the dialogue assistant role in the large model), have had multiple rounds of dialogue with the user. The following is the original dialogue between you and the user for the first {k} rounds: {first k rounds dialogue text}. Subsequently, you had multiple more rounds of dialogue, summarized as follows: {summary text}. Based on this, the customer's new question is: {question text}. Please answer the customer's new question." The third prompt is obtained by concatenating the first k rounds of dialogue text at {first k rounds dialogue text}, concatenating the summary text at {summary text}, and concatenating the question text at {question text}. The third prompt can be considered as the target text after token compression of the initial text. If the number of tokens in the third prompt is less than a first preset threshold, the third prompt is sent to the first large model.

[0126] The above example uses third-party information to indicate the content of the context and the user's latest question, enabling the first model to accurately understand the context and efficiently capture the user's intent without needing to distinguish between contexts, thus further improving the accuracy and response speed of the first model's response.

[0127] S140, if the number of tokens in the target text is less than the first preset threshold, send the target text to the first large model and switch the state to non-token compression state.

[0128] At this point, the Token compression management agent ends the Token compression state and re-enters the non-Token compression state.

[0129] For example, the method further includes:

[0130] S150, if the number of tokens in the target text is greater than or equal to the first preset threshold, then interact with the second model again to obtain a new summary text of the intermediate dialogue text. Then, concatenate the previous k rounds of dialogue text, the new summary text, and the question text to obtain a new target text.

[0131] S160: If the number of tokens in the new target text is less than the first preset threshold, send the new target text to the first large model; otherwise, repeat S150 until the number of tokens in the new target text is less than the first preset threshold.

[0132] For example, before concatenating the previous k rounds of dialogue text, the new summary text, and the question text, a new summary text approved by the user is obtained through iterative interaction. The specific method is described in the foregoing embodiments and will not be repeated here.

[0133] The example above iteratively compresses the intermediate dialogue text to ensure that the number of tokens in the target text meets a preset threshold requirement, thus ensuring that the first model can respond to the user's question.

[0134] Assuming the historical dialogue text includes q rounds of dialogue text, where q is an integer greater than or equal to 0, this application typically sets the value of k to meet the needs of most scenarios (i.e., token compression of the initial text can be performed according to S120-S130, where q > k). However, in extreme cases, when q ≤ k, the number of tokens in the initial text may be greater than or equal to the first preset threshold. For example, when k = 5, and q = 5 (i.e., the question text is question text 6), if the number of tokens in the initial text is greater than or equal to the first preset threshold, since the intermediate dialogue text is the part of the historical dialogue text excluding the first 5 rounds of dialogue text, and the historical dialogue text contains a total of 5 rounds of dialogue text, the intermediate dialogue text does not exist in the historical dialogue text and will be set to "blank". Correspondingly, the summary text of the intermediate dialogue text is also "blank". Therefore, concatenating the first 5 rounds of dialogue text, the summary text ("blank"), and the question text results in the target text having no change in the number of tokens compared to the initial text, still greater than the first preset threshold. At this point, the Token compression management agent will perform token compression by lowering the k value until the number of tokens in the initial text meets the threshold requirement. The specific steps include:

[0135] If q = 0 and the number of tokens in the initial text is greater than or equal to the first preset threshold, the summary text of the initial text is obtained by interacting with the second large model;

[0136] The summary text of the initial text is identified as the target text and sent to the first large model.

[0137] For example, before determining the target text from the initial text's summary text, a user-approved summary text is obtained through iterative interaction with the user. Specific details are provided in the description of the foregoing embodiments and will not be repeated here.

[0138] In the example above, q=0 means there is no historical dialogue text, and the initial text is the question text. That is, the number of tokens in the user's first input question text is already greater than or equal to the first preset threshold. At this point, compression of the initial text cannot be achieved based on the initial k value. Therefore, the token compression management agent does not need to concern itself with the k value; it can directly interact with the second large model to perform token compression on the initial text, obtain its summary text, and send it as the target text to the first large model.

[0139] In one feasible design, if the number of tokens in the initial text is greater than or equal to a first preset threshold, a summary text of the intermediate dialogue text is obtained by interacting with a second large model, including:

[0140] If 1≤q≤k and the number of tokens in the initial text is greater than or equal to the first preset threshold, after determining the value of k as q-1, the summary text of the intermediate dialogue text is obtained by interacting with the second largest model based on the latest k.

[0141] After concatenating the first k rounds of dialogue text, summary text, and question text to obtain the target text after token compression of the initial text, it also includes:

[0142] If the number of tokens in the target text is greater than or equal to the first preset threshold, the value of k is reduced by 1, and then the new intermediate dialogue text is obtained by interacting with the second large model again based on the latest k. The dialogue texts from the previous k rounds, the new summary text, and the question text are concatenated to obtain a new target text after token compression of the initial text. If the number of tokens in the new target text is less than the first preset threshold, the new target text is sent to the first large model. If the number of tokens in the new target text is greater than or equal to the first preset threshold, the value of k is reduced by 1 again, and the initial text is token compressed based on the latest k to obtain a new target text again. This process continues until the number of tokens in the new target text is less than the first preset threshold, at which point the new target text is sent to the first large model.

[0143] Here, obtaining the summary text of the intermediate dialogue text by interacting with the second largest model based on the latest k means that the intermediate dialogue text is determined based on the latest k, and then the summary text is obtained by interacting with the second largest model to obtain the number of tokens of the intermediate dialogue text compressed by the second largest model.

[0144] For example, before concatenating the first k rounds of dialogue text, summary text, and question text, a summary text approved by the user is obtained through iterative interaction. The specific method is described in the foregoing embodiments and will not be repeated here.

[0145] For example Figure 4As shown, when q=4 and k=5, the number of tokens in the initial text is already greater than or equal to the first preset threshold. At this time, the historical dialogue text contains 4 rounds of dialogue text, and the question text is the 5th question text. If token compression is performed according to k=5, the intermediate dialogue text at this time is the part of the historical dialogue text excluding the first 5 rounds of dialogue text, that is, the intermediate dialogue text is "blank" (i.e., there is no intermediate dialogue text in the historical dialogue text), so the summary text of the intermediate dialogue text is also "blank". Then, concatenating the first 4 rounds of dialogue text, the summary text ("blank"), and the question text, the number of tokens in the target text is unchanged compared to the initial text, still greater than the first preset threshold. Therefore, the value of k is first adjusted to 3 (i.e., q-1) so that the intermediate dialogue text exists. Based on the latest k=3, the intermediate dialogue text is determined to be the 4th round of dialogue text in the historical dialogue text. Then, by interacting with the second large model, the summary text of the intermediate dialogue text is obtained, thereby realizing the token compression of the initial text to obtain the target text. If the number of tokens in the target text is less than the first preset threshold, the target text is sent to the first large model.

[0146] If the number of tokens in the target text is greater than or equal to the first preset threshold, it means that the compression strength of k=3 is insufficient. Therefore, k can be adjusted to 2 by performing k=k-1 (i.e., decrementing k), and the intermediate dialogue text is redefined as the third and fourth rounds of the historical dialogue text. By interacting with the second large model again, a new summary text is obtained, thus achieving token compression of the initial text to obtain a new target text. This process continues until the number of tokens in the new target text is less than the first preset threshold.

[0147] It should be noted that the termination condition is "until the number of tokens in the newly obtained target text is less than the first preset threshold" rather than "k decreases to 0". This is because when k decreases to 0, it means that the intermediate dialogue text is the entire historical dialogue text, and the second model generates a summary text based on the historical dialogue text. Then, the first 0 rounds of dialogue text (i.e., "blank"), the summary text, and the 5th question text are concatenated to obtain the target text. Since this embodiment checks whether the number of tokens in the target text meets the threshold requirement, if it does not, it will iteratively interact with the second model to generate a summary text until the number of tokens in the target text meets the threshold requirement. Therefore, even if k decreases to 0, there may still be cases where the number of tokens in the target text does not meet the threshold requirement. Therefore, this embodiment uses "the number of tokens in the target text is less than the first preset threshold" as the termination condition.

[0148] In the example above, when the number of tokens in the initial text exceeds or equals the first preset threshold (1≤q≤k), the token compression management agent dynamically adjusts the value of k to reduce the number of rounds of retained dialogue text. This allows intermediate dialogue text to be compressed, generating effective summary text, and subsequently generating target text with a token count less than the first preset threshold. In this way, the token compression management agent retains as much key information as possible from the early dialogue text while ensuring that the number of tokens in the target text input to the first model meets the requirements, thus ensuring smooth dialogue flow and a positive user experience.

[0149] This application's research found that during multi-turn dialogues between users and a large model, the text of the first few rounds typically contains a relatively complete overall intent of the user, reflecting the user's topic and the context in which the dialogue began. Based on this, the token compression management agent in this application's embodiment monitors the number of tokens in the initial text to be sent to the first large model in real time. When the number of tokens in the initial text is greater than or equal to a first preset threshold, the text of the first k rounds of dialogue is retained without compression, while the portion of the historical dialogue text excluding the first k rounds is compressed to generate a summary text. This achieves the goal of compressing the number of tokens in the initial text while retaining the key information of the first k rounds of dialogue text, resulting in target text that meets the token quantity requirement. Compared to compressing the initial text by truncating the early parts of the historical dialogue text, this solution avoids misunderstandings in the large model due to the loss of key information, which could lead to a decline in dialogue quality and user experience, thus ensuring the coherence and accuracy of multi-turn dialogues.

[0150] In summary, the embodiments of this application, while compressing information, emphasize the preservation of complete overall user intent information. Different compression strategies are adopted according to the importance of different stages of a multi-turn dialogue, ensuring that important information from the first k turns of dialogue is not damaged during compression. This achieves the effect of protecting information from all stages of the dialogue history, avoiding the excessive damage to early dialogue information caused by simply truncating early portions of historical dialogue text.

[0151] like Figure 5 As shown, this application provides a token compression device for large-scale model dialogue, applied to a token compression management agent, comprising:

[0152] The text acquisition module is used to determine the number of tokens in the initial text to be sent to the first large model after acquiring the question text from the user's input terminal. The initial text includes historical dialogue text and question text.

[0153] The Token compression module is used to switch the state to Token compression state if the number of tokens in the initial text is greater than or equal to the first preset threshold. By interacting with the second large model, it obtains the summary text of the intermediate dialogue text. The intermediate dialogue text is the part of the historical dialogue text excluding the dialogue text of the first k rounds, where k is an integer greater than or equal to 0. The summary text is generated by the second large model by compressing the number of tokens in the intermediate dialogue text while preserving the semantics of the intermediate dialogue text.

[0154] The Token compression module is also used to concatenate the text of the first k rounds of dialogue, the summary text, and the question text to obtain the target text after Token compression of the initial text.

[0155] The text sending module is used to send the target text to the first large model if the number of tokens in the target text is less than a first preset threshold, and switch the state to non-token compression state.

[0156] In one feasible design, the historical dialogue text includes q rounds of dialogue text, where q is an integer greater than or equal to 0. The token compression module is also used to determine the value of k as q-1 if 1≤q≤k and the number of tokens in the initial text is greater than or equal to the first preset threshold. Based on the latest k, the module interacts with the second largest model to obtain the summary text of the intermediate dialogue text.

[0157] The Token compression module is also used to concatenate the text of the first k rounds of dialogue, the summary text, and the question text to obtain the target text after Token compression of the initial text.

[0158] The text sending module is also used to send the target text to the first large model if the number of tokens in the target text is less than a first preset threshold.

[0159] The token compression module is also used to, after concatenating the first k rounds of dialogue text, summary text, and question text to obtain the target text after token compression of the initial text, if the number of tokens in the target text is greater than or equal to a first preset threshold, decrement the value of k by 1, and then, based on the latest k, interact with the second large model to obtain the summary text of the new intermediate dialogue text. Concatenate the first k rounds of dialogue text, the new summary text, and question text to obtain the new target text after token compression of the initial text. The text sending module is also used to, if the number of tokens in the new target text is less than the first preset threshold, send the new target text to the first large model. The token compression module is also used to, if the number of tokens in the new target text is greater than or equal to the first preset threshold, decrement the value of k by 1 again and perform token compression on the initial text based on the latest k to obtain the new target text again, until the number of tokens in the new target text obtained again is less than the first preset threshold. The text sending module is also used to, send the new target text to the first large model.

[0160] In one feasible design, the token compression module obtains a summary text of the intermediate dialogue text by interacting with the second major model in the following way:

[0161] Generate a first prompt word and send the first prompt word to the second large model. The first prompt word includes a first instruction information and intermediate dialogue text. The first instruction information instructs the second large model to generate intermediate summary text by compressing the number of tokens in the intermediate dialogue text while preserving the semantics of the intermediate dialogue text. The number of tokens in the intermediate summary text is less than a second preset threshold.

[0162] After receiving the intermediate summary text sent by the second largest model, the system obtains user feedback on the intermediate summary text through interaction with the user.

[0163] If the feedback indicates that the user approves the intermediate summary text, the intermediate summary text will be determined as the summary text of the intermediate dialogue text;

[0164] If the feedback indicates that the user has suggested modifications to the intermediate summary text, a second prompt word is generated and sent to the second large model. The second prompt word includes the second instruction information, the modification suggestion, and the intermediate summary text. The second instruction information instructs the second large model to adjust the intermediate summary text according to the modification suggestion, and the number of tokens in the modified intermediate summary text is less than the second preset threshold.

[0165] After receiving the adjusted intermediate summary text sent by the second largest model, the system interacts with the user to obtain new feedback from the user regarding the adjusted intermediate summary text.

[0166] If the new feedback indicates that the user approves of the adjusted intermediate summary text, the adjusted intermediate summary text will be determined as the summary text of the intermediate dialogue text.

[0167] If new feedback indicates that the user has made new suggestions for modifying the adjusted intermediate summary text, the system will interact with the second model again to adjust the previously generated intermediate summary text according to the new suggestions, until the user feedback indicates that the user approves the latest adjusted intermediate summary text. The latest adjusted intermediate summary text will then be determined as the summary text of the intermediate dialogue text.

[0168] In one feasible design, a token compression management agent is used to store the historical dialogue text between the user and the first model. The token compression module regenerates summary text based on the original historical dialogue text content in the following way:

[0169] If the feedback indicates that the user needs to regenerate the intermediate summary text, the first prompt word is regenerated based on the saved historical dialogue text and sent to the second large model;

[0170] After receiving the new intermediate summary text sent by the second major model, the system generates an intermediate summary text approved by the user through interaction with the user, and determines the user-approved intermediate summary text as the summary text of the intermediate dialogue text.

[0171] In a feasible design, the Token compression management agent is used to store the intermediate summary texts generated by the second-largest model, the user feedback corresponding to each intermediate summary text, and the final summary text. If the second-largest model generates multiple intermediate summary texts, before concatenating the first k rounds of dialogue text, summary text, and question text, the Token compression module provides the user with multiple compressed versions to choose from in the following way:

[0172] After obtaining each intermediate summary text, the user feedback corresponding to each intermediate summary text, and the final summary text, the text selected by the user from each intermediate summary text and the final summary text is redefined as the summary text of the intermediate dialogue text through interaction with the user.

[0173] In one feasible design, the Token compression module is also used to obtain a summary text of the initial text by interacting with the second large model if q = 0 and the number of Tokens in the initial text is greater than or equal to a first preset threshold.

[0174] The text sending module is also used to determine the target text from the summary text of the initial text and send it to the first large model.

[0175] In one feasible design, the Token compression module is also used to generate third indication information, which indicates the content of the context and the user's latest question; and to concatenate the third indication information, the previous k rounds of dialogue text, the summary text, and the question text to obtain the third prompt word.

[0176] The text sending module is also used to send the third prompt word to the first large model if the number of tokens in the third prompt word is less than the first preset threshold. The third prompt word is the target text after token compression of the initial text.

[0177] In a viable design, the token compression management agent is deployed on the terminal or server.

[0178] Other implementations and effects of the above-described device can be found in the description of the embodiments of the Token Compression Method for Large Model Dialogue, and will not be repeated here.

[0179] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0180] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0181] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0182] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0183] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0184] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A token compression method for large-scale model dialogue, characterized in that, Applications to token compression management agents include: After obtaining the question text from the user's input terminal, determine the number of Tokens in the initial text to be sent to the first large model. The initial text includes the historical dialogue text and the question text. If the number of tokens in the initial text is greater than or equal to a first preset threshold, the state is switched to token compression state. Through interaction with the second large model, a summary text of the intermediate dialogue text is obtained. The intermediate dialogue text is the portion of the historical dialogue text excluding the first k rounds of dialogue text, where k is an integer greater than or equal to 0. The summary text is generated by the second large model by compressing the number of tokens in the intermediate dialogue text while preserving its semantics. The step of obtaining the summary text of the intermediate dialogue text through interaction with the second large model includes: Generate a first prompt word and send the first prompt word to the second large model. The first prompt word includes a first instruction information and the intermediate dialogue text. The first instruction information instructs the second large model to generate intermediate summary text by compressing the number of tokens in the intermediate dialogue text while preserving the semantics of the intermediate dialogue text. The number of tokens in the intermediate summary text is less than a second preset threshold. After receiving the intermediate summary text sent by the second major model, the system obtains the user's feedback on the intermediate summary text by interacting with the user. If the feedback indicates that the user approves the intermediate summary text, the intermediate summary text is determined as the summary text of the intermediate dialogue text; If the feedback indicates that the user has suggested modifications to the intermediate summary text, a second prompt word is generated and sent to the second large model. The second prompt word includes second indication information, the suggested modifications, and the intermediate summary text. The second indication information indicates that the second large model adjusts the intermediate summary text according to the suggested modifications, and the number of tokens in the modified intermediate summary text is less than the second preset threshold. After receiving the adjusted intermediate summary text sent by the second major model, the system obtains new feedback from the user regarding the adjusted intermediate summary text by interacting with the user. If the new feedback indicates that the user approves the adjusted intermediate summary text, the adjusted intermediate summary text will be determined as the summary text of the intermediate dialogue text. If new feedback indicates that the user has made new suggestions for modification to the adjusted intermediate summary text, the intermediate summary text generated last time is adjusted again by interacting with the second large model according to the new suggestions, until the user's feedback indicates that the user approves the latest adjusted intermediate summary text, and the latest adjusted intermediate summary text is determined as the summary text of the intermediate dialogue text. The first k rounds of dialogue text, the summary text, and the question text are concatenated to obtain the target text after token compression of the initial text; If the number of tokens in the target text is less than the first preset threshold, the target text is sent to the first large model, and the state is switched to non-token compression state.

2. The method according to claim 1, characterized in that, The historical dialogue text includes q rounds of dialogue text, where q is an integer greater than or equal to 0. If the number of tokens in the initial text is greater than or equal to a first preset threshold, a summary text of the intermediate dialogue text is obtained through interaction with the second large model, including: If 1≤q≤k and the number of tokens in the initial text is greater than or equal to the first preset threshold, after determining the value of k as q-1, the summary text of the intermediate dialogue text is obtained by interacting with the second largest model based on the latest k. After concatenating the first k rounds of dialogue text, the summary text, and the question text to obtain the target text after token compression of the initial text, the method further includes: If the number of tokens in the target text is greater than or equal to the first preset threshold, the value of k is reduced by 1, and then the new intermediate dialogue text is obtained by interacting with the second large model again based on the latest k. The previous k rounds of dialogue text, the new summary text, and the question text are concatenated to obtain a new target text after token compression of the initial text. If the number of tokens in the new target text is less than the first preset threshold, the new target text is sent to the first large model. If the number of tokens in the new target text is greater than or equal to the first preset threshold, the value of k is reduced by 1 again, and the initial text is token compressed based on the latest k to obtain a new target text again. This process continues until the number of tokens in the new target text is less than the first preset threshold, at which point the new target text is sent to the first large model.

3. The method according to claim 1, characterized in that, The token compression management agent is used to store the historical dialogue text between the user and the first large model, and the method further includes: If the feedback indicates that the user needs to regenerate the intermediate summary text, the first prompt word is regenerated based on the saved historical dialogue text and sent to the second large model; After receiving the new intermediate summary text sent by the second major model, an intermediate summary text approved by the user is generated through interaction with the user, and the intermediate summary text approved by the user is determined as the summary text of the intermediate dialogue text.

4. The method according to claim 3, characterized in that, The token compression management agent is used to store the intermediate summary texts generated by the second large model, the user feedback corresponding to each intermediate summary text, and the final summary text; if the second large model generates multiple intermediate summary texts, before concatenating the first k rounds of dialogue text, the summary text, and the question text, the method further includes: After obtaining each intermediate summary text, the user feedback corresponding to each intermediate summary text, and the final summary text, the text selected by the user from each intermediate summary text and the final summary text is re-determined as the summary text of the intermediate dialogue text through interaction with the user.

5. The method according to claim 2, characterized in that, The method further includes: If q=0 and the number of tokens in the initial text is greater than or equal to the first preset threshold, the summary text of the initial text is obtained by interacting with the second large model; The summary text of the initial text is determined as the target text and sent to the first large model.

6. The method according to claim 1 or 2, characterized in that, The first k rounds of dialogue text, the summary text, and the question text are concatenated to obtain the target text after token compression of the initial text; If the number of tokens in the target text is less than the first preset threshold, the target text is sent to the first large model, including: Generate a third instruction message, which indicates the content of the context and the user's latest question; The third prompt word is obtained by concatenating the third instruction information, the text of the previous k rounds of dialogue, the summary text, and the question text. If the number of tokens in the third prompt word is less than the first preset threshold, the third prompt word is sent to the first large model. The third prompt word is the target text after token compression of the initial text.

7. The method according to claim 1 or 2, characterized in that, The token compression management agent is deployed on the terminal, server, or front-end page.

8. A token compression device for large-scale model dialogue, characterized in that, Applications to token compression management agents include: The text acquisition module is used to determine the number of tokens in the initial text to be sent to the first large model after acquiring the question text from the user's input terminal. The initial text includes historical dialogue text and the question text. The token compression module is used to switch to token compression mode if the number of tokens in the initial text is greater than or equal to a first preset threshold. It then interacts with the second large model to obtain a summary text of the intermediate dialogue text. The intermediate dialogue text is the portion of the historical dialogue text excluding the first k rounds of dialogue text, where k is an integer greater than or equal to 0. The summary text is generated by the second large model by compressing the number of tokens in the intermediate dialogue text while preserving its semantics. The step of obtaining the summary text of the intermediate dialogue text through interaction with the second large model includes: Generate a first prompt word and send the first prompt word to the second large model. The first prompt word includes a first instruction information and the intermediate dialogue text. The first instruction information instructs the second large model to generate intermediate summary text by compressing the number of tokens in the intermediate dialogue text while preserving the semantics of the intermediate dialogue text. The number of tokens in the intermediate summary text is less than a second preset threshold. After receiving the intermediate summary text sent by the second major model, the system obtains the user's feedback on the intermediate summary text by interacting with the user. If the feedback indicates that the user approves the intermediate summary text, the intermediate summary text is determined as the summary text of the intermediate dialogue text; If the feedback indicates that the user has suggested modifications to the intermediate summary text, a second prompt word is generated and sent to the second large model. The second prompt word includes second indication information, the suggested modifications, and the intermediate summary text. The second indication information indicates that the second large model adjusts the intermediate summary text according to the suggested modifications, and the number of tokens in the modified intermediate summary text is less than the second preset threshold. After receiving the adjusted intermediate summary text sent by the second major model, the system obtains new feedback from the user regarding the adjusted intermediate summary text by interacting with the user. If the new feedback indicates that the user approves the adjusted intermediate summary text, the adjusted intermediate summary text will be determined as the summary text of the intermediate dialogue text. If new feedback indicates that the user has made new suggestions for modification to the adjusted intermediate summary text, the intermediate summary text generated last time is adjusted again by interacting with the second large model according to the new suggestions, until the user's feedback indicates that the user approves the latest adjusted intermediate summary text, and the latest adjusted intermediate summary text is determined as the summary text of the intermediate dialogue text. The Token compression module is also used to concatenate the first k rounds of dialogue text, the summary text, and the question text to obtain the target text after Token compression of the initial text; The text sending module is used to send the target text to the first large model if the number of tokens in the target text is less than the first preset threshold, and to switch the state to a non-token compression state.

9. The apparatus according to claim 8, characterized in that, The historical dialogue text includes q rounds of dialogue text, where q is an integer greater than or equal to 0. The token compression module is further used to determine the value of k as q-1 if 1≤q≤k and the number of tokens in the initial text is greater than or equal to the first preset threshold, and then obtain the summary text of the intermediate dialogue text by interacting with the second large model based on the latest k. The token compression module is further configured to, after concatenating the first k rounds of dialogue text, the summary text, and the question text to obtain the target text after token compression of the initial text, if the number of tokens in the target text is greater than or equal to the first preset threshold, decrement the value of k by 1, and then, based on the latest k, interact with the second large model again to obtain a new summary text of the intermediate dialogue text. The first k rounds of dialogue text, the new summary text, and the question text are then concatenated to obtain a new target text after token compression of the initial text. The text sending module is further configured to, if the number of tokens in the new target text is less than the first preset threshold, send the new target text to the first large model. The token compression module is further configured to, if the number of tokens in the new target text is greater than or equal to the first preset threshold, decrement the value of k by 1 again and perform token compression on the initial text based on the latest k to obtain a new target text again, until the number of tokens in the newly obtained target text is less than the first preset threshold. The text sending module is further configured to, send the new target text to the first large model.

Images