Session method and device, computer equipment, storage medium and program product

Abstract

The invention discloses a session method and device, computer equipment, a storage medium and a program product, and relates to the field of a data processing technology, a text generation technology, a large model technology and a large language model.The method comprises the steps that current question content of an interaction object and terminal equipment is obtained; determining at least part of target session content associated with the current question content in the historical session content; based on a first text sequence corresponding to the historical session content, a second text sequence corresponding to the target session content is determined, and the second text sequence is a sequence containing input text units arranged according to a word order; and inputting the second text sequence into the target language model to obtain reply content corresponding to the current question content, and displaying the reply content through the terminal device. According to the method and the device, the session content input into the language model can be managed, so that the reasoning accuracy and the answering speed of the language model in the edge intelligent equipment are improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of data processing technology, text generation technology, large model technology, and large language model technology, and specifically to conversation methods, devices, computer equipment, storage media, and program products. Background Art

[0002] In the use scenario of edge smart devices, the language model is usually deployed on an edge box or edge server or edge mobile device with certain computing resources, such as the smart voice assistant in the smart cockpit. In this use scenario, there will only be one or a few specific interaction objects to use the edge smart device. Therefore, when the interaction object has a conversation with the edge smart device, a user will usually accumulate multiple rounds of conversations, and there may be strong correlations between these multiple conversations.

[0003] However, when the context length of multiple rounds of conversation exceeds the context length threshold supported by the language model, the multiple rounds of conversation need to be truncated and only the latest conversation history is retained. The lack of context causes the language model to be unable to understand the content of the current conversation well or causes deviations in understanding. At the same time, it may also take more time to understand the meaning of the current conversation content and make a response, affecting the user experience. Summary of the invention

[0004] In view of this, the present disclosure provides a conversation method, apparatus, computer device, storage medium and program product to solve the problem of context loss in truncated multi-round conversations, which causes the language model to be unable to understand the content of the current conversation well or causes deviation in understanding.

[0005] In a first aspect, the present disclosure provides a conversation method, the method comprising:

[0006] Obtaining the current question content of the interactive object and the terminal device;

[0007] Determining at least part of the target conversation content associated with the current question content in the historical conversation content, wherein the target conversation content includes the question content and the reply content of the terminal device;

[0008] Determining a second text sequence corresponding to the target conversation content based on the first text sequence corresponding to the historical conversation content, wherein the second text sequence is a sequence including input text units arranged in word order;

[0009] The second text sequence is input into the target language model to obtain the reply content corresponding to the current question content, and the reply content is displayed through the terminal device.

[0010] In a second aspect, the present disclosure provides a conversation device, the device comprising:

[0011] An acquisition module, used to acquire the current question content of the interactive object and the terminal device;

[0012] A first determination module is used to determine at least part of the target conversation content associated with the current question content in the historical conversation content, wherein the target conversation content includes the question content and the reply content of the terminal device;

[0013] A second determination module, configured to determine a second text sequence corresponding to the target conversation content based on the first text sequence corresponding to the historical conversation content, wherein the second text sequence is a sequence of input text units arranged in word order;

[0014] An input module is used to input the second text sequence into a target language model, obtain a reply content corresponding to the current question content, and display the reply content through the terminal device.

[0015] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the conversation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0016] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the conversation method of the first aspect or any corresponding embodiment thereof.

[0017] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the conversation method of the first aspect or any corresponding embodiment thereof.

[0018] In an embodiment of the present disclosure, first, the current question content of the interactive object and the smart terminal can be obtained, and at least part of the target conversation content associated with the current question content can be determined in the historical conversation content, wherein the target conversation content includes the question content and the reply content of the terminal device. Next, based on the first text sequence corresponding to the historical conversation content, the second text sequence corresponding to the target conversation content can be determined, wherein the second text sequence is a sequence containing input text units arranged in word order. Then, the second text sequence is input into the target language model to obtain the reply content corresponding to the current question content, and the reply content is displayed through the terminal device, thereby managing the conversation content input into the language model to improve the reasoning accuracy and answering speed of the language model in the edge smart device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is a schematic diagram of a usage scenario of an edge language model corresponding to an edge smart device according to an embodiment of the present disclosure;

[0021] Figure 2 is a schematic diagram of caching the same system prompt words according to an embodiment of the present disclosure;

[0022] Figure 3 is a flow chart of a conversation method according to an embodiment of the present disclosure;

[0023] Figure 4 is a schematic diagram of a language model reasoning process according to an embodiment of the present disclosure;

[0024] Figure 5 is a schematic diagram of determining target conversation content according to an embodiment of the present disclosure;

[0025] Figure 6 is a schematic diagram of a user message input into a language model according to an embodiment of the present disclosure;

[0026] Figure 7 is a flow chart for determining output data according to an embodiment of the present disclosure;

[0027] Figure 8 is another flow chart for determining output data according to an embodiment of the present disclosure;

[0028] Fig. 9 is another flow chart of determining an output result according to an embodiment of the present disclosure;

[0029] Fig.10 is a schematic diagram of a Position ID and a token ID list corresponding to an output result according to an embodiment of the present disclosure;

[0030] Fig.11 It is a schematic diagram of a process of managing a KV cache corresponding to historical session content according to an embodiment of the present disclosure;

[0031] Fig.12 is a schematic diagram of a process of managing the conversation content of a target interactive object according to an embodiment of the present disclosure;

[0032] Fig.13 is a flow chart of another conversation method according to an embodiment of the present disclosure;

[0033] Fig.14 is a structural block diagram of a conversation device according to an embodiment of the present disclosure;

[0034] Fig.15 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0036] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0037] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0038] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0039] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0040] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0041] In conjunction with the application scenarios on which the execution of the session method depends, the application scenarios are described here.

[0042] In the use scenario of edge smart devices, the language model is usually deployed on an edge box or edge server or edge mobile device with certain computing resources, such as the smart voice assistant in the smart cockpit. In this use scenario, there will only be one or a few specific interaction objects to use the edge smart device. Therefore, when the interaction object has a conversation with the edge smart device, a user will usually accumulate multiple rounds of conversations, and there may be strong correlations between these multiple conversations.

[0043] However, when the context length of multiple rounds of conversation exceeds the context length threshold supported by the language model, the multiple rounds of conversation need to be truncated and only the latest conversation history is retained. The lack of context causes the language model to be unable to understand the content of the current conversation well or causes deviations in understanding. At the same time, it may also take more time to understand the meaning of the current conversation content and make a response, affecting the user experience.

[0044] Here, we take the large language model (LLM) as an example. The message input to the large language model usually includes three parts: system prompt words (also called system message), user message (also called user message) and LLM's response (also called assistant message). Here, the system prompt words are often used as the role setting of the LLM, describing the skills and some precautions of the role, etc. The user message is the message that the actual user hopes LLM to respond to, and the LLM's response is the response generated by LLM based on the user's input and the system prompt words. The following is an example of question and answer between a user and a large language model:

[0045] [System prompt words]

[0046] ##Role

[0047] You are an intelligent historical knowledge assistant. You have rich historical knowledge and are familiar with five thousand years of Chinese history.

[0048] Skills and Tools

[0049] 1. You can use the web search tool

[0050] 2. You can use the draw picture tool

[0051] ##Notes

[0052] You can only answer history-related questions and refuse to answer other irrelevant questions.

[0053] [User Question]

[0054] How many emperors were there in the Tang Dynasty?

[0055] [LLM’s response]

[0056] According to historical records, there were 21 emperors of the Tang Dynasty, starting with Emperor Gaozu of Tang, Li Yuan, and ending with Emperor Zhaozong of Tang, Li Ye.

[0057] Typically, users will have multiple rounds of conversations with LLM, and multiple records will be generated. When asking new questions, the old message records including the system prompt words will be input into LLM. The following is an example of multiple rounds of questions and answers between a user and a large language model:

[0058] ##Role

[0059] You are an intelligent historical knowledge assistant. You have rich historical knowledge and are familiar with five thousand years of Chinese history.

[0060] Skills and Tools

[0061] 1. You can use the web search tool

[0062] 2. You can use the draw picture tool

[0063] ##Notes

[0064] You can only answer history-related questions and refuse to answer other irrelevant questions.

[0065] [User Question 1]

[0066] How many emperors were there in the Tang Dynasty?

[0067] [LLM's response 1]

[0068] According to historical records, there were 21 emperors of the Tang Dynasty, starting with Emperor Gaozu of Tang, Li Yuan, and ending with Emperor Zhaozong of Tang, Li Ye.

[0069] [User Question 2]

[0070] How many sons did Emperor Gaozu of Tang Li Yuan have?

[0071] [LLM's response 2]

[0072] Emperor Gaozu of Tang, Li Yuan, had 4 sons

[0073] It should be understood that in the LLM reasoning process, the speed of generating each new token is increased by caching the keys and values ​​of the previous token. This is a typical space-for-time approach. Specifically, every time a new token comes, new keys and values ​​are calculated and concatenated into the cached keys and values. In this way, after each new query token comes, there is no need to recalculate the attention, but only focus on calculating the attention of the new token, thereby reducing the computational complexity.

[0074] However, considering that LLM is usually deployed in the central cloud service node, and the central cloud service node will serve a large number of different users, if the keys and values ​​of all users' message histories are to be cached, a large amount of GPU memory will be consumed. The capacity of GPU memory is usually limited and very expensive, which makes this caching solution too expensive and unfeasible.

[0075] Therefore, when a new conversation is input to LLM, LLM will re-execute the pre-fill and decode phases. In other words, the KV cache (i.e., the cache of keys and values) generated by the previous message conversation will not be reused. For example, when the user enters the first message, LLM will first perform the pre-fill phase, then generate the KV cache, and then perform the decoding phase until the current conversation ends. When the first message is answered, LLM will usually release all KVCaches and then pre-fill and decode the second message.

[0076] Then, different from the LLM scenarios in the cloud involved in the above solutions, such as Figure 1 The figure shows the usage scenario of edge LLM corresponding to edge smart devices, where edge LLM is usually deployed on an edge box or edge server or edge mobile device with a certain computing power. Especially when using edge boxes and edge mobile devices, because the edge computing power is limited, usually only one specific or a relatively small number of users will use the LLM deployed at the edge, that is, there will only be a specific user or a small number of users' message history. Therefore, in the edge scenario, it becomes possible to cache and manage the KV cache in the edge LLM in multiple rounds of conversations. For example, an intelligent voice assistant in a smart cockpit, in which the cockpit LLM deployed in the car will only receive information from the single user of this voice assistant.

[0077] In the relevant edge LLM usage scenarios, caching the same system prompt words can improve some of the reasoning performance, such as Figure 2 As shown, when the system prompt words of the request are the same, the reasoning performance can be improved to a certain extent.

[0078] In edge scenarios, users usually continue to accumulate context from multiple rounds of conversations, resulting in limited performance improvements in the system prompt word cache. When the context length of multiple rounds of conversations exceeds the context length of LLM, the user's conversation needs to be truncated to retain only the latest conversation history. In some cases, LLM will truncate the front part of the system prompt word, causing LLM to fail to accurately understand the user's query intent, resulting in irrelevant or inaccurate answers, which in turn leads to a decline in the overall performance of the model, including response time and accuracy.

[0079] Based on this, an embodiment of the present disclosure provides a conversation method, which can first obtain the current question content of the interactive object and the smart terminal, and determine at least part of the target conversation content associated with the current question content in the historical conversation content, wherein the target conversation content includes the question content and the reply content of the terminal device. Next, based on the first text sequence corresponding to the historical conversation content, the second text sequence corresponding to the target conversation content can be determined, wherein the second text sequence is a sequence containing input text units arranged in word order. Then, the second text sequence is input into the target language model to obtain the reply content corresponding to the current question content, and the reply content is displayed through the terminal device, so as to manage the conversation content input into the language model to improve the reasoning accuracy and answering speed of the language model in the edge smart device.

[0080] According to an embodiment of the present disclosure, a conversation method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0081] According to an embodiment of the present disclosure, a conversation method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0082] In this embodiment, a conversation method is provided, which can be used for the above-mentioned smart devices, such as edge devices such as smart voice assistants in smart cockpits. Figure 3 is a flow chart of a conversation method according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:

[0083] Step S301, obtaining the current question content of the interactive object and the terminal device.

[0084] In the embodiments of the present disclosure, the terminal device may be the above-mentioned edge device, such as the intelligent voice assistant of the intelligent cockpit. In addition, the current question content may include the question currently initiated by the interactive object to the intelligent terminal. For example, the question may be input by the interactive object through voice or the display screen of the edge device. The present disclosure does not specifically limit the method for obtaining the current question content, which is subject to what can be implemented.

[0085] Step S302: determining at least part of the target conversation content associated with the current question content in the historical conversation content, wherein the target conversation content includes the question content and the reply content of the terminal device.

[0086] In the disclosed embodiment, the historical conversation content may include N conversation contents between the interactive object and the edge device, and the N conversation contents may include multiple topics. For example, the topic of the conversation content between the interactive object and the intelligent voice assistant at the beginning is work, while the topic of the current question content is where to eat. Here, the target conversation content may be at least part or all of the historical conversation content that is related to the topic discussed in the current question content.

[0087] Specifically, firstly, feature extraction can be performed on the historical conversation content and the current question content to obtain text vectors, and the similarity between the text vectors can be calculated by similarity calculation, so that the historical conversation content whose similarity with the current question content exceeds a threshold is determined as the target conversation content.

[0088] Step S303: determining a second text sequence corresponding to the target conversation content based on the first text sequence corresponding to the historical conversation content, wherein the second text sequence is a sequence including input text units arranged in word order.

[0089] In the disclosed embodiment, the content input into the target language model may be preprocessed to obtain a second text sequence, wherein the second text sequence may include an ID sequence of tokens in the target conversation content, that is, a token ID list corresponding to the target conversation content. Specifically, a tokenizer may be performed on the tokens corresponding to the target conversation content to obtain a token ID list.

[0090] Step S304: input the second text sequence into the target language model, obtain the reply content corresponding to the current question content, and display the reply content through the terminal device.

[0091] In the disclosed embodiment, the target language model may be an LLM in an edge device, and the reply content may be "LLM's response" displayed by the LLM to the interactive object. Here, the second text sequence may be input into the LLM, so that the LLM calls the KV cache corresponding to each token in the target session content based on the second text sequence, and performs reasoning based on the KV cache to obtain the reply content.

[0092] Specifically, LLM usually includes a prefilling phase and a decoding phase when performing reasoning. In the prefilling phase, LLM first processes the input message and converts it into a Token. Then, each Token is processed one by one to calculate the intermediate state (key key and value value). These intermediate states are used to generate the "first" new Token, where the generation of each new Token depends on all previous Tokens.

[0093] In the decoding phase, LLM generates output tokens one by one in an autoregressive manner until the stopping condition is met, where each sequentially output token utilizes the output state (key and value) of all previous iterations. Therefore, the keys and values ​​generated in the pre-filling phase are cached (also called KV Cache) to optimize the speed and performance of the decoding process. KVcache is an important factor for accelerating the decoding phase.

[0094] Here, if Figure 4 The figure shows the schematic diagram of the LLM reasoning process, where in the pre-filling stage, the input message is converted into 4 tokens, which are processed and converted into query, key and value respectively. Then, the formula softmax(q*k t +Mask) to calculate the Attention matrix. Finally, the fifth Token is output according to this Attention matrix. In the decoding stage, the fifth Token output above can be used and converted into Query, Key and Value (here only the fifth Token is included). Then, the pre-cached KV cache is obtained to calculate the next Token, that is, the sixth Token.

[0095] It can be known from the above description that in the embodiment of the present disclosure, first, the current question content of the interactive object and the smart terminal can be obtained, and at least part of the target conversation content associated with the current question content can be determined in the historical conversation content, wherein the target conversation content includes the question content and the reply content of the terminal device. Next, based on the first text sequence corresponding to the historical conversation content, the second text sequence corresponding to the target conversation content can be determined, wherein the second text sequence is a sequence containing input text units arranged in word order. Then, the second text sequence is input into the target language model to obtain the reply content corresponding to the current question content, and the reply content is displayed through the terminal device, thereby managing the conversation content input into the language model to improve the reasoning accuracy and answering speed of the language model in the edge smart device.

[0096] In some optional implementations, the above step S301, determining at least part of the target conversation content associated with the current question content in the historical conversation content, includes:

[0097] The historical conversation content and the current question content are acquired based on the lightweight language model, and based on the text features of the historical conversation content, at least a portion of the target conversation content associated with the current question content in the historical conversation content is determined.

[0098] In the embodiments of the present disclosure, Figure 5 Shown is a schematic diagram for determining the target conversation content, wherein a small language model (i.e., the lightweight language model) can be set in the edge device to manage the historical conversation content and determine at least part of the target conversation content associated with the current question content.

[0099] For example, the above-mentioned lightweight language models usually use smaller neural network architectures, such as recurrent neural networks, convolutional neural networks, or Transformers, etc. These architectures can effectively process text data and have good performance under limited computing resources.

[0100] In an embodiment of the present disclosure, a small language model may be set in an edge device to determine at least part of the target conversation content associated with the current question content in the historical conversation content, thereby reducing the demand for computing resources of the edge device, reducing computing delays, and improving user experience.

[0101] In some optional implementations, the above step S303, based on the first text sequence corresponding to the historical conversation content, determines the second text sequence corresponding to the target conversation content, including:

[0102] Step S3031, determining the position of the target conversation content in the historical conversation content.

[0103] Step S3032: When it is determined according to the position that the target conversation content is not adjacent to the current question content, irrelevant content between the target conversation content and the current question content in the historical conversation content is filtered out to obtain a second text sequence.

[0104] In the embodiment of the present disclosure, considering that the target conversation content associated with the current question content may not be associated in the historical conversation content, it is necessary to filter out irrelevant content between the target conversation content and the current question content, thereby truncating the historical conversation content to reduce the length of the text input into the LLM.

[0105] For example, if Figure 6 The figure shows a schematic diagram of a user message input into the LLM. Here, the user message may consist of a system prompt word and a user message question and answer, wherein the user message question and answer may include two parts: a user question and an LLM answer. It should be understood that user message question and answer-1 and user message question and answer 2 are the target answer contents of the current question content N, and the irrelevant contents between user message question and answer 2 and the target answer contents are user message question and answer 3-user question and answer message (N-3).

[0106] After filtering out irrelevant content in the historical conversation content, the second text sequence can be determined based on the filtered historical conversation content, and a corresponding ID can be assigned to each token in the second text sequence to obtain a Token ID list generated by the second text sequence. By looking up the ID corresponding to each token, LLM can convert the second text sequence into a series of integers for easy calculation and prediction.

[0107] Specifically, the above step S3032, filtering out irrelevant content between the target conversation content and the current question content in the historical conversation content to obtain the second text sequence, includes:

[0108] Step a1: Obtain a third text sequence corresponding to the target conversation content.

[0109] Step a2: Match the third text sequence with the first text sequence to obtain irrelevant text sequences for which there is no matching result in the first text sequence, and filter out irrelevant text sequences to obtain a second text sequence.

[0110] In the disclosed embodiment, after the target conversation content is obtained, the third text sequence corresponding to the target conversation content can be tokenized to obtain Token ID list1, and the Token ID list2 corresponding to the first text sequence can be obtained. Here, the Position ID and the Token ID list1 can be input into the LLM, so that the LLM compares the Token ID list1 with the Token ID list2 based on the Position ID, wherein the Position ID can be used to indicate the encoding position of the token in the text sequence.

[0111] If the comparison results are the same, it means that there are no irrelevant text sequences, such as Figure 7 It is a flow chart for determining output data, wherein Position ID 14 and 15 are Position IDs corresponding to the current question content.

[0112] In addition, if the comparison results are different, it means that the first text sequence is truncated, that is, irrelevant text sequences in the first text sequence are filtered out, such as Figure 8 Another flow chart for determining output data is shown, wherein, through comparison, it can be determined that token ID3981, 869, 1068 and 15 corresponding to Position ID5-8 in Token ID list2 are irrelevant text sequences. Therefore, the irrelevant text sequences in Token ID list2 can be filtered out to obtain the second text sequence.

[0113] Then, the KV cache can be truncated according to the second text sequence. Specifically, the KV cache corresponding to okenID3981, 869, 1068 and 15 can be filtered out to obtain the filtering result, that is, Figure 8 KV Catch Itme1-KV Catch Itme4 and KV Catch Itme9-KV Catch Itme12.

[0114] In the disclosed embodiment, the historical conversation contents irrelevant to the current question content can be filtered out, thereby managing the text input into the LLM, improving the quality of the input content, and thus improving the performance of the LLM.

[0115] In some optional implementations, the above step S304, inputting the second text sequence into the target language model to obtain the reply content corresponding to the current question content, includes:

[0116] Step S3041, obtaining a first vector cache corresponding to a key vector and a second vector cache corresponding to a value vector of a text unit in a second text sequence.

[0117] Step S3042: Position encoding is performed on the first vector cache based on the target language model, and an output result is determined according to the encoding result and the second vector cache.

[0118] Step S3043, determine the reply content according to the output result.

[0119] In the embodiment of the present disclosure, the key vector is key, the first vector cache is key cache, the value vector is value, and the second vector cache is value cache.

[0120] Specifically, Fig. 9 The figure shows another flow chart for determining the output result, in which the key cache and the query can be positionally encoded based on RoPE, and the formula softmax(q*k t +Mask) to calculate the Attention matrix, perform matrix multiplication matmul operation on the Attention matrix to obtain the output token, and determine the output result according to the output token.

[0121] In the disclosed embodiment, the key cache may be position-encoded based on RoPE instead of the key, thereby updating the position coding of the KV cache of the output result and avoiding the use of the position coding before the update.

[0122] In some optional embodiments, the above Figure 3 The corresponding embodiment also includes:

[0123] The text position identifier corresponding to the historical conversation content is input into the target language model so that the target language model updates the text position identifier based on the first text sequence corresponding to the reply content, wherein the text position identifier is used to indicate the position encoding of the text unit in the first text sequence.

[0124] In the disclosed embodiment, when reasoning is performed through LLM, the above-mentioned Position ID can be input into LLM together, so that LLM updates the Position ID and the token ID list of KVcache corresponding to the Position ID based on the output token corresponding to the reply content.

[0125] Specifically, Fig.10 Shown above Fig. 9The Position ID and token IDlist corresponding to the output result in the , wherein the token ID list2 corresponding to the historical session content can be updated according to the token ID list, and the KV cache corresponding to the output result and the token ID list2 corresponding to the updated historical session content are stored in the GPU video memory of the edge device.

[0126] In the disclosed embodiment, the KV cache corresponding to the output result and the token ID list2 corresponding to the updated historical session content can be stored in the GPU video memory of the edge device, so as to facilitate direct calling of the KV cache according to the token ID list2 in subsequent reasoning without repeated calculation, so as to improve the response speed of LLM.

[0127] In some optional embodiments, the above Figure 3 The corresponding embodiment also includes:

[0128] Step S21, when there are multiple interactive objects, determine the historical conversation content according to the conversation content of the preset interactive objects.

[0129] Step S22, when the sum of the text lengths corresponding to the historical conversation content and the system prompt content exceeds the text length threshold of the target language model, execute the step of determining at least part of the target conversation content associated with the current question content in the historical conversation content.

[0130] In the embodiment of the present disclosure, the preset interaction object may be an interaction object that supports conversation in the edge device. Specifically, the interaction object that is detected with the highest priority and meets the preset interaction object quantity requirement may be determined as the preset interaction object.

[0131] For example, if the number k of preset interaction objects supported by the edge device is 3, the first 3 interaction objects detected can be determined as preset interaction objects, and the conversation content with the preset interaction objects in the interaction scenario can be determined as historical conversation content to manage the KV cache corresponding to the historical conversation content.

[0132] Here, if Fig.11 It is a schematic diagram of the process of managing the KV cache corresponding to the historical conversation content, wherein, after the preset interaction objects 1-3 are determined, the object identifiers of the preset interaction objects 1-3, ie userID1-user ID3, can be determined respectively.

[0133] Then, the conversation content of each preset interactive object can be determined in the historical conversation content, and the token ID list of the text sequence corresponding to the conversation content, that is, history token IDs-1 to history tokenIDs-3, can be obtained to manage the KV cache of the conversation content of each preset interactive object based on the token ID list. For example, the user ID is associated with the token used by the user in the past, and then the token is mapped with the KV cache to obtain the mapping relationship between the user ID, the token ID list and the KV cache.

[0134] It should be understood that the number of preset interaction objects can be managed in advance. For example, the optimal number of supported preset interaction objects can be calculated based on the maximum video memory supported by the current GPU in the edge device, the preset maximum number of user IDs, the maximum context length supported by the model, and the video memory required for the current model KV cache.

[0135] In the disclosed embodiment, considering that the conversation content between the preset interactive object and the LLM can be continuously accumulated, it is possible to determine whether the sum of the text lengths corresponding to the historical conversation content and the system prompt content exceeds the text length threshold of the LLM to ensure the normal response of the LLM.

[0136] Specifically, when the total length of the token length of the conversation content of K preset interaction objects and the expected number of LLM response tokens exceeds the LLM text length threshold, it is necessary to determine whether the number of K needs to be reduced one by one.

[0137] In the disclosed embodiment, the number of preset interactive objects supported by LLM can be managed, so that the performance of LLM reasoning is improved while LLM can support multiple interactive objects.

[0138] In some optional embodiments, the above Figure 3 The corresponding embodiment also includes:

[0139] Step S31, matching the current interactive object that initiates the current question content with the object identifier of the preset interactive object.

[0140] In step S32, if the preset interactive objects do not include the current interactive object, any target interactive object is determined in the preset interactive objects.

[0141] Step S33: Store the conversation content corresponding to the target interactive object in the historical conversation content into the local storage of the terminal device.

[0142] In an embodiment of the present disclosure, when the current interaction object is not a preset interaction object, the conversation content of the preset interaction object can be replaced in the local storage of the edge device, such as memory or disk, and the current interaction object is determined as a new preset interaction object.

[0143] Specifically, any target interactive object can be determined from the preset interactive objects, and the target interactive object can be replaced by the current interactive object, that is, the conversation content of the current interactive object is stored in the video memory, and the conversation content of the target interactive object is stored in the local storage.

[0144] For example, if Fig.12 The figure shows a flow chart of managing the conversation content of the target interactive object, wherein the object identifier of the target interactive object may be determined as user ID4, and the token ID list and KV cache corresponding to the mapping relationship of the user ID4 are stored in the memory or disk.

[0145] In an embodiment of the present disclosure, when the current interaction object is not a preset interaction object, the session content of any interaction object in the preset interaction objects can be replaced in the local storage of the edge device, for example, in the memory or disk, and the current interaction object can be determined as a new preset interaction object, so that when the session content of any interaction object needs to be called, it can be called from the local storage, which is equivalent to expanding the number of preset interaction objects supported by the edge device.

[0146] In this embodiment, another conversation method is provided, which can be used for the above-mentioned smart devices, such as edge devices such as smart voice assistants in smart cockpits. Fig.13 is a flow chart of a conversation method according to an embodiment of the present disclosure, such as Fig.13 As shown, the process includes the following steps:

[0147] Step S1301, detecting the request of the interactive object and obtaining the current question content.

[0148] Step S1302: Convert the current question content into a token ID through a tokenizer.

[0149] Step S1303: Acquire the KV cache of the historical conversation content corresponding to the interactive object.

[0150] Step S1304: Acquire, build and manage the KV cache corresponding to the token ID.

[0151] Step S1305, constructing a Postion ID corresponding to each token ID.

[0152] Step S1306: input the KV cache and Position ID into the target language model for forward calculation.

[0153] Step S1307: updating the token ID corresponding to the historical conversation content of the interactive object based on the calculation result.

[0154] Step S1308: Update the KV cache according to the token ID.

[0155] Step S1309: determine the output result according to the KV cache.

[0156] Step S1310, determine the reply content based on the output result and display it.

[0157] In the embodiment of the present disclosure, the implementation method of determining the reply content is as described above. Figure 1 The corresponding embodiments are described in detail and will not be repeated here.

[0158] In summary, in the embodiments of the present disclosure, the current question content of the interactive object and the smart terminal can be first obtained, and at least part of the target conversation content associated with the current question content can be determined in the historical conversation content, wherein the target conversation content includes the question content and the reply content of the terminal device. Next, based on the first text sequence corresponding to the historical conversation content, the second text sequence corresponding to the target conversation content can be determined, wherein the second text sequence is a sequence of input text units arranged in word order. Then, the second text sequence is input into the target language model to obtain the reply content corresponding to the current question content, and the reply content is displayed through the terminal device, thereby managing the conversation content input into the language model to improve the reasoning accuracy and answering speed of the language model in the edge smart device.

[0159] In this embodiment, a conversation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0160] This embodiment provides a conversation device, such as Fig.14 As shown, including:

[0161] The acquisition module 1401 is used to acquire the current question content of the interactive object and the intelligent terminal;

[0162] A first determination module 1402 is used to determine at least part of the target conversation content associated with the current question content in the historical conversation content;

[0163] A second determination module 1403 is used to determine a second text sequence corresponding to the target conversation content based on the first text sequence corresponding to the historical conversation content, wherein the second text sequence is a sequence including input text units arranged in word order;

[0164] The input module 1404 is used to input the second text sequence into the target language model to obtain the reply content corresponding to the current question content.

[0165] In some optional implementations, the second determining module 1403 is further configured to:

[0166] Determine the position of the target conversation content in the historical conversation content;

[0167] When it is determined according to the position that the target conversation content is not adjacent to the current question content, irrelevant content between the target conversation content and the current question content in the historical conversation content is filtered out to obtain a second text sequence.

[0168] In some optional implementations, the second determining module 1403 is further configured to:

[0169] Obtaining a third text sequence corresponding to the target conversation content;

[0170] The third text sequence is matched with the first text sequence to obtain irrelevant text sequences for which no matching result exists in the first text sequence, and the irrelevant text sequences are filtered out to obtain a second text sequence.

[0171] In some optional implementations, the input module 1404 is further configured to:

[0172] Obtain a first vector cache corresponding to a key vector and a second vector cache corresponding to a value vector of a text unit in a second text sequence;

[0173] Position encoding is performed on the first vector cache based on the target language model, and an output result is determined according to the encoding result and the second vector cache;

[0174] Determine the reply content based on the output results.

[0175] In some optional embodiments, the device is also used for:

[0176] The text position identifier corresponding to the historical conversation content is input into the target language model so that the target language model updates the text position identifier based on the first text sequence corresponding to the reply content, wherein the text position identifier is used to indicate the position encoding of the text unit in the first text sequence.

[0177] In some optional embodiments, the device is also used for:

[0178] When there are multiple interaction objects, the historical conversation content is determined according to the conversation content of the preset interaction objects;

[0179] When the sum of the text lengths corresponding to the historical conversation content and the system prompt content exceeds the text length threshold of the target language model, a step of determining at least part of the target conversation content associated with the current question content in the historical conversation content is performed.

[0180] In some optional implementations, the first determining module 1402 is further configured to:

[0181] The historical conversation content and the current question content are acquired based on the lightweight language model, and based on the text features of the historical conversation content, at least a portion of the target conversation content associated with the current question content in the historical conversation content is determined.

[0182] In some optional embodiments, the device is also used for:

[0183] Matching the object identifiers of the current interactive object that initiated the current question content with the preset interactive object;

[0184] If the preset interactive objects do not include the current interactive object, determine any target interactive object in the preset interactive objects;

[0185] The session content corresponding to the target interactive object in the historical session content is stored in the local storage of the terminal device.

[0186] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0187] The conversation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0188] The present disclosure also provides a computer device having the above Fig.14 The conversation device shown.

[0189] See also Fig.15 , Fig.15 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Fig.15As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.15 A processor 10 is taken as an example.

[0190] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0191] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0192] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0193] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0194] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0195] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0196] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0197] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0198] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0199] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0200] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0201] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A conversation method, characterized in that: The method comprises: Get the current question content of the interactive object and the terminal device; Determining at least part of the target conversation content associated with the current question content in the historical conversation content, wherein the target conversation content includes the question content and the reply content of the terminal device; Determining a second text sequence corresponding to the target conversation content based on the first text sequence corresponding to the historical conversation content, wherein the second text sequence is a sequence including input text units arranged in word order; The second text sequence is input into the target language model to obtain the reply content corresponding to the current question content, and the reply content is displayed through the terminal device.

2. The method according to claim 1, characterized in that: The determining, based on the first text sequence corresponding to the historical conversation content, a second text sequence corresponding to the target conversation content includes: Determine the position of the target conversation content in the historical conversation content; When it is determined according to the position that the target conversation content is not adjacent to the current question content, irrelevant content between the target conversation content and the current question content in the historical conversation content is filtered out to obtain a second text sequence.

3. The method according to claim 2, characterized in that The filtering out irrelevant content between the target conversation content and the current question content to obtain a second text sequence includes: Acquire a third text sequence corresponding to the target conversation content; The third text sequence is matched with the first text sequence to obtain irrelevant text sequences for which no matching result exists in the first text sequence, and the irrelevant text sequences are filtered out to obtain a second text sequence.

4. The method according to claim 1, characterized in that The step of inputting the second text sequence into a target language model to obtain a reply content corresponding to the current question content includes: Obtaining a first vector cache corresponding to a key vector and a second vector cache corresponding to a value vector of a text unit in the second text sequence; Performing position encoding on the first vector cache based on the target language model, and determining an output result according to the encoding result and the second vector cache; The reply content is determined according to the output result.

5. The method according to claim 1, characterized in that The method further comprises: The text position identifier corresponding to the historical conversation content is input into the target language model so that the target language model updates the text position identifier based on the first text sequence corresponding to the reply content, wherein the text position identifier is used to indicate the position encoding of the text unit in the first text sequence.

6. The method according to claim 1, characterized in that The method further comprises: When there are multiple interactive objects, the historical conversation content is determined according to the conversation content of the preset interactive objects; When the sum of the text lengths corresponding to the historical conversation content and the system prompt content exceeds the text length threshold of the target language model, a step of determining at least part of the target conversation content associated with the current question content in the historical conversation content is performed.

7. The method according to claim 1, characterized in that The step of determining at least a portion of the target conversation content associated with the current question content in the historical conversation content includes: The historical conversation content and the current question content are acquired based on a lightweight language model, and at least part of the target conversation content associated with the current question content in the historical conversation content is determined based on text features of the historical conversation content.

8. The method according to claim 1, characterized in that The method further comprises: Matching the object identifiers of the current interactive object that initiated the current question content with the preset interactive object; If the preset interactive objects do not include the current interactive object, determining any target interactive object from the preset interactive objects; The conversation content corresponding to the target interactive object in the historical conversation content is stored in the local storage of the terminal device.

9. A conversation device, characterized in that: The device comprises: The acquisition module is used to obtain the current question content of the interactive object and the terminal device; A first determination module is used to determine at least part of the target conversation content associated with the current question content in the historical conversation content, wherein the target conversation content includes the question content and the reply content of the terminal device; A second determination module, configured to determine a second text sequence corresponding to the target conversation content based on the first text sequence corresponding to the historical conversation content, wherein the second text sequence is a sequence of input text units arranged in word order; An input module is used to input the second text sequence into a target language model, obtain a reply content corresponding to the current question content, and display the reply content through the terminal device.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the conversation method according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the conversation method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the conversation method according to any one of claims 1 to 8.

Images