Multi-round dialogue reasoning acceleration method based on long and short memory and related device

Abstract

The invention discloses a multi-round dialogue reasoning acceleration method based on long and short memory, which is applied to a man-machine dialogue scene realized based on a model. In the reasoning acceleration method, for a question text obtained in a question and answer scene, features of a part of historical dialogue contents under the same dialogue role and features of abstracts of other dialogue contents before the part of historical dialogue contents are obtained, and then a target model is assisted to process the question text based on the features. And the problem text processing precision of the target model is ensured. Moreover, the scheme is based on the AI native storage, that is, the memory (that is, the features extracted by the model in advance) of the large model is stored, secondary reasoning of the large model is not needed, and the features can be taken and used immediately through calculation by search, so that the reasoning efficiency of the large model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and device for accelerating multi-turn dialogue reasoning based on long short-term memory. Background Technology

[0002] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data. They can not only generate natural language text, but also deeply understand the meaning of text and handle various natural language tasks, such as text summarization, intelligent question answering, and text translation.

[0003] With the development of large language models, their growth in scale is particularly remarkable, with the number of parameters jumping from billions to trillions. This increase in parameters allows large language models to capture the subtleties of human language more precisely and to gain a deeper understanding of its complexity.

[0004] However, due to the sheer number of parameters in large language models, each use incurs significant computational and time overhead due to the complexity of parameter calculation. This is especially true in long-sequence reasoning scenarios (such as question-answering), where the input to a large language model is a long sequence of text. Large language models often require a substantial amount of time to execute the reasoning process, resulting in slow response times and difficulty in meeting user needs. Summary of the Invention

[0005] This application provides a method for accelerating multi-turn dialogue reasoning based on long short-term memory, which can improve the reasoning speed of models in human-computer dialogue scenarios.

[0006] Firstly, a reasoning acceleration method is provided, applied to an execution device in a human-computer dialogue scenario. This reasoning acceleration method includes: firstly, the execution device acquires question text, which indicates the question to be answered. This question text is, for example, provided to a user in a human-computer dialogue scenario.

[0007] Then, based on the dialogue role to which the question text belongs, the execution device obtains the target historical dialogue content and target historical dialogue summary corresponding to the dialogue role. The target historical dialogue content includes the dialogue content of at least one round of the first dialogue prior to the question text, i.e., the dialogue content between the execution device and the dialogue role prior to the question text. Furthermore, the at least one round of the first dialogue can be adjacent to the question text; that is, at least one round of the first dialogue is both prior to and adjacent to the question text. The target historical dialogue summary includes a summary of the dialogue content of at least one round of the second dialogue prior to the first dialogue. That is, the target historical dialogue summary is not the complete historical dialogue content, but a summary of the dialogue content. "At least one round of the first dialogue prior to the question text" means that the at least one round of the first dialogue occurred before the question text in the conversation between the dialogue role and the execution device. Similarly, "the second dialogue prior to the first dialogue" means that the second dialogue occurred before the first dialogue; the first dialogue occurred after the second dialogue between the execution device and the dialogue role.

[0008] Finally, based on the target historical dialogue summary and the target historical dialogue content, the execution device processes the question text through the target model to obtain the answer text. For example, the execution device can use the target historical dialogue summary and the target historical dialogue content as reference information, and then input the target historical dialogue summary, the target historical dialogue content, and the question text together into the target model, which processes the question text to obtain the answer text output by the target model.

[0009] In this solution, for the question text obtained in a question-answering scenario, a summary of a portion of the historical dialogue content under the same dialogue character, as well as summaries of other dialogue content preceding this portion of historical dialogue content, is obtained. This summary of historical dialogue content and other dialogue content is then used to assist the target model in processing the question text, ensuring the accuracy of the target model's processing. Furthermore, since this solution selects a portion of the complete historical dialogue content that is relatively recent in time to the current question text, as well as summaries of historical dialogue content from even earlier times, as input to the target model, rather than selecting all historical dialogue content, it effectively reduces the length of the input text for the target model and improves the inference speed of the target model.

[0010] Furthermore, since historical dialogue content from a long time ago has low relevance to the current question text, extracting summaries of these historical dialogues as input to the target model can effectively reduce redundant information input to the target model, thereby improving its accuracy. In other words, in some cases, compared to inputting all historical dialogue content into the target model, this approach achieves higher inference accuracy by using summaries of older historical dialogues as input.

[0011] In one possible implementation, when processing the question text through the target model, the execution device queries the feature library to obtain the first feature corresponding to the target historical dialogue summary and the second feature corresponding to the target historical dialogue content. The first feature and the second feature are extracted based on the target model.

[0012] Then, based on the first and second features, the execution device processes the question text through the target model. That is, the execution device uses the first feature as the feature corresponding to the target historical dialogue summary and the second feature as the feature of the target historical dialogue content, thus directly reusing the first and second features, without needing to perform feature extraction processing on the input target historical dialogue summary and target historical dialogue content through the target model.

[0013] In this application, since both the target historical dialogue summary and the target historical dialogue content are obtained based on previous dialogue content, by performing feature extraction in advance and storing the extracted features in a feature library, the stored features can be directly reused in the actual reasoning process, thereby eliminating the need to perform feature extraction on the historical dialogue content, realizing query-based calculation, and effectively improving the reasoning speed of the target model.

[0014] In one possible implementation, the first feature includes the original features obtained by performing feature extraction on the dialogue content of at least one round of the second dialogue; or, the first feature includes the summary features obtained by performing feature extraction on the summary of the dialogue content of at least one round of the second dialogue.

[0015] In other words, the first feature stored in the feature library can be either a feature of the target historical dialogue summary itself or a feature of the original dialogue content corresponding to the target historical dialogue summary. Generally speaking, the target historical dialogue summary is obtained by summarizing and generalizing the original dialogue content, so the features of the target historical dialogue summary itself focus more on the macro-level information of the dialogue; while the features of the original dialogue content corresponding to the target historical dialogue summary focus more on the detailed information of the dialogue.

[0016] In one possible implementation, the target historical dialogue summary includes a first historical dialogue summary, which includes a summary of the dialogue content of a third dialogue, wherein the third dialogue is a round of dialogue in at least one round of the second dialogue. That is, the target historical dialogue summary may include one or more historical dialogue summaries, and the first historical dialogue summary is one of the historical dialogue summaries included in the target historical dialogue summary.

[0017] The first feature includes the original features of the third dialogue or the summary features of the third dialogue. The original features of the third dialogue include features obtained by performing feature extraction on the dialogue content of the third dialogue, and the summary features of the third dialogue include features obtained by performing feature extraction on the summary of the dialogue content of the third dialogue.

[0018] In this method, the execution device can determine a memory score of a first historical dialogue summary, which indicates the usage of the first historical dialogue summary. Generally, the more frequently the first historical dialogue summary is used, the higher its memory score; conversely, the less frequently it is used, the lower its memory score.

[0019] Then, based on the memory score, the execution device determines the storage status of the original features or summary features of the third dialogue in the feature library.

[0020] In this scheme, the usage of historical dialogue summaries is measured by calculating a memory score. Based on the memory score, it is possible to determine whether to store features of the historical dialogue summaries or features of the original dialogue content of the historical dialogue summaries. This allows for a balance between improving the accuracy of the target model and saving storage space, thereby enhancing the practicality of the scheme.

[0021] In one possible implementation, the storage status of the original features or summary features of the third dialogue in the feature library is determined based on the memory score, specifically including:

[0022] If the memory score is within the first range and the original features of the third dialogue are not stored in the feature library, then the original features of the third dialogue are obtained by performing feature extraction on the dialogue content of the third dialogue through the target model, and the original features of the third dialogue are stored in the feature library.

[0023] In one possible implementation, the storage status of the original features or summary features of the third dialogue in the feature library is determined based on the memory score, specifically including:

[0024] If the memory score is in the second range and the feature library does not store the summary features of the third dialogue, then the summary features of the third dialogue are obtained by performing feature extraction on the summary of the first historical dialogue through the target model, and the summary features of the third dialogue are stored in the feature library, wherein the value in the second range is less than the value in the first range.

[0025] In other words, if the memory score corresponding to the summary of the third dialogue content is relatively high, it means that the summary of the third dialogue content is used more frequently. Therefore, the original features of the third dialogue can be stored in the feature library, allowing more attention to the detailed information of the third dialogue when processing the question text. If the memory score corresponding to the summary of the third dialogue content is average, it means that the summary of the third dialogue content is used less frequently. Therefore, the summary features of the third dialogue can be stored in the feature library, allowing more attention to the macroscopic information of the third dialogue when processing the question text.

[0026] In one possible implementation, if the memory score falls within the second range, and the feature library contains the original features of the third dialogue, then the original features of the third dialogue are deleted from the feature library. That is, the feature library only stores one set of features corresponding to the third dialogue, such as the original features of the third dialogue or the summary features of the third dialogue, and does not store both the original features and the summary features of the third dialogue simultaneously, thereby reducing storage space overhead.

[0027] In one possible implementation, the storage status of the original features or summary features of the third dialogue in the feature library is determined based on the memory score, including:

[0028] If the memory score falls within the third range: if the original features of the third dialogue are stored in the feature library, then the original features of the third dialogue are deleted from the feature library; or, if the summary features of the third dialogue are stored in the feature library, then the summary features of the third dialogue are deleted from the feature library. The values ​​in the third range are less than the values ​​in the second range, and the values ​​in the second range are less than the values ​​in the first range.

[0029] In other words, when the memory score corresponding to the summary of the dialogue content of the third dialogue is in the third range, it indicates that the memory score is low, which means that the summary of the dialogue content of the third dialogue is not used very frequently. In this case, the feature corresponding to the third dialogue in the feature library can be deleted to save storage space as much as possible.

[0030] In one possible implementation, the memory score is determined based on at least one of the following: the total number of times the target historical dialogue summary is used, the number of times the target historical dialogue summary is used within a preset time period, the time interval since the last use of the target historical dialogue summary, the duration since the target historical dialogue summary was generated, the number of consecutive uses of the first historical dialogue summary, the number of times the original features of the first historical dialogue summary are reused, the number of times the summary features of the first historical dialogue summary are reused, the length of the original dialogue content corresponding to the first historical dialogue summary, the length of the first historical dialogue summary, and the time interval since the last change in the range of the memory score.

[0031] In one possible implementation, the memory score is determined periodically, for example, the execution device determines the memory score every 5 hours, 10 hours, or one day.

[0032] In one possible implementation, the target historical dialogue summary includes at least two historical dialogue summaries, which correspond to dialogue content in different historical time periods.

[0033] When processing the question text using the target model based on the first and second features, the execution device first determines the first sub-feature of the first feature, which includes features corresponding to some tokens in the target historical dialogue summary. Based on the first feature, the execution device re-extracts the second sub-feature corresponding to some tokens using the target model.

[0034] Then, the execution device updates the first sub-feature in the first feature to the second sub-feature, obtaining the updated first feature. That is, the execution device recalculates the second sub-feature corresponding to this part of the token based on the first feature. In this way, based on the updated first and second features, the execution device processes the question text through the target model.

[0035] In this scheme, when the target historical dialogue summary includes at least two historical dialogue summaries, the features of some tokens are recalculated based on the features of the entire target historical dialogue summary. This enables cross-attention between the features of different historical dialogue summaries, making it easier for the target model to understand the reused features and improving the accuracy of the target model.

[0036] In one possible implementation, the question text originates from the first session; the target historical dialogue content includes dialogue content from the second session. The first session may include one or more rounds of dialogue content. The second session may also include one or more rounds of dialogue content; for example, the second session may include dialogue content from at least one round of the first dialogue mentioned above.

[0037] In one possible implementation, to obtain the target historical dialogue summary, the execution device acquires multiple historical dialogue summaries, each corresponding to dialogue content from different historical time periods. Based on the relevance of the multiple historical dialogue summaries to the question text, the execution device then determines the target historical dialogue summary from among the multiple historical dialogue summaries. The target historical dialogue summary includes one or more of the aforementioned multiple historical dialogue summaries.

[0038] In this solution, for historical question-and-answer dialogues between the execution device and the user from a relatively long time ago, the execution device performs summary extraction on the dialogue content from different historical time periods, thereby obtaining multiple historical dialogue summaries. In this way, the execution device can select the most relevant historical dialogue summaries from these summaries based on the question text, thus avoiding using all historical dialogue summaries to assist in processing the question text, reducing the amount of content that the target model needs to process, and improving the processing efficiency of the target model.

[0039] In one possible implementation, a single historical dialogue summary among multiple historical dialogue summaries corresponds to the dialogue content of at least one round of dialogue prior to the first round of dialogue, where N is an integer greater than or equal to 1.

[0040] In one possible implementation, the total length of the dialogue content corresponding to a single historical dialogue summary among multiple historical dialogue summaries is no greater than a preset length.

[0041] In one possible implementation, the target model is a large language model, and the first and second features include a key-value cache (KV Cache).

[0042] In this scheme, a storage medium is used to pre-store the features of historical dialogues. This allows the features needed for a specific round of historical dialogues to be read directly from the storage medium without recalculating the features of the historical dialogues. This achieves feature lookup instead of calculation and improves the inference speed of the target model.

[0043] For example, by offloading the KV cache of historical sessions to storage media for persistence, the KV cache can be reused in subsequent sessions through similarity retrieval, which can effectively utilize AI native storage to improve the inference speed of the model.

[0044] Secondly, a reasoning acceleration device is provided, comprising: an acquisition module for acquiring question text, the question text indicating a question to be answered; a processing module for acquiring target historical dialogue content and a target historical dialogue summary corresponding to the dialogue role to which the question text belongs, the target historical dialogue content including the dialogue content of at least one round of first dialogue before the question text was acquired, and the target historical dialogue summary including a summary of the dialogue content of at least one round of second dialogue before the first dialogue; the processing module is further configured to process the question text through a target model based on the target historical dialogue summary and the target historical dialogue content to obtain answer text.

[0045] In one possible implementation, the processing module is further configured to: query the feature library to obtain a first feature corresponding to the target historical dialogue summary and a second feature corresponding to the target historical dialogue content, wherein the first feature and the second feature are extracted based on the target model;

[0046] Based on the first and second features, the problem text is processed through the target model.

[0047] In one possible implementation, the first feature includes the original features obtained by performing feature extraction on the dialogue content of at least one round of the second dialogue; or,

[0048] The first feature includes the summary features obtained by performing feature extraction on a summary of the dialogue content of at least one round of the second dialogue.

[0049] In one possible implementation, the target historical dialogue summary includes a first historical dialogue summary, which includes a summary of the dialogue content of a third dialogue, wherein the third dialogue is one round of at least one round of a second dialogue. The first feature includes either the original feature of the third dialogue or the summary feature of the third dialogue. The original feature of the third dialogue includes features obtained by performing feature extraction on the dialogue content of the third dialogue. The summary feature of the third dialogue includes features obtained by performing feature extraction on the summary of the dialogue content of the third dialogue. The processing module is further configured to:

[0050] Determine the memory score of the first historical dialogue summary, which is used to indicate the usage of the first historical dialogue summary;

[0051] Based on the memory score, determine the storage status of the original features or summary features of the third dialogue in the feature library.

[0052] In one possible implementation, the processing module is further configured to: if the memory score is within a first range and the original features of the third dialogue are not stored in the feature library, then perform feature extraction on the dialogue content of the third dialogue through the target model to obtain the original features of the third dialogue, and store the original features of the third dialogue in the feature library.

[0053] In one possible implementation, the processing module is further configured to: if the memory score is within the second range and the feature library does not store the summary features of the third dialogue, then perform feature extraction on the first historical dialogue summary using the target model to obtain the summary features of the third dialogue, and store the summary features of the third dialogue in the feature library, wherein the value in the second range is less than the value in the first range.

[0054] In one possible implementation, the processing module is further configured to: if the feature library stores the original features of the third dialogue, delete the original features of the third dialogue from the feature library.

[0055] In one possible implementation, the processing module is also used for:

[0056] When the memory score is in the third range:

[0057] If the original features of the third dialogue are stored in the feature library, then delete the original features of the third dialogue from the feature library; or,

[0058] If the feature library stores summary features of the third dialogue, then delete the summary features of the third dialogue from the feature library; wherein, the value in the third range is less than the value in the second range, and the value in the second range is less than the value in the first range.

[0059] In one possible implementation, the memory score is determined based on at least one of the following: the total number of times the first historical dialogue summary is used, the number of times the first historical dialogue summary is used within a preset time period, the time interval since the last use of the first historical dialogue summary, the duration since the first historical dialogue summary was generated, the number of consecutive uses of the first historical dialogue summary, the number of times the original features of the first historical dialogue summary are reused, the number of times the summary features of the first historical dialogue summary are reused, the length of the original dialogue content corresponding to the first historical dialogue summary, the length of the first historical dialogue summary, and the time interval since the last change in the range of the memory score.

[0060] In one possible implementation, the memory score is determined periodically.

[0061] In one possible implementation, the target historical dialogue summary includes at least two historical dialogue summaries, which correspond to the dialogue content of dialogues in different historical time periods.

[0062] The processing module is further configured to: determine the first sub-feature in the first feature, the first sub-feature including features corresponding to some tokens in the target historical dialogue summary; based on the first feature, re-extract the second sub-feature corresponding to some tokens through the target model; update the first sub-feature in the first feature to the second sub-feature to obtain the updated first feature; and process the question text through the target model based on the updated first feature and second feature.

[0063] In one possible implementation, the question text comes from the first session; the target historical dialogue content includes dialogue content from the second session.

[0064] In one possible implementation, the acquisition module is further configured to acquire multiple historical dialogue summaries, which correspond to dialogue content in different historical time periods; the processing module is further configured to determine the target historical dialogue summary from the multiple historical dialogue summaries based on the relevance of the multiple historical dialogue summaries to the question text.

[0065] Thirdly, an inference acceleration device is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the inference acceleration device to perform the method of any of the above aspects.

[0066] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods described in any of the preceding aspects.

[0067] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0068] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0069] In a seventh aspect, a computing device is provided, the computing device including an inference acceleration device of the third aspect or a chip system of the sixth aspect, wherein the inference acceleration device or the chip system in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0070] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0071] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0072] Figure 1 A schematic diagram of a system architecture provided for this application;

[0073] Figure 2 A flowchart illustrating a reasoning acceleration method provided in this application;

[0074] Figure 3 A schematic diagram illustrating how to determine a target historical dialogue summary based on historical dialogue content, as provided in this application;

[0075] Figure 4 A schematic diagram illustrating the generation of historical dialogue summaries based on historical dialogue content, as provided in this application;

[0076] Figure 5 Another schematic diagram for generating historical dialogue summaries based on historical dialogue content, provided for this application;

[0077] Figure 6 A schematic diagram illustrating how to accelerate target model inference by reusing features from a feature library, as provided in this application;

[0078] Figure 7 A schematic diagram illustrating how to process historical dialogue summaries based on memory scores, as provided in this application;

[0079] Figure 8 A schematic diagram of the execution flow of an inference acceleration method provided in this application;

[0080] Figure 9 A schematic diagram of a short memory window and a long memory window for performing device maintenance provided in this application;

[0081] Figure 10 This application provides a flowchart illustrating problem text processing based on short memory and long memory.

[0082] Figure 11 A schematic diagram of a reasoning acceleration device provided in this application;

[0083] Figure 12 A schematic diagram of the structure of a computing device provided in this application;

[0084] Figure 13 This application provides a schematic diagram of the structure of a computing device cluster;

[0085] Figure 14This application provides a schematic diagram of another computing device cluster structure.

[0086] Figure 15 This is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0088] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0089] To facilitate understanding, some technical terms used in this application will be introduced below.

[0090] (1) Large Language Model

[0091] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0092] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0093] Currently, large language models are mainly composed of Transformer networks.

[0094] (2) Transformer network

[0095] Transformer networks are powerful sequence models, but the computation time and memory required increase quadratically with sequence length, significantly increasing the hardware's storage and computing power demands. Essentially, Transformer networks employ a self-attention mechanism. Self-attention is a mechanism that associates different positions within a single sequence to compute a representation of the same sequence, playing a crucial role in machine reading, abstract summarization, and image description generation.

[0096] Taking the Transformer network applied to natural language processing as an example, the Transformer network processes input data of arbitrary length and generates new feature representations of the input data, which are then converted into target words. The self-attention network layer in the Transformer network uses an attention mechanism to capture the relationships between all other words, thereby generating new feature representations for each word. The advantage of the Transformer network's self-attention network is that the attention mechanism can directly capture the relationships between all words in a sentence without considering word positions.

[0097] (3) Key-Value Cache (KV Cache)

[0098] When using Transformer networks for natural language processing tasks, they typically employ a self-attention mechanism to process the input sequence. In this mechanism, the Transformer network generates a corresponding key (K) vector, value (V) vector, and query (Q) vector for each token in the input sequence. Further, the Transformer network calculates the degree of matching between each query vector and all key vectors, usually achieved through a dot product. Then, using the matching degree between query and key vectors as weights, it calculates a weighted sum of all value vectors to obtain the final result.

[0099] When the Transformer network processes the input sequence, the key vector generated by the Transformer network for each word in the input sequence can be stored in a single matrix, and the value vector generated by the Transformer network for each word in the input sequence can be stored in another matrix. Therefore, KV Cache refers to the key matrix and value matrix generated by the Transformer network for the words in the input sequence, and these key matrices and value matrices are cached.

[0100] (4) Token

[0101] A lexical is the smallest semantic unit in natural language processing, typically derived from the decomposition of long texts such as words, sentences, paragraphs, and articles. Generally, lexical units can be composed of letters, words, or symbols; for example, they can be composed of one or more letters or one or more words. Simply put, a lexical unit can be understood as a "fragment" of a word or phrase—the smallest indivisible semantic unit. For example, "waterfall" can be broken down into two lexical units: "water" and "fall." Additionally, punctuation marks can also be decomposed into lexical units because they affect the semantic understanding of the entire text. For example, "I don't know." can be decomposed into five lexical units: "I," "don," "'t," "know," and ".".

[0103] (5) Session

[0104] A session is a concept of a temporary interactive state established between a client and a server, used to track user actions over a period of time. Sessions are typically used to maintain user state and information, such as login status and conversation content.

[0105] Specifically, the conversation can be explained from the following aspects.

[0106] Continuous interaction: A session refers to a series of requests and responses between a user and a server over a period of time.

[0107] Lifecycle: A session typically begins when a user logs in or accesses the application and ends when the user logs out or the session times out. The timeout period is usually set by the server; for example, 30 minutes of inactivity will invalidate the session.

[0108] Information storage: The server stores user state data, such as user identity, permissions, and personalized settings. The client informs the server which session it is associated with through some identifier (such as a Session ID).

[0109] Session ID: To identify different users' sessions, the server assigns a unique identifier (Session ID) to each session. This Session ID is usually stored on the client and sent to the server with each request.

[0110] In general, in the human-computer dialogue scenario of this application, a session refers to the interaction between a user and an execution device (such as a server) over a period of time. Therefore, a session can include one or more rounds of dialogue between the user and the execution device.

[0111] Currently, in some long sequence reasoning scenarios, the input of large language models is a long sequence of text. Large language models often need to spend a lot of time to perform the reasoning process, resulting in a slow response speed that is difficult to meet user needs.

[0112] For example, in question-and-answer scenarios in human-computer dialogue, a large language model can engage in multi-turn conversations with the user to answer their questions. In most cases, since the question posed by the user in the current turn of the conversation is often related to previous conversations, to improve the accuracy of the large language model's answers, it's common practice to input both the current question and previous conversation content into the large language model, thus using the previous conversation content to assist in processing the current question. However, when there have already been many turns of conversation with the user, inputting all previous conversation content into the large language model results in a long input text, leading to a slower response time.

[0113] In view of this, this application provides a reasoning acceleration method. For question text obtained in a question-answering scenario, it acquires a portion of historical dialogue content from the same dialogue character, along with summaries of other dialogue content preceding this portion of historical dialogue content. This summary of historical dialogue content and other dialogue content is then used to assist the target model in processing the question text, ensuring the accuracy of the target model's processing. Furthermore, since this solution selects a portion of complete historical dialogue content that is relatively recent in time to the current question text, as well as summaries of historical dialogue content from even earlier times, as input to the target model, rather than selecting all complete historical dialogue content, it effectively reduces the length of the input text for the target model and improves the reasoning speed of the target model.

[0114] Furthermore, since historical dialogue content from a long time ago has low relevance to the current question text, extracting summaries of these historical dialogues as input to the target model can effectively reduce redundant information input to the target model, thereby improving its accuracy. In other words, in some cases, compared to inputting all historical dialogue content into the target model, this approach achieves higher inference accuracy by using summaries of older historical dialogues as input.

[0115] Please see Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided for this application. (For example...) Figure 1 As shown, in this system architecture, the execution device 10 can be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster). The execution device 10 includes an accelerator 101 and a processor 102. The accelerator 101 is used to run the target model (e.g., a large language model) to process the inference tasks passed by the processor 102. The processor 102 is used to obtain task requests from clients (e.g., requests to answer a question) and schedule the accelerator 101 to process the specified inference task based on the task request from the client.

[0116] Optionally, the execution device 10 can be used in conjunction with other computing devices, such as data storage devices, load balancers, etc.; the execution device 10 can be deployed on a single physical site or distributed across multiple physical sites.

[0117] In addition, the system architecture also includes a data storage system 11, which is used to store data such as problem text, reference text, KV cache, or program code.

[0118] Optionally, for persistent data storage, the data storage system 11 can be located external to the execution device 10 and exchange data with the execution device 10 via a network. Alternatively, if the execution device 10 is a physical host, the data storage system 11 can also be located internally to the execution device 10, such as exchanging data with the processor via a bus. In this case, the data storage system 11 functions as a hard disk. With the data storage system 11, the execution device 10 can use data in the data storage system 11 (such as KV Cache) or call program code in the data storage system to implement the inference acceleration method provided in this application.

[0119] Optionally, users can interact with execution device 10 using their respective local devices. For example, a client 121 is deployed on local device 12, and users interact with the execution device through client 121 on local device 12. Local device 12 can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, laptop, or smart car.

[0120] Local device 12 can interact with execution device 10 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0121] Optionally, during the implementation of the inference acceleration method by the execution device 10, the local device 12 can provide the execution device 10 with task requests or prompts, so that the execution device 10 can process the input text through the model to complete the task. Furthermore, after the execution device 10 executes the inference acceleration method and obtains the output result, it can feed the output result back to the local device 12.

[0122] Please see Figure 2 , Figure 2 This is a flowchart illustrating a reasoning acceleration method provided in this application. Figure 2 As shown, the execution flow of this inference acceleration method includes the following steps 201-203.

[0123] Step 201: Obtain the question text, which indicates the question to be answered.

[0124] In this application, the inference acceleration method can be applied to scenarios where question answering is performed based on a target model, such as human-computer dialogue scenarios. In a question answering scenario, the user interacts with an execution device that has the target model deployed on it via a client on their local device, sending question text to the execution device to request an answer. That is, the question text provided by the user indicates the question that the execution device needs to answer. Therefore, the execution device can obtain the question text to be processed from the user, which can be, for example, "How many goals did Messi score in the World Cup?" or "Who is the author of Harry Potter?" or any question composed of text.

[0125] Step 202: Based on the dialogue role to which the question text belongs, obtain the target historical dialogue content and target historical dialogue summary corresponding to the dialogue role. The target historical dialogue content includes the dialogue content of at least one round of first dialogue before the question text was obtained. The target historical dialogue summary includes a summary of the dialogue content of at least one round of second dialogue before the first dialogue.

[0126] It is understandable that, for the execution device, different users can engage in human-computer dialogue with the execution device based on their own user devices, enabling the execution device to converse with different users based on the target model. Therefore, the execution device can actually converse with multiple different dialogue roles (i.e., users), and the execution device maintains the dialogue content with each dialogue role separately. Here, a dialogue role refers to the role that engages in dialogue with the execution device, such as a user or other intelligent agent. For the same dialogue role, the questions asked by that role at different times often have a certain correlation; for example, questions asked consecutively by a dialogue role within a certain period are often related to a specific topic. Therefore, to facilitate the execution device in outputting more accurate answers when processing the question text provided by the dialogue role through the target model, this application can provide the target model with historical dialogue content between the execution device and the same dialogue role (i.e., dialogue content from previous times) as reference information. The target model then processes the question text provided by the current dialogue role based on this historical dialogue content.

[0127] In this application, after obtaining the question text, the execution device obtains the target historical dialogue content and target historical dialogue summary under the same dialogue role (i.e., the dialogue role that provided the question text) to which the question text belongs. The target historical dialogue content includes the dialogue content of at least one round of first dialogue preceding the question text, that is, the dialogue content between the execution device and the dialogue role preceding the question text. Furthermore, the dialogue content of the at least one round of first dialogue included in the target historical dialogue content is adjacent to the dialogue round to which the question text belongs. The target historical dialogue summary includes a summary of the dialogue content preceding the target historical dialogue content; that is, the target historical dialogue summary is not the complete historical dialogue content, but a summary of the dialogue content. Specifically, the target historical dialogue summary includes a summary of the dialogue content of at least one round of second dialogue, and the at least one round of second dialogue precedes the first dialogue. "Second dialogue preceding first dialogue" also means that the second dialogue precedes the first dialogue in time; the first dialogue occurred only after the second dialogue between the execution device and the dialogue role. Therefore, the target historical dialogue summary has a smaller data volume than the original historical dialogue content and can summarize the main semantics of the original historical dialogue content. In this application, a round of dialogue includes a question text provided by the user and an answer text returned by the execution device. Of course, if the user provides a reference text, a round of dialogue may include the question text, the reference text, and the answer text.

[0128] For example, if the execution device obtains the aforementioned question text during the 10th round of dialogue with the dialogue character, then the target historical dialogue content could include the dialogue content between the execution device and the dialogue character from rounds 6 to 9, and the target historical dialogue summary could include a summary of the dialogue content between the execution device and the dialogue character from rounds 1 to 5. The target historical dialogue summary could be a shorter summary obtained by processing the dialogue content preceding the target historical dialogue content using a summary model. Alternatively, the target historical dialogue summary could be obtained by processing the dialogue content preceding the target historical dialogue content using a target model.

[0129] Optionally, the question text originates from the first session; the target historical dialogue content includes dialogue content from the second session. The first session may include one or more rounds of dialogue content. The second session may also include one or more rounds of dialogue content; for example, the second session may include dialogue content from at least one round of the first dialogue mentioned above. In other words, the question text and the historical dialogue content used to process the question text can originate from different sessions; that is, the selection of historical dialogue content is not limited to the current session.

[0130] Step 203: Based on the target historical dialogue summary and target historical dialogue content, process the question text through the target model to obtain the answer text.

[0131] In this application, the execution device can use the target historical dialogue summary and target historical dialogue content as reference information, and then input the target historical dialogue summary, target historical dialogue content, and question text into the target model. The target model processes the question text to obtain the answer text output by the target model. Specifically, the answer text can be the text directly returned to the user in the dialogue. Alternatively, the answer text can be one or more texts to be recommended generated by the target model. The execution device can process one or more texts to be recommended before returning them to the user in the dialogue (e.g., the execution device selects one text to return to the user).

[0132] In other words, the execution device does not directly use all historical dialogue content with the user to process the currently acquired question text. Instead, it divides all historical dialogue content into two parts: one part consists of question-and-answer dialogue content that is closer in time to the current question text, and the other part consists of question-and-answer dialogue content that is farther in time. Furthermore, the question-and-answer dialogue content that is farther in time to the current question text needs to undergo text summarization extraction, and the extracted dialogue content summary is used as the text content for processing the question text. Thus, the dialogue content that the execution device actually uses to process the current question text includes both some recent question-and-answer dialogue content itself and summaries of older question-and-answer dialogue content.

[0133] In this application, the target historical dialogue content that is closer to the question text in time can be considered as short-term memory, while the content corresponding to the target historical dialogue summary that is farther away from the question text in time can be considered as long-term memory.

[0134] Since this approach selects the complete historical dialogue content that is closer in time to the current question text (i.e., the complete short memory) and summaries of historical dialogue content from even earlier times (i.e., summaries of long memory) as input to the target model, rather than selecting all complete historical dialogue content, it can effectively reduce the length of the input text to the target model and improve the inference speed of the target model.

[0135] Furthermore, regarding the question text currently acquired by the execution device, the closer the question-and-answer dialogue between the execution device and the user is temporally to the current question text, the greater the relevance between the dialogue and the question text; conversely, the further back in time the dialogue is from the current question text, the smaller the relevance, and there may even be no relevance. Therefore, by extracting summaries of these historical dialogues as input to the target model, redundant information input to the target model can be effectively reduced, thereby improving the accuracy of the target model. That is, in some cases, compared to inputting all historical dialogue content into the target model, this solution achieves higher inference accuracy by using summaries of older historical dialogues as input.

[0136] In summary, this approach extracts summaries of question-and-answer dialogues that are temporally distant from the current question text and uses these summaries to assist in processing the question text. This does not affect the accuracy of the target model in processing the question text, and because the amount of content that the target model needs to process is greatly reduced, it can effectively improve the processing efficiency of the target model.

[0137] Optionally, when there is a large amount of historical question-and-answer dialogue content between the execution device and the user, the execution device can obtain multiple pre-generated historical dialogue summaries, which correspond to dialogue content in different historical time periods.

[0138] Then, based on the relevance of multiple historical dialogue summaries to the question text, a target historical dialogue summary is determined from among the multiple historical dialogue summaries. The target historical dialogue summary includes one or more of the aforementioned multiple historical dialogue summaries.

[0139] For example, the execution device can use a target model or other feature extraction models to extract features from the historical dialogue summary and the question text, respectively, and compare the distance between the features of the historical dialogue summary and the features of the question text, thereby selecting one or more historical dialogue summaries with the closest distance as the target historical dialogue summary. In this way, the execution device can select one or more historical dialogue summaries with the highest relevance to the question text from multiple historical dialogue summaries to assist the target model in processing the question text.

[0140] In other words, for the historical question-and-answer dialogue content between the execution device and the user from a long time ago, the execution device performs summary extraction on the question-and-answer dialogue content from different historical time periods, thereby obtaining multiple historical dialogue summaries. In this way, the execution device can select the historical dialogue summaries that are most relevant to the question text from multiple historical dialogue summaries, thereby avoiding the use of all historical dialogue summaries to assist in processing the question text, reducing the amount of content that the target model needs to process, and improving the processing efficiency of the target model.

[0141] For example, please refer to Figure 3 , Figure 3 This application provides a schematic diagram for determining a target historical dialogue summary based on historical dialogue content. For example... Figure 3 As shown, during the entire dialogue between the user and the execution device, the content of their conversation, in chronological order, is as follows: Dialogue Content 1, Dialogue Content 2, Dialogue Content 3, Target Historical Dialogue Content, and the question text currently acquired by the execution device. The Target Historical Dialogue Content is the dialogue content most recent to the current question text in time, while Dialogue Contents 1 through 3 are dialogue contents further removed from the current question text in time. Furthermore, Dialogue Contents 1 through 3, as well as the Target Historical Dialogue Content, can all include one or more rounds of dialogue between the user and the execution device.

[0142] For dialogue content 1 to dialogue content 3, the device performs summary extraction, thereby extracting shorter historical dialogue summaries 1 to 3. Then, the device identifies the historical dialogue summary with the highest relevance to the question text from historical dialogue summaries 1 to 3 as the target historical dialogue summary, for example, identifying historical dialogue summary 2 as the target historical dialogue summary.

[0143] Furthermore, there are several ways for the execution device to generate historical dialogue summaries based on historical dialogue content.

[0144] Implementation method 1: The execution device selects the content to be generated as a historical dialogue summary based on the dialogue round.

[0145] In implementation method 1, the execution device can select which rounds of historical dialogue content need to be combined to generate a historical dialogue summary based on the number of dialogue rounds. For example, the execution device can generate a historical dialogue summary every N rounds of dialogue content. In this case, each time the execution device generates a historical dialogue summary, it is based on all the content of the N rounds of historical dialogue content.

[0146] Therefore, a single historical dialogue summary among the multiple historical dialogue summaries generated by the execution device corresponds to N rounds of dialogue content, where N is an integer greater than or equal to 1. That is, a historical dialogue summary is generated from N rounds of dialogue content.

[0147] For example, please refer to Figure 4 , Figure 4 This application provides a schematic diagram of generating historical dialogue summaries based on historical dialogue content. For example... Figure 4 As shown, the execution device can generate a historical dialogue summary every 5 rounds of dialogue. Throughout the entire dialogue between the user and the execution device, the content of their conversation, in chronological order, is as follows: content of round 1 to round 10, target historical dialogue content, and question text. The execution device can generate historical dialogue summary 1 based on the content of round 1 to round 5, and historical dialogue summary 2 based on the content of round 6 to round 10.

[0148] Implementation method 2: The execution device selects the content to be generated as a historical dialogue summary based on the length of the historical dialogue content.

[0149] In implementation method 2, the execution device can select which rounds of historical dialogue content need to be combined to generate a historical dialogue summary based on the length of the historical dialogue content. For example, the execution device can continuously determine whether the total length of the historical dialogue content for which no historical dialogue summary has been generated reaches a preset length. If the total length of the historical dialogue content for which no historical dialogue summary has been generated reaches the preset length, then historical dialogue content can be selected to generate a historical dialogue summary, and the total length of the historical dialogue content used to generate the historical dialogue summary does not exceed the preset length. The length of the historical dialogue content can be measured by the number of tokens included in the historical dialogue content; therefore, the aforementioned preset length can be, for example, 500 tokens or 1000 tokens.

[0150] Therefore, the total length of the dialogue content corresponding to a single historical dialogue summary among the multiple historical dialogue summaries generated by the execution device is no greater than the preset length.

[0151] For example, please refer to Figure 5 , Figure 5 This application provides another schematic diagram for generating historical dialogue summaries based on historical dialogue content. For example... Figure 5As shown, the execution device can have a preset length of 1000 tokens to ensure that the total length of the dialogue content used to generate a single historical dialogue summary does not exceed 1000 tokens. Throughout the entire dialogue between the user and the execution device, the content of the dialogue, in chronological order, is as follows: Round 1 dialogue content - Round 10 dialogue content, target historical dialogue content, and question text. If the total length of Round 1 to Round 4 dialogue content is 998 tokens, and the total length of Round 1 to Round 5 dialogue content is greater than 1000 tokens, the execution device can generate historical dialogue summary 1 based on Round 1 to Round 4 dialogue content. Furthermore, if the total length of Round 5 to Round 10 dialogue content is exactly 1000 tokens, the execution device can generate historical dialogue summary 2 based on Round 5 to Round 10 dialogue content.

[0152] The above describes the process of determining the target historical dialogue summary and the target historical dialogue content. The following will combine the target historical dialogue summary and the target historical dialogue content to explain how to further improve the inference speed of the target model.

[0153] Understandably, during the target model's processing of the input text (i.e., the target historical dialogue summary, the target historical dialogue content, and the question text), the target model extracts features from each token (e.g., the Q-vector, K-vector, and V-vector of each token) one by one, using the tokens in the input text as the smallest granularity. When extracting features from a token, the model relies on the features of all tokens preceding it, thus obtaining the features of the input text. Finally, the target model processes the extracted features of the input text to obtain the output answer text. The process of the target model extracting features from the input text is typically called the prefill stage, while the stage where the target model processes the features of the input text is called the generation stage.

[0154] Optionally, based on the principle of processing input text by the target model, in order to improve the inference speed of the target model in practical applications, this application may store the features of the target historical dialogue summary and the features of the target historical dialogue content that have been extracted in advance, and directly reuse the stored features when the target model needs to process the target historical dialogue summary and the target historical dialogue content, thereby saving the process of extracting features and achieving the purpose of trading storage space for time.

[0155] For example, please refer to Figure 6 , Figure 6 This application provides a schematic diagram illustrating how reusing features from a feature library can accelerate target model inference. For example... Figure 6As shown, in step 203 above, the execution device can first obtain the first feature corresponding to the target historical dialogue summary and the second feature corresponding to the target historical dialogue content from the feature library. The first and second features are extracted based on the target model. The feature library is used to persistently store the features extracted based on the target model. For example, the features included in the feature library are all persistently stored on a persistent storage medium such as a hard disk. Optionally, when the target model is a large language model, the first and second features are specifically a KV cache.

[0156] In this scheme, a storage medium is used to pre-store the features of historical dialogues. This allows the features needed for a specific round of historical dialogues to be read directly from the storage medium without recalculating the features of the historical dialogues. This achieves feature lookup instead of calculation and improves the inference speed of the target model.

[0157] For example, by offloading the KV cache of historical sessions to storage media for persistence, the KV cache can be reused in subsequent sessions through similarity retrieval, which can effectively utilize AI native storage to improve the inference speed of the model.

[0158] In this way, based on the first and second features, the execution device can process the question text using the target model. Specifically, the execution device can use the first feature as the feature corresponding to the target historical dialogue summary and the second feature as the feature of the target historical dialogue content, thus directly reusing the first and second features without needing to perform feature extraction processing on the input target historical dialogue summary and target historical dialogue content through the target model. That is, based on the first and second features, the execution device performs feature extraction on the question text through the target model, and finally performs further processing on the first feature, the second feature, and the extracted features of the question text (i.e., the execution generation stage) to obtain the answer text.

[0159] In this application, since both the target historical dialogue summary and the target historical dialogue content are obtained based on previous dialogue content, by performing feature extraction in advance and storing the extracted features in a feature library, the stored features can be directly reused in the actual reasoning process, thereby eliminating the need to perform feature extraction on the historical dialogue content, realizing query-based calculation, and effectively improving the reasoning speed of the target model.

[0160] In one possible implementation, the first feature includes the original features obtained by performing feature extraction on the dialogue content of at least one round of the second dialogue; or, the first feature includes the summary features obtained by performing feature extraction on the summary of the dialogue content of at least one round of the second dialogue.

[0161] In other words, the first feature stored in the feature library can be either a feature of the target historical dialogue summary itself or a feature of the original dialogue content corresponding to the target historical dialogue summary. Generally speaking, the target historical dialogue summary is obtained by summarizing and generalizing the original dialogue content, so the features of the target historical dialogue summary itself focus more on the macro-level information of the dialogue; while the features of the original dialogue content corresponding to the target historical dialogue summary focus more on the detailed information of the dialogue.

[0162] The above describes the process of retrieving and reusing pre-stored features from a feature library when processing problematic text. For ease of understanding, the following will describe how to store features in a feature library.

[0163] First, for dialogue content for which historical dialogue summaries have not yet been generated, since this part of the dialogue content will later be used as the target historical dialogue content closest to the question text and input into the target model, the features of this part of the dialogue content can be directly stored in the feature library for later reuse. For example, assuming that historical dialogue summaries have been generated for dialogue content from round 1 to round 10, when the target model responds to the question text of round 11, it will generate a KV cache for the question text and answer text of round 11. At this time, the KV cache corresponding to the question text and answer text of round 11 can be stored in the feature library. Then, when the target model responds to the question text of round 12, the question text and answer text of round 11 are used as the aforementioned target historical dialogue content, and the KV cache corresponding to the question text and answer text of round 11 can be directly retrieved from the feature library for reuse.

[0164] Furthermore, for the generated historical dialogue summaries, since not all historical dialogue summaries corresponding to the same dialogue role need to be used, and storing the features in the feature library requires a large amount of storage space, in order to reduce storage costs, this application may calculate a memory score for the historical dialogue summaries and determine how to use and store the features corresponding to the historical dialogue summaries based on the memory score of the historical dialogue summaries.

[0165] For example, regarding the aforementioned target historical dialogue summary, the target historical dialogue summary includes a first historical dialogue summary, which includes a summary of the dialogue content of a third dialogue, and the third dialogue is a round of dialogue in at least one round of a second dialogue. That is, the target historical dialogue summary may include one or more historical dialogue summaries, and the first historical dialogue summary is one of the historical dialogue summaries included in the target historical dialogue summary.

[0166] The first feature includes the original features of the third dialogue or the summary features of the third dialogue. The original features of the third dialogue include features obtained by performing feature extraction on the dialogue content of the third dialogue, and the summary features of the third dialogue include features obtained by performing feature extraction on the summary of the dialogue content of the third dialogue.

[0167] In this method, the execution device can determine a memory score of a first historical dialogue summary, which indicates the usage of the first historical dialogue summary. Generally, the more frequently the first historical dialogue summary is used, the higher its memory score; conversely, the less frequently it is used, the lower its memory score.

[0168] Then, based on the memory score, the execution device determines the storage status of the original features or summary features of the third dialogue in the feature library.

[0169] In one possible implementation, the storage status of the original features or summary features of the third dialogue in the feature library is determined based on the memory score, specifically including:

[0170] If the memory score is within the first range and the original features of the third dialogue are not stored in the feature library, then the original features of the third dialogue are obtained by performing feature extraction on the dialogue content of the third dialogue through the target model, and the original features of the third dialogue are stored in the feature library.

[0171] In one possible implementation, the storage status of the original features or summary features of the third dialogue in the feature library is determined based on the memory score, specifically including:

[0172] If the memory score is in the second range and the feature library does not store the summary features of the third dialogue, then the summary features of the third dialogue are obtained by performing feature extraction on the summary of the first historical dialogue through the target model, and the summary features of the third dialogue are stored in the feature library, wherein the value in the second range is less than the value in the first range.

[0173] In other words, if the memory score corresponding to the summary of the third dialogue content is relatively high, it means that the summary of the third dialogue content is used more frequently. Therefore, the original features of the third dialogue can be stored in the feature library, allowing more attention to the detailed information of the third dialogue when processing the question text. If the memory score corresponding to the summary of the third dialogue content is average, it means that the summary of the third dialogue content is used less frequently. Therefore, the summary features of the third dialogue can be stored in the feature library, allowing more attention to the macroscopic information of the third dialogue when processing the question text.

[0174] In one possible implementation, if the memory score falls within the second range, and the feature library stores the original features of the third dialogue, then those original features are deleted from the feature library. That is, the feature library only stores one set of features corresponding to the third dialogue, such as the original features or the summary features of the third dialogue, rather than storing both simultaneously, thus reducing storage space overhead. In other words, the features stored in the feature library by the execution device for a historical dialogue summary will differ depending on the memory score of that summary. If the memory score of the historical dialogue summary is high (i.e., within the first range), it indicates that the target historical dialogue summary is used frequently. In this case, the features corresponding to the original dialogue content of the historical dialogue summary can be stored in the feature library, thereby improving the accuracy of the target model in processing the problem text. If the memory score of the historical dialogue summary is not very high (i.e., within the second range), it indicates that the historical dialogue summary is not used frequently. In this case, the features corresponding to the target historical dialogue summary can be stored in the feature library, thereby saving storage space as much as possible.

[0175] In this scheme, the usage of historical dialogue summaries is measured by calculating a memory score. Based on the memory score, it is possible to determine whether to store features of the historical dialogue summaries or features of the original dialogue content of the historical dialogue summaries. This allows for a balance between improving the accuracy of the target model and saving storage space, thereby enhancing the practicality of the scheme.

[0176] In one possible implementation, the storage status of the original features or summary features of the third dialogue in the feature library is determined based on the memory score, including:

[0177] If the memory score falls within the third range: if the original features of the third dialogue are stored in the feature library, then the original features of the third dialogue are deleted from the feature library; or, if the summary features of the third dialogue are stored in the feature library, then the summary features of the third dialogue are deleted from the feature library. The values ​​in the third range are less than the values ​​in the second range, and the values ​​in the second range are less than the values ​​in the first range.

[0178] In other words, when the memory score corresponding to the summary of the dialogue content of the third dialogue is in the third range, it indicates that the memory score is low, which means that the summary of the dialogue content of the third dialogue is not used very frequently. In this case, the feature corresponding to the third dialogue in the feature library can be deleted to save storage space as much as possible.

[0179] In practical applications, for any generated historical dialogue summary, the execution device can periodically calculate the memory score of that historical dialogue summary, and then determine how to cache and use the features corresponding to that historical dialogue summary based on the memory score. That is, the memory score of the historical dialogue summary is determined periodically, for example, the execution device determines the memory score every 5 hours, 10 hours, or once a day.

[0180] Optionally, the memory score of the target historical dialogue summary is determined based on at least one of the following: the total number of times the target historical dialogue summary is used, the number of times the target historical dialogue summary is used within a preset time period, the time interval since the last use of the target historical dialogue summary, the duration of the target historical dialogue summary's generation, the number of consecutive uses of the first historical dialogue summary, the number of times the original features of the first historical dialogue summary are reused, the number of times the summary features of the first historical dialogue summary are reused, the length of the original dialogue content corresponding to the first historical dialogue summary, the length of the first historical dialogue summary, and the time interval since the last change in the range of the memory score. The preset time period can specifically be a period of time prior to the time node for calculating the memory score, such as 5 hours, 1 day, or 1 week prior to the time node for calculating the memory score. That is, the number of times the target historical dialogue summary is used within the preset time period can be understood as the number of times the target historical dialogue summary is used in the most recent short period. The duration of the target historical dialogue summary's generation refers to the length of time between the time node for generating the target historical dialogue summary and the time node for calculating the memory score. The number of consecutive uses of the first historical dialogue summary refers to the number of times the first historical dialogue summary is used consecutively when processing different question texts provided by the same user. The number of times the original features of the first historical dialogue summary are reused refers to the number of times the original features of the first historical dialogue summary (i.e., features extracted based on the original dialogue content corresponding to the first historical dialogue summary) are reused in the feature library when processing the question text. The number of times the summary features of the first historical dialogue summary are reused refers to the number of times the summary features of the first historical dialogue summary (i.e., features obtained by performing feature extraction on the first historical dialogue summary) are reused in the feature library when processing the question text. The time interval since the last change in the range of the memory score refers to the time interval from the point in time when the range of the memory score was last changed to the point in time when the memory score is recalculated.

[0181] For example, in one possible instance, the memory score can be specifically calculated based on the following Formula 1.

[0182]

[0183] Where S is the memory score; L is the total number of times the historical dialogue summary is used; F Tis the number of times the historical dialogue summary is used within a preset time period; T last is the time interval since the last use of the historical dialogue summary; T gen is the generated duration of the historical dialogue summary.

[0184] Of course, the memory score can also be calculated using other methods (such as fitting through a neural network), as long as it can measure the frequency of use of the historical dialogue summary. This application does not make specific limitations in this regard.

[0185] Exemplarily, please refer to Figure 7 , Figure 7 is a schematic diagram provided by this application for determining how to process the historical dialogue summary based on the memory score. As Figure 7 shown, after calculating the memory score of the historical dialogue summary based on the total number of uses, the number of uses in the recent short period, the duration of the last use interval, and the generated duration of the historical dialogue summary, it is possible to determine how to process the historical dialogue summary based on the memory score.

[0186] 1. When S < S1, the content to be saved is the historical dialogue summary and the original dialogue content of the historical dialogue summary. Therefore, if there are features of the historical dialogue summary or the original dialogue content in the feature library, the features of the historical dialogue summary or the original dialogue content need to be deleted. In addition, when the historical dialogue summary needs to be used, since the corresponding features are not stored in the feature library, the content actually reused is the historical dialogue summary. Here, S is the memory score of the historical dialogue summary, S1 is a preset value, and S < S1 means that the memory score is in the above-mentioned third range.

[0187] 2. When S1 ≤ S < S2, the content to be saved is the historical dialogue summary, the features of the historical dialogue summary, and the original dialogue content of the historical dialogue summary. Therefore, if there are no features of the historical dialogue summary in the feature library, the features of the historical dialogue summary need to be generated and stored in the feature library. If there are features of the original dialogue content in the feature library, the features of the original dialogue content need to be deleted in the feature library. In addition, when the historical dialogue summary needs to be used, since the features of the historical dialogue summary are stored in the feature library, the content actually reused is the features of the historical dialogue summary. Here, S is the memory score of the historical dialogue summary, S1 and S2 are preset values, and S1 ≤ S < S2 means that the memory score is in the above-mentioned second range.

[0188] 3. When S≥S2, the content that needs to be saved includes the historical dialogue summary, the features of the historical dialogue summary, the original dialogue content of the historical dialogue summary, and the features of the original dialogue content. Therefore, if the feature library does not contain features of the original dialogue content, the features of the original dialogue content need to be generated and stored in the feature library. If the feature library contains features of the historical dialogue summary, the features of the historical dialogue summary need to be deleted from the feature library. Furthermore, when the historical dialogue summary needs to be used, since the feature library stores the features of the original dialogue content, the content actually reused is the features of the original dialogue content. Here, S≥S2 represents the memory score falling within the first range mentioned above.

[0189] Optionally, the target historical dialogue summary may specifically include at least two historical dialogue summaries, and the at least two historical dialogue summaries correspond to dialogue content in different historical time periods.

[0190] After obtaining the first and second features mentioned above, the execution device can determine the first sub-feature within the first feature. The first sub-feature includes features corresponding to some tokens in the target historical dialogue digest. Specifically, the first feature is actually a feature of at least two historical dialogue digests included in the target historical dialogue digest. The first sub-feature is a feature corresponding to some tokens in at least two historical dialogue digests.

[0191] Then, based on the first feature, the execution device re-extracts the second sub-feature corresponding to the determined portion of the tokens using the target model. That is, the execution device recalculates the second sub-feature corresponding to this portion of the tokens based on the first feature.

[0192] Secondly, the execution device updates the first sub-feature in the first feature to the second sub-feature, thus obtaining the updated first feature.

[0193] In this way, the execution device can process the problem text through the target model based on the updated first and second features.

[0194] Understandably, in the first feature, the features corresponding to at least two historical dialogue summaries are extracted independently based on their respective historical dialogue summaries. Since the target model calculates the features of subsequent tokens based on the features of all preceding tokens in the input text during conventional feature extraction, directly concatenating the features extracted from at least two independently (i.e., the features corresponding to at least two historical dialogue summaries) can easily lead to a lack of cross-attention between features, making it difficult for the target model to understand directly reused features. Therefore, after identifying some tokens in the target historical dialogue summaries, the features of these tokens can be recalculated based on the entire first feature, thus achieving the recalculation of some key tokens. In this way, in at least two historical dialogue summaries, the features of tokens in the later historical dialogue summaries are actually calculated based on all preceding historical dialogue summaries, rather than being calculated based on independent historical dialogue summaries. That is, the second sub-feature is actually calculated by combining the features of different historical dialogue summaries. This ensures that the second sub-feature can construct cross-attention between the features of different historical dialogue summaries, so that the updated first feature will no longer have the phenomenon of missing cross-attention between the features corresponding to different historical dialogue summaries. This makes it easier for the target model to understand the reused features and improves the accuracy of the target model.

[0195] In other words, when the target historical dialogue summary includes at least two historical dialogue summaries, by recalculating the features of some tokens based on the features of the entire target historical dialogue summary, cross-attention between the features of different historical dialogue summaries can be achieved, making it easier for the target model to understand the reused features and improving the accuracy of the target model.

[0196] To facilitate understanding, the execution process of the inference acceleration method provided in this application will be explained in detail below with specific examples. Please refer to... Figure 8 , Figure 8 This is a schematic diagram illustrating the execution flow of a reasoning acceleration method provided in this application. Figure 8 As shown, the execution flow of the inference acceleration method includes the following steps 801-809. It should be noted that steps 801-804 can be executed before the inference process using the target model, while steps 805-809 are the process of executing the inference using the target model.

[0197] Step 801: Update the short memory window based on the dialogue content.

[0198] In this application, to facilitate determining the need to input the target model's historical dialogue content along with the user-provided question text, the execution device can simultaneously maintain a short memory window and a long memory window. The short memory window records the complete historical dialogue content (i.e., the target historical dialogue content described in the above embodiments), while the long memory window records a summary of the historical dialogue content (i.e., the target historical dialogue summary described in the above embodiments).

[0199] As the user and the execution device continue their conversations, the execution device can record the most recent conversations with the user in a short memory window. That is, the short memory window records the latest conversations between the execution device and the user.

[0200] Step 802: When the number of historical dialogue rounds in the short memory window reaches the window limit, or the length of the historical dialogue reaches the length limit, summarize the historical dialogue in the short memory window to obtain a historical dialogue summary.

[0201] For example, please refer to Figure 9 , Figure 9 This application provides a schematic diagram of a short memory window and a long memory window for maintaining an execution device. (See attached diagram.) Figure 9 As shown, the execution device maintains both a short memory window and a long memory window. The short memory window records the most recent rounds of dialogue between the execution device and the user. When the number of dialogue rounds stored in the short memory window reaches m rounds (m being a preset value such as 10 or 15), or when the total length of the dialogue content stored in the short memory window reaches a preset length, the execution device can extract a summary of the earliest part of the dialogue in the short memory window and store the generated historical dialogue summary in the long memory pool. For example, when the number of dialogue rounds stored in the short memory window reaches 10 rounds, the execution device extracts a summary of the first 5 rounds of dialogue content to generate a dialogue summary. Furthermore, after the execution device extracts a summary of part of the dialogue content in the short memory window, it clears the extracted dialogue content from the short memory window.

[0202] For example, please refer to Figure 10 , Figure 10 This application provides a flowchart illustrating a problem text processing approach based on short and long memory. For example... Figure 10 As shown, for the original historical dialogue content within the short memory window, the execution device can employ a smaller summary model (i.e., Figure 10 A small model is used to perform the task of summarizing historical dialogues, thereby extracting historical dialogue summaries.

[0203] Step 803: Store the historical dialogue summary in the long memory pool, perform feature extraction on the historical dialogue summary, and store the obtained summary features in the vector database.

[0204] For historical dialogue summaries that need to be stored in a long memory pool, the execution device can use a small text feature extraction model to extract features from the historical dialogue summaries, thereby obtaining summary features. The summary features corresponding to the historical dialogue summaries are stored in a vector database to facilitate subsequent retrieval of historical dialogue summaries.

[0205] Step 804: Calculate the memory score of the historical dialogue summary, and delete the corresponding KV Cache in the feature library according to the memory score.

[0206] Furthermore, for historical dialogue summaries in the long memory pool, the background scheduling process of the execution device will periodically trigger the calculation of the memory score of the historical dialogue summaries, and store or delete the corresponding KV cache in the feature library based on the memory score. For details on how to store or delete the corresponding KV cache in the feature library based on the memory score, please refer to [link to relevant documentation]. Figure 7 The corresponding embodiments will not be described in detail here.

[0207] Step 805: Obtain the question text provided by the user.

[0208] Step 806: Retrieve relevant historical dialogue summaries for the question text from the long memory pool, and store the retrieved historical dialogue summaries in the long memory pool.

[0209] In this step, the long memory window is used to record historical dialogue summaries related to the question text extracted from the long memory pool. Specifically, after obtaining the question text, the execution device can select a portion of the dialogue summaries related to the question text from the long memory pool based on the question text, and store the selected dialogue summaries in the long memory window. For example, the execution device selects n dialogue summaries from the long memory pool and stores them in the long memory window, where n can be a preset value such as 3 or 5.

[0210] Specifically, when retrieving relevant historical dialogue summaries for the question text, the features of the question text can be extracted based on the text feature extraction model described above. Then, by calculating the similarity between the features of the question text and the features of the historical dialogue summaries in the vector database, one or more historical dialogue summaries with the highest relevance to the question text can be selected.

[0211] Step 807: The retrieved historical dialogue summary, the historical dialogue content in the short memory window, and the concatenated question text are used as input to the target model.

[0212] It should be noted that if the question text provided by the user also has a corresponding reference text (i.e., the text that the target model refers to when answering the question text), the reference text also needs to be concatenated and input into the target model.

[0213] Step 808: When processing the problem text using the target model, reuse the features of the historical dialogue summary stored in the feature library and the features of the historical dialogue content in the short memory window.

[0214] Specifically, since the feature library may store features of historical dialogue summaries, when using the target model to process question text, features of historical dialogue summaries can be obtained from the feature library and reused, thereby avoiding re-executing the feature extraction process.

[0215] It should be noted that before reusing the features of the historical dialogue summary, some feature fragments in the obtained historical dialogue summary can also be identified and recalculated to improve the inference accuracy of the target model.

[0216] Step 809: Complete the reasoning process of the target model.

[0217] Once the target model has finished processing the question text and obtained the answer text, it means that the reasoning process of the target model has been completed.

[0218] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0219] Please see Figure 11 , Figure 11 This is a schematic diagram of a reasoning acceleration device provided in this application. Figure 11 As shown, the inference acceleration device includes: an acquisition module 1101 for acquiring question text, which indicates a question to be answered; a processing module 1102 for acquiring target historical dialogue content and target historical dialogue summary corresponding to the dialogue role to which the question text belongs, wherein the target historical dialogue content includes the dialogue content of at least one round of first dialogue before the question text is acquired, and the target historical dialogue summary includes a summary of the dialogue content of at least one round of second dialogue before the first dialogue; the processing module 1102 is also used to process the question text through a target model based on the target historical dialogue summary and target historical dialogue content to obtain answer text.

[0220] In one possible implementation, the processing module 1102 is further configured to: query the feature library to obtain a first feature corresponding to the target historical dialogue summary and a second feature corresponding to the target historical dialogue content, wherein the first feature and the second feature are extracted based on the target model;

[0221] Based on the first and second features, the problem text is processed through the target model.

[0222] In one possible implementation, the first feature includes the original features obtained by performing feature extraction on the dialogue content of at least one round of the second dialogue; or,

[0223] The first feature includes the summary features obtained by performing feature extraction on a summary of the dialogue content of at least one round of the second dialogue.

[0224] In one possible implementation, the target historical dialogue summary includes a first historical dialogue summary, which includes a summary of the dialogue content of a third dialogue, wherein the third dialogue is one round of dialogue in at least one round of a second dialogue. The first feature includes either the original feature of the third dialogue or the summary feature of the third dialogue. The original feature of the third dialogue includes features obtained by performing feature extraction on the dialogue content of the third dialogue. The summary feature of the third dialogue includes features obtained by performing feature extraction on the summary of the dialogue content of the third dialogue. The processing module 1102 is further configured to:

[0225] Determine the memory score of the first historical dialogue summary, which is used to indicate the usage of the first historical dialogue summary;

[0226] Based on the memory score, determine the storage status of the original features or summary features of the third dialogue in the feature library.

[0227] In one possible implementation, the processing module 1102 is further configured to: if the memory score is within a first range and the original features of the third dialogue are not stored in the feature library, then perform feature extraction on the dialogue content of the third dialogue through the target model to obtain the original features of the third dialogue, and store the original features of the third dialogue in the feature library.

[0228] In one possible implementation, the processing module 1102 is further configured to: if the memory score is in the second range and the feature library does not store the summary features of the third dialogue, then perform feature extraction on the first historical dialogue summary through the target model to obtain the summary features of the third dialogue, and store the summary features of the third dialogue in the feature library, wherein the value in the second range is less than the value in the first range.

[0229] In one possible implementation, the processing module 1102 is further configured to: if the feature library stores the original features of the third dialogue, delete the original features of the third dialogue from the feature library.

[0230] In one possible implementation, the processing module 1102 is further configured to:

[0231] When the memory score is in the third range:

[0232] If the original features of the third dialogue are stored in the feature library, then delete the original features of the third dialogue from the feature library; or,

[0233] If the feature library stores summary features of the third dialogue, then delete the summary features of the third dialogue from the feature library; wherein, the value in the third range is less than the value in the second range, and the value in the second range is less than the value in the first range.

[0234] In one possible implementation, the memory score is determined based on at least one of the following: the total number of times the first historical dialogue summary is used, the number of times the first historical dialogue summary is used within a preset time period, the time interval since the last use of the first historical dialogue summary, the duration since the first historical dialogue summary was generated, the number of consecutive uses of the first historical dialogue summary, the number of times the original features of the first historical dialogue summary are reused, the number of times the summary features of the first historical dialogue summary are reused, the length of the original dialogue content corresponding to the first historical dialogue summary, the length of the first historical dialogue summary, and the time interval since the last change in the range of the memory score.

[0235] In one possible implementation, the memory score is determined periodically.

[0236] In one possible implementation, the target historical dialogue summary includes at least two historical dialogue summaries, which correspond to the dialogue content of dialogues in different historical time periods.

[0237] The processing module 1102 is further configured to: determine a first sub-feature in the first feature, the first sub-feature including features corresponding to some tokens in the target historical dialogue summary; based on the first feature, re-extract the second sub-feature corresponding to some tokens through the target model; update the first sub-feature in the first feature to the second sub-feature to obtain the updated first feature; and process the question text through the target model based on the updated first feature and second feature.

[0238] In one possible implementation, the question text comes from the first session; the target historical dialogue content includes dialogue content from the second session.

[0239] In one possible implementation, the acquisition module 1101 is further configured to acquire multiple historical dialogue summaries, which correspond to dialogue content in different historical time periods; the processing module 1102 is further configured to determine the target historical dialogue summary among the multiple historical dialogue summaries based on the relevance of the multiple historical dialogue summaries to the question text.

[0240] Both the acquisition module 1101 and the processing module 1102 can be implemented in software or in hardware. For example, the implementation of the processing module 1102 will be described below. Similarly, the implementation of the acquisition module 1101 can be referenced to that of the processing module 1102.

[0241] As an example of a software functional unit, processing module 1102 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 1102 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0242] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0243] As an example of a hardware functional unit, the processing module 1102 may include at least one computing device, such as a server. Alternatively, the processing module 1102 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0244] The processing module 1102 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1102 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1102 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0245] Please see Figure 12 , Figure 12 A schematic diagram of the structure of a computing device provided in this application. Figure 12 The computing device 1200 shown can be used to execute the inference acceleration method provided in this embodiment. For example... Figure 12 As shown, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0246] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1202 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1202 may include a path for transmitting information between various components of the computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0247] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0248] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0249] The memory 1206 stores executable program code, and the processor 1204 executes this executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the aforementioned model inference acceleration method. That is, the memory 1206 stores instructions for executing the model inference acceleration method.

[0250] The communication interface 1208 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0251] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0252] Please see Figure 13 , Figure 13 This is a schematic diagram of a computing device cluster provided in this application. Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same instructions for executing inference acceleration methods.

[0253] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store a portion of the instructions for executing the inference acceleration method. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions for executing the inference acceleration method.

[0254] It should be noted that the memory 1206 in different computing devices 1200 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the inference acceleration device. That is, the instructions stored in the memory 1206 of different computing devices 1200 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0255] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 14 One possible implementation method is shown. Figure 14 This is a schematic diagram of another computing device cluster structure provided in this application. Figure 14 As shown, in a computing device cluster, two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1206 in computing device 1200A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 1206 in computing device 1200B stores instructions for executing the functions of the processing module.

[0256] It should be understood that Figure 14 The functions of the computing device 1200A shown can also be performed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be performed by multiple computing devices 1200.

[0257] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0258] Please refer to Figure 15 , Figure 15 This is a schematic diagram of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the above-described... Figure 2 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0259] Figure 15 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0260] In one embodiment, the computer-readable storage medium 1500 is provided using a signal bearer medium 1501. The signal bearer medium 1501 may include one or more program instructions 1502, which, when executed by one or more processors, can provide the above-mentioned... Figure 2 The described function or part of the function.

[0261] In some examples, the signal carrying medium 1501 may include a computer-readable medium 1503, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0262] In some embodiments, the signal-bearing medium 1501 may comprise a computer-recordable medium 1504, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1501 may comprise a communication medium 1505, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1501 may be transmitted by a wireless communication medium 1505 (e.g., a wireless communication medium conforming to the IEEE 1202.X standard or other transmission protocols).

[0263] One or more program instructions 1502 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1502 conveyed to the computing device via a computer-readable medium 1503, a computer-recordable medium 1504, and / or a communication medium 1505.

[0264] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0265] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0266] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0267] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for accelerating inference, characterized in that, include: Obtain the question text, which indicates the question to be answered; Based on the dialogue role to which the question text belongs, obtain the target historical dialogue content and target historical dialogue summary corresponding to the dialogue role. The target historical dialogue content includes the dialogue content of at least one round of first dialogue before the question text is obtained. The target historical dialogue summary includes a summary of the dialogue content of the at least one round of second dialogue, which is before the first dialogue. Based on the target historical dialogue summary and the target historical dialogue content, the question text is processed by the target model to obtain the answer text.

2. The method according to claim 1, characterized in that, The process of processing the question text using a target model based on the target historical dialogue summary and the target historical dialogue content includes: The first feature corresponding to the target historical dialogue summary and the second feature corresponding to the target historical dialogue content are obtained by querying the feature library. The first feature and the second feature are extracted based on the target model. The problem text is processed using the target model based on the first feature and the second feature.

3. The method according to claim 2, characterized in that, The first feature includes the raw features obtained by performing feature extraction on the dialogue content of the at least one round of the second dialogue; or, The first feature includes a summary feature obtained by performing feature extraction on a summary of the dialogue content of the at least one round of the second dialogue.

4. The method according to claim 2 or 3, characterized in that, The target historical dialogue summary includes a first historical dialogue summary, which includes a summary of the dialogue content of a third dialogue, wherein the third dialogue is one round of dialogue in the at least one round of the second dialogue. The first feature includes the original feature of the third dialogue or the summary feature of the third dialogue. The original feature of the third dialogue includes features obtained by performing feature extraction on the dialogue content of the third dialogue. The summary feature of the third dialogue includes features obtained by performing feature extraction on the summary of the dialogue content of the third dialogue. The method further includes: Determine the memory score of the first historical dialogue summary, the memory score being used to indicate the usage of the first historical dialogue summary; Based on the memory score, determine the storage status of the original features or summary features of the third dialogue in the feature library.

5. The method according to claim 4, characterized in that, The step of determining the storage status of the original features or summary features of the third dialogue in the feature library based on the memory score includes: If the memory score is within the first range and the original features of the third dialogue are not stored in the feature library, then the original features of the third dialogue are obtained by performing feature extraction on the dialogue content of the third dialogue through the target model, and the original features of the third dialogue are stored in the feature library.

6. The method according to claim 5, characterized in that, The step of determining the storage status of the original features or summary features of the third dialogue in the feature library based on the memory score includes: If the memory score is within the second range and the feature library does not store the summary features of the third dialogue, then the target model is used to perform feature extraction on the first historical dialogue summary to obtain the summary features of the third dialogue, and the summary features of the third dialogue are stored in the feature library, wherein the value in the second range is less than the value in the first range.

7. The method according to claim 6, characterized in that, The method further includes: If the feature library stores the original features of the third dialogue, delete the original features of the third dialogue from the feature library.

8. The method according to any one of claims 5-7, characterized in that, The step of determining the storage status of the original features or summary features of the third dialogue in the feature library based on the memory score includes: In the case where the memory score is in the third range: If the feature library contains the original features of the third dialogue, then delete the original features of the third dialogue from the feature library; or, If the feature library stores the summary features of the third dialogue, then the summary features of the third dialogue are deleted from the feature library; wherein, the value of the third range is less than the value of the second range, and the value in the second range is less than the value in the first range.

9. The method according to any one of claims 4-8, characterized in that, The memory score is determined based on at least one of the following: the total number of times the first historical dialogue summary is used, the number of times the first historical dialogue summary is used within a preset time period, the time interval since the last use of the first historical dialogue summary, the duration since the first historical dialogue summary was generated, the number of consecutive uses of the first historical dialogue summary, the number of times the original features of the first historical dialogue summary are reused, the number of times the summary features of the first historical dialogue summary are reused, the length of the original dialogue content corresponding to the first historical dialogue summary, the length of the first historical dialogue summary, and the time interval since the last change in the range of the memory score.

10. The method according to any one of claims 4-9, characterized in that, The memory score is determined periodically.

11. The method according to any one of claims 2-10, characterized in that, The target historical dialogue summary includes at least two historical dialogue summaries, and the at least two historical dialogue summaries correspond to the dialogue content of dialogues in different historical time periods; The process of processing the problem text using the target model based on the first feature and the second feature includes: Determine the first sub-feature in the first feature, wherein the first sub-feature includes features corresponding to some word tokens in the target historical dialogue summary; Based on the first feature, the second sub-feature corresponding to the partial token is re-extracted through the target model; The first sub-feature in the first feature is updated to the second sub-feature to obtain the updated first feature; The problem text is processed by the target model based on the updated first feature and the second feature.

12. The method according to any one of claims 1-11, characterized in that, The question text is from the first session; The target historical dialogue content includes dialogue content from the second session.

13. The method according to any one of claims 1-12, characterized in that, Obtaining the target historical dialogue summary includes: Obtain multiple historical dialogue summaries, which correspond to dialogue content in different historical time periods; Based on the relevance of the multiple historical dialogue summaries to the question text, the target historical dialogue summary is determined from the multiple historical dialogue summaries.

14. A reasoning acceleration device, characterized in that, include: The acquisition module is used to acquire the question text, which indicates the question to be answered; The processing module is used to obtain the target historical dialogue content and target historical dialogue summary corresponding to the dialogue role to which the question text belongs. The target historical dialogue content includes the dialogue content of at least one round of first dialogue before the question text is obtained. The target historical dialogue summary includes a summary of the dialogue content of the at least one round of second dialogue, where the second dialogue is before the first dialogue. The processing module is further configured to process the question text using a target model based on the target historical dialogue summary and the target historical dialogue content to obtain the answer text.

15. The apparatus according to claim 14, characterized in that, The processing module is further configured to: The first feature corresponding to the target historical dialogue summary and the second feature corresponding to the target historical dialogue content are obtained by querying the feature library. The first feature and the second feature are extracted based on the target model. The problem text is processed using the target model based on the first feature and the second feature.

16. The apparatus according to claim 15, characterized in that, The first feature includes the raw features obtained by performing feature extraction on the dialogue content of the at least one round of the second dialogue; or, The first feature includes a summary feature obtained by performing feature extraction on a summary of the dialogue content of the at least one round of the second dialogue.

17. The apparatus according to claim 15 or 16, characterized in that, The target historical dialogue summary includes a first historical dialogue summary, which includes a summary of the dialogue content of a third dialogue, wherein the third dialogue is one round of dialogue in the at least one round of the second dialogue. The first feature includes the original feature of the third dialogue or the summary feature of the third dialogue. The original feature of the third dialogue includes features obtained by performing feature extraction on the dialogue content of the third dialogue. The summary feature of the third dialogue includes features obtained by performing feature extraction on the summary of the dialogue content of the third dialogue. The processing module is further configured to: Determine the memory score of the first historical dialogue summary, the memory score being used to indicate the usage of the first historical dialogue summary; Based on the memory score, determine the storage status of the original features or summary features of the third dialogue in the feature library.

18. The apparatus according to claim 17, characterized in that, The processing module is further configured to: If the memory score is within the first range and the original features of the third dialogue are not stored in the feature library, then the original features of the third dialogue are obtained by performing feature extraction on the dialogue content of the third dialogue through the target model, and the original features of the third dialogue are stored in the feature library.

19. The apparatus according to claim 18, characterized in that, The processing module is further configured to: If the memory score is within the second range and the feature library does not store the summary features of the third dialogue, then the target model is used to perform feature extraction on the first historical dialogue summary to obtain the summary features of the third dialogue, and the summary features of the third dialogue are stored in the feature library, wherein the value in the second range is less than the value in the first range.

20. The apparatus according to claim 19, characterized in that, The processing module is further configured to: If the feature library stores the original features of the third dialogue, delete the original features of the third dialogue from the feature library.

21. The apparatus according to any one of claims 18-20, characterized in that, The processing module is further configured to: In the case where the memory score is in the third range: If the feature library contains the original features of the third dialogue, then delete the original features of the third dialogue from the feature library; or, If the feature library stores the summary features of the third dialogue, then the summary features of the third dialogue are deleted from the feature library; wherein, the value of the third range is less than the value of the second range, and the value in the second range is less than the value in the first range.

22. The apparatus according to any one of claims 17-21, characterized in that, The memory score is determined based on at least one of the following: the total number of times the first historical dialogue summary is used, the number of times the first historical dialogue summary is used within a preset time period, the time interval since the last use of the first historical dialogue summary, the duration since the first historical dialogue summary was generated, the number of consecutive uses of the first historical dialogue summary, the number of times the original features of the first historical dialogue summary are reused, the number of times the summary features of the first historical dialogue summary are reused, the length of the original dialogue content corresponding to the first historical dialogue summary, the length of the first historical dialogue summary, and the time interval since the last change in the range of the memory score.

23. The apparatus according to any one of claims 17-22, characterized in that, The memory score is determined periodically.

24. The apparatus according to any one of claims 15-23, characterized in that, The target historical dialogue summary includes at least two historical dialogue summaries, and the at least two historical dialogue summaries correspond to the dialogue content of dialogues in different historical time periods; The processing module is further configured to: Determine a first sub-feature in the first feature, wherein the first sub-feature includes features corresponding to a portion of the tokens in the target historical dialogue summary; Based on the first feature, the second sub-feature corresponding to the partial token is re-extracted through the target model; The first sub-feature in the first feature is updated to the second sub-feature to obtain the updated first feature; The problem text is processed by the target model based on the updated first and second features.

25. The apparatus according to any one of claims 14-24, characterized in that, The question text is from the first session; The target historical dialogue content includes dialogue content from the second session.

26. The apparatus according to any one of claims 14-25, characterized in that, The acquisition module is also used to acquire multiple historical dialogue summaries, which correspond to dialogue content in different historical time periods; The processing module is further configured to determine the target historical dialogue summary from the plurality of historical dialogue summaries based on the relevance between the plurality of historical dialogue summaries and the question text.

27. A computing device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the computing device performs the method as described in any one of claims 1 to 13.

28. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 13.

29. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 13.

30. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 13.

Images