Inference acceleration method based on AI native storage and related device

Abstract

A reasoning acceleration method based on AI native storage is applied to the reasoning process of an acceleration model. In the reasoning acceleration method, aiming at an acquired problem text and an acquired reference text, a plurality of feature fragments corresponding to the reference text are firstly acquired, the plurality of feature fragments respectively correspond to different text fragments in the reference text, then partial feature fragments related to the problem text are selected from the plurality of feature fragments, and the partial feature fragments are selected from the reference text. And then partial selected feature segments are used as the features of the reference text to process the problem text, so that the feature quantity needing to be processed in the reasoning process of the target model is effectively reduced, the feature can be searched and calculated on the basis of the pre-stored features, the situation that calculation is executed on the features again is avoided, and the calculation efficiency is improved. The reasoning speed of the target model is improved, and the reasoning precision is not affected. Namely, according to the scheme, the memory of the large model is stored, secondary reasoning of the large model is not needed, and the characteristics can be used and taken by searching instead of calculation, so that the reasoning efficiency of the large model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a reasoning acceleration method and related apparatus based on AI native storage. Background Technology

[0002] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data. They can not only generate natural language text, but also deeply understand the meaning of text and handle various natural language tasks, such as text summarization, intelligent question answering, and text translation.

[0003] With the development of large language models, their growth in scale is particularly remarkable, with the number of parameters jumping from billions to trillions. This increase in parameters allows large language models to capture the subtleties of human language more precisely and to gain a deeper understanding of its complexity.

[0004] However, due to the sheer number of parameters in large language models, each use incurs significant computational and time overhead due to the complexity of parameter calculation. This is especially true in long-sequence reasoning scenarios (such as question-answering), where the input to a large language model is a long sequence of text. Large language models often require a substantial amount of time to execute the reasoning process, resulting in slow response times and difficulty in meeting user needs. Summary of the Invention

[0005] This application provides an inference acceleration method based on AI native storage, which is applied to accelerate the inference process of a model based on stored and pre-extracted features (i.e., AI native storage).

[0006] Firstly, a reasoning acceleration method is provided for scenarios involving question answering based on a target model. This method includes: firstly, the execution device acquires question text and target reference text. The question text indicates the question to be answered, and the target reference text is the text used for reference when answering the question. That is, the target model answers the question by referring to the content in the target reference text.

[0007] Then, based on the target reference text, the device acquires multiple feature fragments corresponding to the target reference text. These feature fragments are obtained by performing feature extraction on multiple text fragments using the target model, and the multiple text fragments are divided based on the target reference text. For example, each feature fragment is a matrix of the same size.

[0008] Secondly, the execution device determines at least one target feature segment from the multiple feature segments based on the degree of relevance between the feature segments and the question text, that is, it selects only a portion of the feature segments that are relevant to the question text from the multiple feature segments.

[0009] Finally, based on at least one target feature fragment, the execution device processes the question text through the target model to obtain the answer text. That is, the execution device actually processes the question text by using at least one identified target feature fragment as a feature of the target reference text, rather than directly processing the target reference text and the question text through the target model.

[0010] In this approach, for the acquired question text and target reference text, multiple feature fragments corresponding to the target reference text are first obtained. These feature fragments correspond to different text segments within the target reference text. Then, feature fragments with a high degree of relevance to the question text are selected from these fragments. These selected feature fragments are then used as features of the target reference text to process the question text. This effectively reduces the number of features the target model needs to process during inference, improving inference speed without affecting inference accuracy. Since the target reference text is the text referenced by the target model when answering questions, it often includes a large amount of text content, including some redundant text with low relevance to the question text. Therefore, by selecting a portion of feature fragments from the multiple feature fragments corresponding to the target reference text, the large language model can focus on features related to the question text, thereby ensuring the inference accuracy of the target model.

[0011] In one possible implementation, the execution device determines at least one target feature segment from multiple feature segments based on the relevance of the feature segments among multiple feature segments to the problem text. Specifically, the execution device first determines whether the storage medium stores features of the reference text, wherein the storage medium stores features of multiple reference texts, and the features of the target reference text include multiple feature segments.

[0012] When the features of the target reference text are already stored in the storage medium, the execution device reads at least one target feature fragment from the storage medium based on the relevance of the feature fragment among multiple feature fragments to the problem text.

[0013] In other words, this solution uses a storage medium to pre-store the features of the reference text, so that when the features of a specific reference text are needed, the required features can be directly read from the storage medium without recalculating the features of the reference text. This achieves feature lookup instead of calculation and improves the inference speed of the target model.

[0014] For example, by offloading the key-value cache of hot reference texts to a storage medium for persistence, the relevant key-value caches can be reused in subsequent sessions through similarity retrieval, which can avoid the recalculation of key-value caches and improve the inference speed of the model based on AI native storage.

[0015] In one possible implementation, the storage medium that stores the features of multiple reference texts includes a persistent storage medium, such as a hard disk.

[0016] In one possible implementation, during the process of reading at least one target feature fragment from the storage medium based on the relevance of the feature fragments among multiple feature fragments to the question text, the execution device may first read multiple feature fragments from the storage medium, and then select at least one target feature fragment based on the relevance of the feature fragments among the multiple feature fragments to the question text.

[0017] That is, the execution device first reads the feature fragments from the storage medium, then calculates the relevance of the feature fragments to the question text, and then selects the target feature fragments.

[0018] Furthermore, when each feature segment has a corresponding representative representation, the execution device can first read the representative representation of each feature segment from the storage medium and calculate the relevance between the representative representation and the question text to determine at least one target feature segment. Then, the execution device reads the determined at least one target feature segment from the storage medium. That is, the execution device improves the efficiency of calculating the relevance by calculating the relevance between the representative representation (which requires less data) and the question text, instead of directly calculating the relevance between the feature segments and the question text, thereby improving the inference efficiency of the target model.

[0019] In one possible implementation, during the process of reading at least one target feature fragment from a storage medium based on the relevance of a feature fragment among multiple feature fragments to the question text, the execution device may determine the relevance of a feature fragment among multiple feature fragments to the question text via a processor in the storage device where the storage medium is located, and determine at least one target feature fragment based on the relevance.

[0020] After the processor in the storage device where the storage medium resides determines at least one target feature fragment, the execution device then reads at least one target feature fragment from the storage medium. That is, the storage device where the storage medium resides is equipped with a processor, and the execution device can use this processor to determine the target feature fragment (equivalent to the execution device sending the calculation process of determining the target feature fragment to the processor on the storage device for execution), and then directly read at least one target feature fragment from the storage medium.

[0021] In one possible implementation, during the process of determining at least one target feature fragment, the execution device performs feature extraction on the question text using a target model to obtain question features. Then, based on the correlation between the question features and feature fragments among multiple feature fragments stored in the storage medium, the execution device determines at least one target feature fragment and reads at least one target feature fragment from the storage medium.

[0022] In this scheme, by calculating the correlation between the problem features and feature fragments corresponding to the problem text, it is possible to determine the feature fragments with a high degree of correlation with the problem text based on the operation between features, and then select the feature fragments that can provide effective information for processing the problem text from multiple feature fragments, thereby improving the feasibility of the scheme.

[0023] In one possible implementation, the question text appears in the first session, and multiple feature fragments are extracted and stored in the storage medium before the first session. That is, the multiple feature fragments are extracted before the question text appears in the session. Therefore, based on the multiple feature fragments that have been extracted and stored in the storage medium in advance, the corresponding feature fragments can be obtained by searching for feature fragments in the storage medium, thereby achieving lookup instead of calculation and avoiding re-performing the calculation of feature fragments.

[0024] In one possible implementation, the target reference text appears in a second session prior to the first session, and multiple feature fragments are extracted and stored in the storage medium during the second session. That is, the target reference text has already appeared in other sessions. When the target reference text appears in the second session, the execution device can calculate and store the features of the target reference text. Therefore, if the target reference text continues to appear in the first session, the features of the target reference text can be quickly retrieved from the storage medium, avoiding recalculation of the target reference text's features.

[0025] In one possible implementation, if the features of the target reference text are not stored in the storage medium, the execution device determines the relevance of the first text fragment to the problem text and extracts feature fragments of the first text fragment through the target model, wherein the first text fragment is one of a plurality of text fragments.

[0026] Then, the execution device determines at least one target feature segment based on the relevance of the multiple text segments to the question text. Specifically, the relevance of the multiple text segments to the question text can refer to the relevance of each individual text segment to the question text.

[0027] In other words, since there is a one-to-one correspondence between multiple text fragments and multiple feature fragments of the target reference text, the execution device can determine the degree of correlation between the text fragments and the question text, and then regard the degree of correlation between the text fragments and the question text as the degree of correlation between the feature fragments and the question text, thereby ultimately achieving the filtering of feature fragments.

[0028] In one possible implementation, the execution device processes the first text fragment and the question text using a target model to obtain an output result. The first text fragment is one of multiple text fragments. Furthermore, the execution device determines the confidence level of the output result and uses this confidence level as the degree of relevance between the first text fragment and the question text.

[0029] In other words, the execution device actually combines each text fragment with the question text separately and inputs them into the target model for processing, thereby obtaining the output result corresponding to the combined text of each text fragment and the question text. In this way, based on the confidence level of the output result corresponding to each combined text, the degree of relevance between the text fragment and the question text can be determined.

[0030] In one possible implementation, in order to obtain multiple feature fragments corresponding to the reference text, the execution device first divides the reference text into multiple text fragments, and then performs feature extraction on the multiple text fragments respectively through the target model to obtain multiple feature fragments.

[0031] In this scheme, the reference text is first divided into multiple text segments, and then feature extraction is performed on each of the multiple text segments. This allows the feature extraction process of multiple text segments to be executed in parallel, and the length of the text processed by the target model at one time is greatly reduced, which can effectively improve the efficiency of the execution device in extracting features from the reference text.

[0032] In one possible implementation, the question text appears in the Nth round of dialogue in the first session, where the first session consists of multiple rounds of dialogue and N is a positive integer.

[0033] To better process the question text, the execution device can also acquire historical dialogue content or a summary of the target historical dialogue. The historical dialogue content includes the content of at least one round of the first target dialogue, and the target historical dialogue summary includes a summary of the content of at least one round of the second target dialogue. The first target dialogue is the dialogue preceding the Nth round in the first session; for example, the first target dialogue is one or more rounds of dialogue adjacent to the Nth round in the first session. The second target dialogue is the dialogue in the first session preceding the first target dialogue.

[0034] Then, the execution device performs feature extraction on the historical dialogue summary or historical dialogue content using the target model to obtain historical interaction features. Alternatively, the execution device can also perform feature extraction on both the historical dialogue summary and historical dialogue content using the target model to obtain the historical interaction features.

[0035] Finally, based on historical interaction features and at least one target feature fragment, the execution device processes the problem text through the target model.

[0036] In other words, in addition to the feature fragments corresponding to the reference text, the execution device also obtains the dialogue content preceding the question text and uses the dialogue content to assist the target model in processing the question text.

[0037] In this solution, by extracting the summary content of question-and-answer interaction content that is far removed from the current question text in time, and using the summary content to assist in processing the question text, the accuracy of the target model in processing the question text is not affected. Moreover, since the content that the target model needs to process is greatly reduced, the processing efficiency of the target model can be effectively improved.

[0038] In one possible implementation, the target historical dialogue summary is selected based on the relevance of multiple historical dialogue summaries to the question text, with the multiple historical interaction summaries corresponding to question-and-answer interaction content in different historical time periods.

[0039] In other words, for historical dialogue content between the execution device and the user from a long time ago, the execution device performs summary extraction on the dialogue content at different historical times, thereby obtaining multiple historical dialogue summaries. In this way, the execution device can select the historical dialogue summaries that are most relevant to the question text from multiple historical dialogue summaries, thereby avoiding the use of all historical dialogue summaries to assist in processing the question text, reducing the amount of content that the target model needs to process, and improving the processing efficiency of the target model.

[0040] In one possible implementation, the target model is a large language model, and multiple feature fragments include a key-value cache (KV cache). In another possible implementation, the combined length of the question text and the target reference text is greater than a preset length.

[0041] Secondly, a reasoning acceleration device is provided, comprising: an acquisition module for acquiring question text and target reference text, wherein the question text indicates the question to be answered and the target reference text is text used for reference when answering the question; a processing module for determining at least one target feature segment from multiple feature segments based on the relevance of the feature segments among multiple feature segments to the question text; wherein the multiple feature segments are obtained by performing feature extraction on the multiple text segments respectively by a target model, and the multiple text segments are divided based on the target reference text; and the processing module is further configured to process the question text based on at least one target feature segment using a target model to obtain answer text.

[0042] In one possible implementation, the processing module is further configured to: determine whether the storage medium stores features of the reference text, wherein the storage medium stores features of multiple reference texts, and the features of the target reference text include multiple feature fragments; if the storage medium already stores features of the target reference text, read at least one target feature fragment from the storage medium based on the relevance of the feature fragments among the multiple feature fragments to the problem text.

[0043] In one possible implementation, the processing module is further configured to: perform feature extraction on the problem text using the target model to obtain problem features; determine at least one target feature segment based on the correlation between the problem features and feature segments among multiple feature segments stored in the storage medium; and read at least one target feature segment from the storage medium.

[0044] In one possible implementation, the problem text appears in the first session, and multiple feature fragments are obtained by feature extraction performed before the first session and stored in the storage medium.

[0045] In one possible implementation, the target reference text appeared in a second session prior to the first session, and multiple feature fragments were extracted and stored in the storage medium during the second session.

[0046] In one possible implementation, the processing module is further configured to: determine the relevance of a first text fragment to the problem text and extract feature fragments of the first text fragment by means of a target model when the features of the target reference text are not stored in the storage medium, wherein the first text fragment is one of a plurality of text fragments; and determine at least one target feature fragment based on the relevance of the plurality of text fragments to the problem text.

[0047] In one possible implementation, the processing module is further configured to: process the first text fragment and the question text through the target model to obtain an output result; and determine the degree of correlation between the first text fragment and the question text based on the confidence level of the output result.

[0048] In one possible implementation, the question text appears in the Nth round of dialogue in the first session, where the first session includes multiple rounds of dialogue and N is a positive integer. The acquisition module is further configured to acquire historical dialogue content or a target historical dialogue summary, where the historical dialogue content includes content from at least one round of the first target dialogue, and the target historical dialogue summary includes a summary of content from at least one round of the second target dialogue. The first target dialogue is the dialogue preceding the Nth round of dialogue in the first session, and the second target dialogue is the dialogue preceding the first target dialogue in the first session. The processing module is further configured to perform feature extraction on the historical dialogue summary or historical dialogue content using a target model to obtain historical interaction features. The processing module is further configured to process the question text using a target model based on the historical interaction features and at least one target feature fragment.

[0049] In one possible implementation, the target historical dialogue summary is selected based on the relevance of multiple historical dialogue summaries to the question text, with the multiple historical interaction summaries corresponding to question-and-answer interaction content in different historical time periods.

[0050] In one possible implementation, the target model is a large language model, with multiple feature fragments including a KV cache.

[0051] In one possible implementation, the sum of the lengths of the question text and the target reference text is greater than a preset length.

[0052] Thirdly, an inference acceleration device is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the inference acceleration device to perform the method of any of the above aspects.

[0053] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods described in any of the preceding aspects.

[0054] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0055] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0056] In a seventh aspect, a computing device is provided, the computing device including an inference acceleration device of the third aspect or a chip system of the sixth aspect, wherein the inference acceleration device or the chip system in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0057] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0058] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0059] Figure 1 A schematic diagram of a system architecture provided for this application;

[0060] Figure 2 A flowchart illustrating a reasoning acceleration method provided in this application;

[0061] Figure 3 A flowchart illustrating a process for processing problem text and reference text using a target model, as provided in this application;

[0062] Figure 4 A schematic diagram illustrating a method for determining feature segments by calculating attention scores, as provided in this application;

[0063] Figure 5 A schematic diagram provided for this application to determine the relevance of a text fragment to the question text;

[0064] Figure 6 A schematic diagram illustrating how to determine historical interaction features based on historical interaction content, as provided in this application;

[0065] Figure 7 This application provides a flowchart illustrating the process of handling question text in a human-computer dialogue scenario.

[0066] Figure 8 A schematic diagram illustrating how an execution device determines historical interaction content requiring feature extraction, as provided in this application;

[0067] Figure 9 A flowchart illustrating the process of determining a feature fragment of a reference text using an execution device, as provided in this application;

[0068] Figure 10 This application provides a schematic diagram of combining text fragments with question text to determine text fragments;

[0069] Figure 11 A schematic diagram of a reasoning acceleration device provided in this application;

[0070] Figure 12A schematic diagram of the structure of a computing device provided in this application;

[0071] Figure 13 This application provides a schematic diagram of the structure of a computing device cluster;

[0072] Figure 14 This application provides a schematic diagram of another computing device cluster structure.

[0073] Figure 15 This is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0075] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0076] To facilitate understanding, some technical terms used in this application will be introduced below.

[0077] (1) Large Language Model

[0078] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0079] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0080] Currently, large language models are mainly composed of Transformer networks.

[0081] (2) Transformer network

[0082] Transformer networks are powerful sequence models, but the computation time and memory required increase quadratically with sequence length, significantly increasing the hardware's storage and computing power demands. Essentially, Transformer networks employ a self-attention mechanism. Self-attention is a mechanism that associates different positions within a single sequence to compute a representation of the same sequence, playing a crucial role in machine reading, abstract summarization, and image description generation.

[0083] Taking the Transformer network applied to natural language processing as an example, the Transformer network processes input data of arbitrary length and generates new feature representations of the input data, which are then converted into target words. The self-attention network layer in the Transformer network uses an attention mechanism to capture the relationships between all other words, thereby generating new feature representations for each word. The advantage of the Transformer network's self-attention network is that the attention mechanism can directly capture the relationships between all words in a sentence without considering word positions.

[0084] (3) Key-Value Cache (KV Cache)

[0085] When using Transformer networks for natural language processing tasks, they typically employ a self-attention mechanism to process the input sequence. In this mechanism, the Transformer network generates a corresponding key (K) vector, value (V) vector, and query (Q) vector for each token in the input sequence. Further, the Transformer network calculates the degree of matching between each query vector and all key vectors, usually achieved through a dot product. Then, using the matching degree between query and key vectors as weights, it calculates a weighted sum of all value vectors to obtain the final result.

[0086] When the Transformer network processes the input sequence, the key vector generated by the Transformer network for each word in the input sequence can be stored in a single matrix, and the value vector generated by the Transformer network for each word in the input sequence can be stored in another matrix. Therefore, KV Cache refers to the key matrix and value matrix generated by the Transformer network for the words in the input sequence, and these key matrices and value matrices are cached.

[0087] (4) Prompt

[0088] Prompts originated as an input format designed by researchers for downstream tasks. Their purpose is to help pre-trained models "recall" what they "learned" during pre-training, hence the name "cue word." For large language models, a prompt is the user's input, instructing the model on the task to be performed. A prompt can be a simple question, a longer text, or a set of instructions, depending on the user's specific needs. Generally, a prompt is a short text string that provides context and task-related information to help the model better understand the requirements and generate the correct output. For example, in question-answering tasks, a prompt might contain a description of the question or topic to help the large language model generate the correct answer. Furthermore, prompts are often designed by humans to help large language models better understand specific tasks or domains.

[0089] In this way, when a large language model generates content, it first processes the prompt and then outputs content based on its understanding of the prompt. The working principle of the large language model is to predict the probability of the next word appearing based on the preceding context of the user input, thereby generating the following text word by word. Therefore, differences in the user's input prompt directly affect the quality of the large language model's output. In some cases, even a difference of just a few words in the user's input prompt can result in significantly different content generated by the large language model.

[0090] (5) Token

[0091] A lexical is the smallest semantic unit in natural language processing, typically derived from the decomposition of long texts such as words, sentences, paragraphs, and articles. Generally, lexical units can be composed of letters, words, or symbols; for example, they can be composed of one or more letters or one or more words. Simply put, a lexical unit can be understood as a "fragment" of a word or phrase—the smallest indivisible semantic unit. For example, "waterfall" can be broken down into two lexical units: "water" and "fall." Additionally, punctuation marks can also be decomposed into lexical units because they affect the semantic understanding of the entire text. For example, "I don't know." can be decomposed into five lexical units: "I," "don," "'t," "know," and ".".

[0093] (6) Session

[0094] A session is a concept of a temporary interactive state established between a client and a server, used to track user actions over a period of time. Sessions are typically used to maintain user state and information, such as login status and conversation content.

[0095] Specifically, the conversation can be explained from the following aspects.

[0096] Continuous interaction: A session refers to a series of requests and responses between a user and a server over a period of time.

[0097] Lifecycle: A session typically begins when a user logs in or accesses the application and ends when the user logs out or the session times out. The timeout period is usually set by the server; for example, 30 minutes of inactivity will invalidate the session.

[0098] Information storage: The server stores user state data, such as user identity, permissions, and personalized settings. The client informs the server which session it is associated with through some identifier (such as a Session ID).

[0099] Session ID: To identify different users' sessions, the server assigns a unique identifier (Session ID) to each session. This Session ID is usually stored on the client and sent to the server with each request.

[0100] In general, in the human-computer dialogue scenario of this application, a session refers to the interaction between a user and an execution device (such as a server) over a period of time. Therefore, a session can include one or more rounds of dialogue between the user and the execution device.

[0101] Currently, in some long-sequence reasoning scenarios, large language models (MLMs) are inputted with lengthy sequences of text. These MLMs often spend a significant amount of time executing the reasoning process, resulting in slow response times and difficulty meeting user needs. For example, in question-answering scenarios, to ensure high-quality answers from MLMs, reference texts (such as papers, user manuals, and search results) are often provided simultaneously. This allows the MLM to provide targeted answers based on the content of the reference texts. In some cases, the reference texts may contain a lot of information, leading to a long input text for the MLM, which can negatively impact its reasoning speed.

[0102] Furthermore, due to limited training resources, the length of training data used by large language models during training is usually less than or equal to a certain upper limit. In actual inference, if the length of the input text to the large language model exceeds the upper limit defined during training, a significant gap will occur between the inference data and the training data, leading to a rapid decrease in the inference accuracy of the large language model.

[0103] In view of this, this application provides an inference acceleration method. For the obtained question text and reference text, firstly, multiple feature fragments corresponding to the reference text are obtained, with each feature fragment corresponding to a different text segment within the reference text. Then, a subset of feature fragments with a high degree of relevance to the question text is selected from these multiple feature fragments. These selected feature fragments are then used as features of the reference text to process the question text. This effectively reduces the number of features the target model needs to process during inference, improving the inference speed of the target model without affecting its inference accuracy. Since the reference text is the text referenced by the target model when answering the question, it often includes a large amount of text content, including some redundant text with low relevance to the question text. Therefore, by selecting a subset of feature fragments from the multiple feature fragments corresponding to the reference text, the large language model can focus on features related to the question text, thereby ensuring the inference accuracy of the target model.

[0104] Please see Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided for this application. (For example...) Figure 1As shown, in this system architecture, the execution device 10 can be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster). The execution device 10 includes an accelerator 101 and a processor 102. The accelerator 101 is used to run the target model (e.g., a large language model) to process the inference tasks passed by the processor 102. The processor 102 is used to obtain task requests from clients (e.g., requests to answer a question) and schedule the accelerator 101 to process the specified inference task based on the task request from the client.

[0105] Optionally, the execution device 10 can be used in conjunction with other computing devices, such as data storage devices, load balancers, etc.; the execution device 10 can be deployed on a single physical site or distributed across multiple physical sites.

[0106] In addition, the system architecture also includes a data storage system 11, which is used to store data such as problem text, reference text, KV cache, or program code.

[0107] Optionally, for persistent data storage, the data storage system 11 can be located external to the execution device 10 and exchange data with the execution device 10 via a network. Alternatively, if the execution device 10 is a physical host, the data storage system 11 can also be located internally to the execution device 10, such as exchanging data with the processor via a bus. In this case, the data storage system 11 functions as a hard disk. With the data storage system 11, the execution device 10 can use data in the data storage system 11 (such as KV Cache) or call program code in the data storage system to implement the inference acceleration method provided in this application.

[0108] Optionally, users can interact with execution device 10 using their respective local devices. For example, a client 121 is deployed on local device 12, and users interact with the execution device through client 121 on local device 12. Local device 12 can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, laptop, or smart car.

[0109] Local device 12 can interact with execution device 10 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0110] Optionally, during the implementation of the inference acceleration method by the execution device 10, the local device 12 can provide the execution device 10 with task requests or prompts, so that the execution device 10 can process the input text through the model to complete the task. Furthermore, after the execution device 10 executes the inference acceleration method and obtains the output result, it can feed the output result back to the local device 12.

[0111] Please see Figure 2 , Figure 2 This is a flowchart illustrating a reasoning acceleration method provided in this application. Figure 2 As shown, the execution flow of this inference acceleration method includes the following steps 201-203.

[0112] Step 201: Obtain the question text and the target reference text. The question text indicates the question to be answered, and the target reference text is the text to be referenced when answering the question.

[0113] In this application, the inference acceleration method can be applied to scenarios where question answering is performed based on a target model. In a question answering scenario, a user interacts with an execution device deployed with the target model via a client on their local device, sending question text to the execution device and requesting an answer. Therefore, the execution device can obtain the question text to be processed from the user. This question text can be, for example, "How many goals did Messi score in the World Cup?" or "Who is the author of Harry Potter?"—any question composed of text. The question text can be text sent directly by the user through their local device, or it can be obtained by converting speech sent by the user through their local device.

[0114] Generally, to help the target model respond more effectively to the user-provided question text, users often provide reference text (such as the target reference text mentioned above). This allows the target model to refer to the content of the user-provided reference text when answering questions, thereby improving the accuracy of the target model's answers. For example, when a user asks an academic question, the user can provide some academic papers as reference text. Similarly, when a user asks a current affairs question, the user can provide some web search results as reference text. In general, the reference text includes text content related to the question text, and the reference text can be provided in the form of a document or a webpage; this application does not impose specific limitations on this.

[0115] Optionally, users may choose not to provide any reference text when providing the question text. After obtaining the question text, the execution device retrieves relevant reference text from the knowledge base (the aforementioned target reference text). The knowledge base pre-stores a large number of reference texts for the execution device to retrieve based on the specific question text to be processed. For example, in a question-and-answer scenario in the financial field, the knowledge base stores numerous user manuals. When the user provides a question related to credit cards, the execution device can retrieve a credit card user manual from the knowledge base as a reference text.

[0116] Furthermore, when the execution device retrieves reference texts related to the question text in the knowledge base based on the question text, the execution device may extract the feature vector of the question text and calculate the distance between the feature vector of the question text and the feature vectors of each reference text in the knowledge base, thereby selecting the reference text with the closest feature vector distance in the knowledge base as the reference text corresponding to the question text.

[0117] In general, the aforementioned target reference text can be provided by the user providing the question text, or it can be retrieved by the execution device from a knowledge base based on the question text; this application does not impose any specific limitations on this. Furthermore, the aforementioned target reference text may include one or more reference texts.

[0118] Optionally, the total length of the question text and the target reference text can be greater than a preset length. This total length can refer to the sum of the number of tokens included in the question text and the number of tokens included in the target reference text; in this case, the preset length is actually a preset number (e.g., 5000 or 10000). The more tokens included in the question text and the target reference text, the greater their total length. Alternatively, the total length can refer to the sum of the data size of the question text and the data size of the target reference text; in this case, the preset length is actually a preset data size (e.g., 8 kilobytes or 10 kilobytes). In general, the greater the total length of the question text and the target reference text, the more text content they contain.

[0119] Step 202: Based on the relevance of the feature segments among the multiple feature segments to the problem text, determine at least one target feature segment from the multiple feature segments; the multiple feature segments are obtained by performing feature extraction on the multiple text segments by the target model respectively, and the multiple text segments are divided based on the target reference text.

[0120] Specifically, for the target reference text, feature extraction can be performed on the target reference text using a target model to obtain its features. These features can be a matrix extracted by the target model. Furthermore, based on dividing the target reference text into multiple text segments, its features can be divided into multiple feature segments, each corresponding to one of the multiple text segments obtained from the target reference text. That is, there is a one-to-one correspondence between the multiple feature segments and the multiple text segments of the target reference text.

[0121] Optionally, the target model is a large language model, and multiple feature segments include KV caches extracted based on the target model. The target model, as a large language model, includes numerous neural network layers, each with a large number of parameters. Therefore, when multiple feature segments are KV caches, these feature segments can actually be K and V vectors extracted from a single neural network layer. That is, each feature segment includes a Key matrix and a Value matrix, with each Key matrix containing one or more K vectors and each Value matrix containing one or more V vectors.

[0122] For multiple feature segments extracted from different neural network layers, step 202 can be performed independently to determine at least one target feature segment from the multiple feature segments extracted from each neural network layer.

[0123] Generally, when the target model is a large language model, the target reference text can be viewed as a long sequence of text composed of multiple tokens. For each token in the reference text, each neural network layer in the target model can extract a K vector and a V vector. Therefore, when the target reference text is divided into multiple text segments, a text segment actually includes one or more tokens, and the features of the target reference text can be divided into multiple feature segments, each of which has a corresponding text segment.

[0124] Understandably, in most cases, not all textual content included in the reference text is highly relevant to the question text. The reference text often contains redundant content with low relevance to the question text, and this redundant content rarely provides valuable reference for the target model's response. Therefore, during the target model's processing of the question text, the features corresponding to the redundant content with low relevance to the question text in the target reference text often fail to provide effective information for the target model.

[0125] Based on this, this step determines at least one target feature segment from the multiple feature segments based on the relevance of each feature segment to the question text. For example, it selects only the feature segments most relevant to the question text from the multiple feature segments. The number of at least one feature segment selected from the multiple feature segments can be determined based on the number of feature segments, such as selecting 10% or 20% of the feature segments. Alternatively, the number of at least one selected feature segment can be a pre-set fixed value, such as 5 or 10. This application does not specifically limit the number of at least one selected feature segment.

[0126] Optionally, the execution device determines at least one target feature segment from multiple feature segments based on the relevance of the feature segments to the problem text. Specifically, the execution device first determines whether the storage medium stores features of the reference text, wherein the storage medium stores features of multiple reference texts, and the features of the target reference text include multiple feature segments.

[0127] When the features of the target reference text are already stored in the storage medium, the execution device reads at least one target feature fragment from the storage medium based on the degree of relevance between the feature fragments among multiple feature fragments and the problem text.

[0128] In other words, this solution uses a storage medium to pre-store the features of the reference text, so that when the features of a specific reference text are needed, the required features can be directly read from the storage medium without recalculating the features of the reference text. This achieves feature lookup instead of calculation and improves the inference speed of the target model.

[0129] For example, by offloading the key-value cache of hot reference texts to a storage medium for persistence, the relevant key-value caches can be reused in subsequent sessions through similarity retrieval, which can avoid the recalculation of key-value caches and improve the inference speed of the model based on AI native storage.

[0130] Among them, the storage medium that stores the features of multiple reference texts includes persistent storage media, such as hard disks.

[0131] In one possible implementation, the execution device may first read multiple feature fragments from the storage medium, and then select at least one target feature fragment based on the relevance of the feature fragments to the question text. That is, the execution device first reads feature fragments from the storage medium, then calculates the relevance of the feature fragments to the question text, and then selects the target feature fragment.

[0132] Furthermore, when each feature segment has a corresponding representative representation, the execution device can first read the representative representation of each feature segment from the storage medium and calculate the relevance between the representative representation and the question text to determine at least one target feature segment. Then, the execution device reads the determined at least one target feature segment from the storage medium. That is, the execution device improves the efficiency of calculating the relevance by calculating the relevance between the representative representation (which requires less data) and the question text, instead of directly calculating the relevance between the feature segments and the question text, thereby improving the inference efficiency of the target model.

[0133] In another possible implementation, during the process of reading at least one target feature fragment from the storage medium based on the relevance of the feature fragments among the multiple feature fragments to the question text, the execution device may determine the relevance of the feature fragments among the multiple feature fragments to the question text through the processor in the storage device where the storage medium is located, and determine at least one target feature fragment based on the relevance.

[0134] After the processor in the storage device where the storage medium resides determines at least one target feature fragment, the execution device then reads at least one target feature fragment from the storage medium. That is, the storage device where the storage medium resides is equipped with a processor, and the execution device can use this processor to determine the target feature fragment (equivalent to the execution device sending the calculation process of determining the target feature fragment to the processor on the storage device for execution), and then directly read at least one target feature fragment from the storage medium.

[0135] Step 203: Based on at least one target feature fragment, process the question text through the target model to obtain the answer text.

[0136] In related technologies, the execution device typically uses reference text and question text as input to the model, and sequentially performs feature extraction on the tokens in the reference text and question text. When extracting features corresponding to subsequent tokens, it is necessary to use the features corresponding to previous tokens.

[0137] In this step, at least one identified target feature fragment is used as the feature corresponding to the target reference text. Based on this target feature fragment, feature extraction of the question text can be performed using a target model to obtain the features of the question text. Furthermore, the execution device will continue to process the extracted feature fragment and the features of the question text using the target model, thereby outputting the tokens of the answer text one by one, ultimately obtaining the answer text. Specifically, the answer text can be the text directly returned to the user in the dialogue. Alternatively, the answer text can be one or more texts to be recommended generated by the target model. The execution device can process one or more texts to be recommended before returning them to the user in the dialogue (e.g., the execution device selects one text to return to the user).

[0138] For example, please refer to Figure 3 , Figure 3 This application provides a flowchart illustrating the process of processing problem text and target reference text using a target model. For example... Figure 3 As shown, the target reference text can be specifically divided into N text segments, namely text segment 1 to text segment N. Correspondingly, the features of the target reference text are also divided into N feature segments, namely feature segment 1 to feature segment N. Based on the question text, at least one target feature segment with the highest relevance to the question text can be determined from the N feature segments (e.g., Figure 3 (Feature fragments 2 and 5 in the model). Thus, based on at least one determined target feature fragment, the question text can be processed by the target model to obtain the answer text output by the target model.

[0139] In this application, by using selected feature fragments as features of the reference text to process the question text, the amount of features that the target model needs to process during inference can be effectively reduced, the inference speed of the target model can be improved, and the memory occupied by the features can be reduced. Furthermore, since the reference text is the text that the target model refers to when answering the question, it often includes a large amount of text content, including some redundant text that is not highly relevant to the question text. Therefore, by selecting a portion of the feature fragments corresponding to the reference text, the large language model can focus on features relevant to the question text, thereby ensuring the inference accuracy of the target model.

[0140] In this application, there are multiple ways to determine at least one target feature segment among multiple feature segments.

[0141] Implementation method 1 involves selecting at least one target feature segment from multiple feature segments by calculating the attention scores of the question features and feature segments corresponding to the question text.

[0142] For example, the execution device first performs feature extraction on the question text using the target model to obtain the question features. It should be noted that the execution device at this time performs feature extraction only on the question text using the target model, without including the reference text as input to the target model.

[0143] The execution device then determines the attention score for each feature segment among the multiple feature segments and identifies at least one feature segment with the highest attention score. The attention score characterizes the similarity between the question feature and the feature segment. Generally, a higher attention score indicates a higher similarity between the question feature and the feature segment, and thus a higher relevance between the feature segment and the question text. Therefore, the execution device can use the at least one target feature segment with the highest attention score as the feature segment most relevant to the question text.

[0144] In the case of a large language model as the target model, the question features extracted by the execution device can include a query matrix, and the feature fragments can include a key matrix. In one possible example, the attention score between the question features and the feature fragments can be obtained by multiplying the query matrix included in the question features with the key matrix in the feature fragments. Of course, the attention score between the question features and the feature fragments can also be obtained in other ways, as long as the obtained attention score can represent the similarity between the question features and the feature fragments. This application does not impose any specific limitations on this.

[0145] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating a method for determining feature segments by calculating attention scores, as provided in this application. Figure 4 As shown, for the question text, feature extraction can be performed on the question text using the target model to obtain the question features. When the target reference text is divided into N text segments (text segment 1 to text segment N), each text segment corresponds to a feature segment (feature segment 1 to feature segment N). By calculating the attention score between the question features and each feature segment in feature segment 1 to feature segment N, N attention scores (i.e., attention score 1 to attention score N) can be obtained. Each attention score (attention score 1 to attention score N) corresponds one-to-one with each feature segment (feature segment 1 to feature segment N). Therefore, based on the attention score corresponding to each feature segment, at least one feature segment with the highest attention score (e.g., feature segment 2 and feature segment 5) can be selected from the N feature segments.

[0146] In this scheme, by calculating the attention score between the problem features and feature fragments corresponding to the problem text, it is possible to determine the feature fragments with the highest relevance to the problem text based on the operation between features. Then, feature fragments that can provide effective information for processing the problem text can be selected from multiple feature fragments, thereby improving the feasibility of the scheme.

[0147] Optionally, multiple feature fragments are obtained in a target storage medium used to store the pre-extracted feature fragments. The target storage medium can be, for example, memory or a hard disk.

[0148] In other words, before acquiring the question text, the execution device has already acquired the reference text and pre-extracted its features from the reference text using the target model. Furthermore, the execution device stores these features in the target storage medium for later direct reuse. After acquiring the question text, the execution device can retrieve the features of the reference text from the target storage medium and divide these features into the aforementioned multiple feature segments, thereby calculating the correlation between each feature segment and the question text.

[0149] There are several ways in which the execution device can obtain the reference text in advance. For example, a user may have conducted multiple rounds of question-and-answer interactions with the execution device through a local device, and the execution device may have already obtained the reference text provided by the user during previous interactions. Another example is that the administrators deploying the target model may collect frequently used reference texts in the current question-and-answer scenario in advance and provide these texts to the execution device, thus enabling the execution device to obtain the reference text beforehand.

[0150] In this solution, by pre-storing multiple feature fragments corresponding to the reference text on the execution device, the execution device can directly reuse the stored multiple feature fragments when processing the problem text, avoiding re-performing feature extraction on the reference text and improving the efficiency of the execution device in processing the problem text.

[0151] For example, the question text appears in the first session, and multiple feature fragments were extracted and stored in the storage medium before the first session. That is, multiple feature fragments were extracted before the question text appeared in the session. Therefore, based on the multiple feature fragments that have been extracted and stored in the storage medium in advance, the corresponding feature fragments can be obtained by searching for feature fragments in the storage medium, thereby achieving lookup instead of calculation and avoiding re-performing the calculation of feature fragments.

[0152] Optionally, the target reference text appeared in a second session prior to the first session, and multiple feature fragments were extracted and stored in the storage medium during the second session. The first and second sessions can be sessions between the executing device and the same user at different time periods; or, the first and second sessions can be sessions between the executing device and different users. Specifically, a session (such as the first or second session) can include the dialogue content between the executing device and the same user over a period of time.

[0153] That is, the target reference text has already appeared in other sessions. When the target reference text appears in the second session, the execution device can calculate and store the features of the target reference text. Then, if the target reference text continues to appear in the first session, the features of the target reference text can be quickly retrieved from the storage medium, avoiding recalculation of the features of the target reference text.

[0154] Implementation method 2 involves determining the relevance between multiple text fragments obtained from the target reference text and the question text, and then selecting at least one target feature fragment from the multiple feature fragments corresponding to the multiple text fragments.

[0155] For example, the execution device may first divide the target reference text into multiple text segments and determine the relevance of each text segment to the question text. Then, the execution device determines at least one target feature segment corresponding to at least one text segment with the highest relevance to the question text.

[0156] In other words, since there is a one-to-one correspondence between multiple text fragments of the reference text and multiple feature fragments, the execution device can determine the degree of correlation between the text fragments and the question text, and then regard the degree of correlation between the text fragments and the question text as the degree of correlation between the feature fragments and the question text, thus ultimately achieving the filtering of feature fragments.

[0157] Optionally, to determine the relevance between the text fragment and the question text, the execution device can process the first text fragment and the question text using a target model to obtain an output result, where the first text fragment is one of multiple text fragments. Then, the execution device determines the confidence level of the output result and uses this confidence level as the relevance between the first text fragment and the question text.

[0158] In other words, the execution device actually combines each text fragment with the question text separately and inputs them into the target model for processing, thereby obtaining the output result corresponding to the combined text of each text fragment and the question text. In this way, based on the confidence level of the output result corresponding to each combined text, the degree of relevance between the text fragment and the question text can be determined.

[0159] For example, please refer to Figure 5 , Figure 5 This application provides a schematic diagram for determining the relevance of a text fragment to the question text. For example... Figure 5 As shown, when the target reference text is divided into text fragments 1 to N, multiple combined texts can be obtained by combining each text fragment with the question text (e.g., combining text fragment 1 with the question text, text fragment 2 with the question text, ..., combining text fragment N with the question text). Then, each of these combined texts is input into the target model to obtain multiple output results (e.g., ...). Figure 5 The output results are 1 to N. By calculating the confidence level of each output result, we can use that confidence level as the degree of relevance between the corresponding text fragment and the question text, thus obtaining the degree of relevance between each text fragment and the question text.

[0160] Generally, the higher the confidence level of the target model's output, the higher the accuracy of the output predicted by the target model based on the input text. With a fixed question text, the higher the relevance between the text fragments combined with the question text and the question text itself, the higher the quality of the input information obtained by the target model when processing the combination of text fragments and question text, and the easier it is for the target model to output a result with high confidence. Conversely, the lower the relevance between the text fragments combined with the question text and the question text itself, the lower the quality of the input information obtained by the target model when processing the combination of text fragments and question text, and the easier it is for the target model to output a result with low confidence. Therefore, by calculating the confidence level of the output predicted by the target model for the combination of text fragments and question text, the degree of relevance between the text fragments and question text can be determined.

[0161] Specifically, to determine the confidence level of the output result, the execution device can process the first text fragment and the question text to obtain the probability distribution of the first token predicted by the execution device. Based on the probability distribution of the first token predicted by the execution device, entropy and perplexity can be calculated, and then the confidence level of the output result can be determined based on the calculated entropy and perplexity. Here, the entropy of the probability distribution is a measure of the uncertainty of a random variable. The more uniform the probability distribution of the random variable, the greater the entropy, and the lower the confidence level of the output result; the more non-uniform the probability distribution of the random variable, the smaller the entropy, and the higher the confidence level of the output result. Furthermore, perplexity is an indicator of the quality of a natural language model's prediction of a sample; the lower the perplexity of the output result, the more accurate the output result predicted by the target model.

[0162] In addition, the execution device may determine the confidence level of the output result in other ways, and this application does not make specific limitations on this.

[0163] Optionally, in implementation method 2, the execution device acquires multiple feature fragments corresponding to the target reference text. Specifically, the execution device first divides the reference text into multiple text fragments. Then, the execution device performs feature extraction on each of the multiple text fragments using the target model to obtain multiple feature fragments.

[0164] In other words, the execution device does not pre-store the multiple feature fragments corresponding to the target reference text. The execution device first divides the target reference text into text fragments, and then processes each text fragment separately through the target model to obtain multiple feature fragments.

[0165] In this scheme, the reference text is first divided into multiple text segments, and then feature extraction is performed on each of the multiple text segments. This allows the feature extraction process of multiple text segments to be executed in parallel, and the length of the text processed by the target model at one time is greatly reduced, which can effectively improve the efficiency of the execution device in extracting features from the reference text.

[0166] The above describes how the execution device processes the question text based on feature fragments of the target reference text when it acquires the question text and the target reference text. In some scenarios, users may engage in multi-turn dialogues with the execution device via a local device, and when the execution device acquires the question text from a particular turn of dialogue, the user and the execution device may have already completed one or more turns of dialogue. In this case, the execution device can often use the historical dialogue content between the user and the execution device as input to the target model to assist in processing the currently acquired question text.

[0167] However, if the user and the execution device have continuously conducted many rounds of dialogue, the historical dialogue content between the user and the execution device may contain a lot of information. This can lead to a slow inference speed of the target model when the execution device uses the historical dialogue content as input. Based on this, this application proposes to summarize the historical dialogue content between the user and the execution device, thereby reducing the amount of information input to the target model and improving the inference speed of the target model.

[0168] For example, the question text appears in the Nth round of the first session, where the first session includes multiple rounds of dialogue, and N is a positive integer. Each round of dialogue includes a question text provided by the user and an answer text provided by the execution device; however, if the user provides reference text, a round of dialogue may include the question text, reference text, and answer text. When the execution device obtains the question text, it also obtains historical dialogue content or a target historical dialogue summary. The historical dialogue content includes content from at least one round of the first target dialogue, and the target historical dialogue summary includes a summary of content from at least one round of the second target dialogue. The first target dialogue is the dialogue preceding the Nth round of the first session, such as one or more rounds of dialogue adjacent to the Nth round of the first session. The second target dialogue is the dialogue preceding the first target dialogue in the first session. For example, if the execution device obtains the question text during the 10th round of question-and-answer interaction with the user, then the historical dialogue content could include the question-and-answer interaction content between the execution device and the user from rounds 6 to 9, and the target historical dialogue summary could include a summary of the question-and-answer interaction content between the execution device and the user from rounds 1 to 5. The target historical dialogue summary can be a summary with less content obtained by processing the question-and-answer interaction content through a summary model.

[0169] Then, the execution device performs feature extraction on the target historical dialogue summary or historical dialogue content using the target model to obtain historical interaction features. Alternatively, the execution device can also perform feature extraction on both the historical dialogue summary and historical dialogue content using the target model to obtain the historical interaction features.

[0170] At this point, when processing the question text using the target model, the execution device can specifically process the question text based on historical interaction features and at least one target feature fragment. That is, the execution device performs feature extraction on the question text using the target model based on historical interaction features and at least one target feature fragment, thereby obtaining the features of the question text. Furthermore, the execution device will continue to process the extracted historical interaction features, at least one target feature fragment, and the features of the question text using the target model, thereby outputting the tokens of the answer text one by one, ultimately obtaining the answer text.

[0171] In other words, the execution device does not directly use all historical dialogue content with the user to process the currently acquired question text. Instead, it divides all historical interactions into two parts: one part consists of dialogue content that is more recent in time to the current question text, and the other part consists of dialogue content that is more distant in time. Furthermore, the dialogue content that is more distant in time needs to undergo text summarization extraction, and the extracted dialogue content summary is used as the text content for processing the question text. Therefore, the dialogue content that the execution device actually uses to process the current question text includes both the most recent dialogue content itself and summaries of older dialogue content.

[0172] Understandably, for the question text currently acquired by the execution device, the closer the dialogue content between the execution device and the user is temporally to the current question text, the greater the relevance between the dialogue content and the question text; conversely, the further away the dialogue content is temporally from the current question text, the smaller the relevance, and there may even be no relevance. Therefore, this solution extracts summaries of dialogue content that is temporally distant from the current question text and uses these summaries to assist in processing the question text. This does not affect the accuracy of the target model in processing the question text, and because the amount of content that the target model needs to process is greatly reduced, it can effectively improve the processing efficiency of the target model.

[0173] Optionally, when there is a large amount of historical question-and-answer interaction content between the execution device and the user, the target historical dialogue summary is selected based on the relevance of multiple historical dialogue summaries to the question text. For example, the target historical dialogue summary is the historical interaction summary with the highest relevance to the question text among multiple historical interaction summaries, and the multiple historical interaction summaries correspond to question-and-answer interaction content in different historical time periods.

[0174] In other words, for historical dialogue content between the execution device and the user from a long time ago, the execution device performs summary extraction on the dialogue content at different historical times, thereby obtaining multiple historical dialogue summaries. In this way, the execution device can select the historical dialogue summaries that are most relevant to the question text from multiple historical dialogue summaries, thereby avoiding the use of all historical dialogue summaries to assist in processing the question text, reducing the amount of content that the target model needs to process, and improving the processing efficiency of the target model.

[0175] For example, please refer to Figure 6 , Figure 6 This application provides a schematic diagram illustrating how to determine historical interaction characteristics based on historical dialogue content. For example... Figure 6As shown, during the entire interaction between the user and the execution device, the content interacted by the user and the execution device in chronological order is as follows: Dialogue content 1, Dialogue content 2, Dialogue content 3, historical dialogue content, and the question text currently acquired by the execution device. Among these, historical dialogue content is the dialogue content most recent to the current question text in time, while dialogue content 1-3 is the dialogue content further removed from the current question text in time. Furthermore, dialogue content 1-3 and historical dialogue content can all include one or more rounds of dialogue between the user and the execution device.

[0176] For dialogue content 1 through dialogue content 3, the execution device performs summary extraction, thereby extracting shorter historical dialogue summaries 1 through 3. Then, the execution device identifies the historical dialogue summary with the highest relevance to the question text from among these summaries as the target historical dialogue summary; for example, it identifies historical dialogue summary 2 as the target historical dialogue summary. In this way, by employing a target model to perform feature extraction processing on historical dialogue summary 2 and the historical dialogue content, the execution device can obtain historical interaction features.

[0177] To facilitate understanding, the following will take a question-and-answer scenario as an example to introduce in detail the application process of the reasoning acceleration method provided in this application.

[0178] Please see Figure 7 , Figure 7 This application provides a flowchart illustrating the process of processing question text in a human-computer dialogue scenario. For example... Figure 7 As shown, the process of processing question text in a human-computer dialogue scenario includes the following steps 701-709.

[0179] Step 701: Execute the device problem text.

[0180] After the dialogue between the user and the execution device begins, the execution device can obtain the question text provided by the user. For example, the user communicates with the execution device through a client on the local device and sends a question-and-answer request to the execution device, which includes the question text.

[0181] Step 702: The device determines whether there is historical dialogue content between itself and the user.

[0182] That is, the execution device determines whether there is any other dialogue content between the execution device and the user before obtaining the current question text.

[0183] Step 703: If there is historical dialogue content between the execution device and the user, the execution device extracts historical interaction features based on the historical dialogue content.

[0184] Specifically, the execution device can extract historical interaction features corresponding to historical dialogue content through a target model.

[0185] In addition, if there is no historical dialogue content between the execution device and the user, the execution device does not need to extract historical interaction features.

[0186] Step 704: After obtaining the problem text, the device determines whether the user has provided corresponding reference text.

[0187] Generally, if a user needs to provide reference text to the execution device, the user will send the corresponding reference text to the execution device along with the problem text. If the user only sends the problem text to the execution device, it can be assumed that the user has not provided the corresponding reference text.

[0188] Step 705: If the user provides reference text, the device determines whether the reference text has already been cached.

[0189] Step 706: If the execution device does not cache the features of the reference text, the execution device performs semantic compression on the reference text to obtain feature fragments.

[0190] Specifically, the execution device may perform semantic compression on the reference text using the implementation method 2 described in the above embodiments to obtain feature fragments.

[0191] Step 707: If the user does not provide reference text, the execution device retrieves reference text related to the question text from the knowledge base.

[0192] For example, the execution device first calculates a vector to represent the semantics of the question text through a model, and then retrieves one or more texts with the closest vector distance in the knowledge base based on the vector of the question text as reference texts related to the question text.

[0193] Step 708: If the execution device caches the features of the reference text, the execution device selects a subset of feature fragments from the cached features of the reference text.

[0194] Generally, if the reference text is retrieved by the execution device from a knowledge base, the execution device will typically cache the features of the reference text. If the reference text is provided by a user and is a frequently accessed text, the execution device may also cache the features of the reference text. Therefore, when the features of the reference text are cached, the execution device can select a subset of feature fragments from the cached features of the reference text based on implementation method 1 described in the above embodiments.

[0195] It should be noted that steps 702-703 above are actually the process of the execution device determining historical interaction features, while steps 704-708 are the process of the execution device determining feature fragments of the reference text, and these two processes do not affect each other. Therefore, the execution device can execute the process of determining historical interaction features and the process of determining feature fragments of the reference text in parallel, thereby improving the efficiency of the execution device in obtaining historical interaction features or feature fragments of the reference text.

[0196] Step 709: Concatenate the historical interaction features and feature fragments from the reference text, and perform inference on the question text based on the concatenated features.

[0197] Specifically, the execution device can extract features from the question text based on the concatenated features using a target model, thereby obtaining the features of the question text. Furthermore, the execution device will continue to process the extracted historical interaction features, at least one feature fragment, and the features of the question text using the target model, thereby outputting the tokens of the answer text one by one, ultimately obtaining the answer text.

[0198] Specifically, in step 703 above, the execution device can extract historical interaction features by setting a short memory window and a long memory window, so as to minimize the amount of data of historical interaction features extracted, and ensure that features related to the problem text are extracted as much as possible.

[0199] For example, please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating how an execution device determines historical interaction content requiring feature extraction, as provided in this application. Figure 8 As shown, the execution device maintains both a short memory window and a long memory window. The short memory window records the most recent rounds of dialogue between the execution device and the user. When the number of dialogue rounds stored in the short memory window reaches m rounds (m being a preset value such as 10 or 15), or when the total length of the dialogue content stored in the short memory window reaches a preset length, the execution device can extract a summary of the earliest portion of the dialogue from the short memory window and store the generated dialogue summary in the long memory pool. For example, when the number of dialogue rounds stored in the short memory window reaches 10 rounds, the execution device extracts a summary of the first 5 rounds of dialogue content to generate a dialogue summary. Furthermore, after the execution device extracts a summary of part of the dialogue content from the short memory window, it clears the extracted dialogue content from the short memory window.

[0200] The long memory window is used to record dialogue summaries related to the question text extracted from the long memory pool. Specifically, after obtaining the question text, the execution device can select a portion of the dialogue summaries related to the question text from the long memory pool based on the question text, and store the selected dialogue summaries in the long memory window. For example, the execution device selects n dialogue summaries from the long memory pool and stores them in the long memory window, where n can be a preset value such as 3 or 5.

[0201] Finally, the execution device can concatenate the dialogue summary stored in the long memory window (i.e., the aforementioned historical dialogue summary) with the dialogue content stored in the short memory window (i.e., the aforementioned historical dialogue content), and perform feature extraction on the concatenated content through the target model to obtain historical interaction features.

[0202] Furthermore, for ease of understanding, steps 706 and 707 above will be described in detail below.

[0203] For example, please refer to Figure 9 , Figure 9 This is a flowchart illustrating an execution device for determining characteristic segments of reference text, as provided in this application. Figure 9 As shown, step 706 above includes the following steps 7061-7063.

[0204] Step 7061: The execution device divides the reference text into multiple text fragments.

[0205] Specifically, the execution device can divide the reference text according to a set length (such as 64 or 128 tokens), so that the resulting multiple text fragments all include the same number of tokens.

[0206] Step 7062: Combine multiple text fragments with the question text and input them into the target model to obtain multiple output results.

[0207] For example, please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating a method for determining a text fragment by combining a text fragment with the question text, as provided in this application. Figure 10 As shown, assuming that the multiple text segments obtained based on the reference text include text segment 1 to text segment N, then each text segment can be combined with the question text, and the resulting combined text can be input into the target model to obtain multiple output results, such as output result 1 to output result N.

[0208] It should be noted that in step 7062, after combining the text fragments with the question text and inputting them into the target model, the features extracted by the target model from the text fragments (such as the K matrix and V matrix) can be cached, thus obtaining the KV Cache corresponding to each text fragment (e.g., ...). Figure 9 (KV Cache1-KV Cache N in the context of KV Cache1-KV CacheN).

[0209] Step 7063: Based on the confidence scores of multiple output results, select the feature segment corresponding to at least one text segment with the highest confidence score.

[0210] For the output results corresponding to different text segments, the confidence level of each output result can be calculated, and then the feature segment corresponding to at least one text segment with the highest confidence level can be selected as the feature segment corresponding to the reference text. For example, in Figure 9 In the process, the output results with the highest confidence levels, output result 2 and output result N, are selected from multiple output results. Then, the corresponding KV Cache2 and KV Cache N are concatenated to obtain the feature fragments used to process the problem text.

[0211] like Figure 9 As shown, step 707 above includes the following steps 7071-7073.

[0212] Step 7071: Divide the cached features into multiple feature fragments.

[0213] If the execution device has already cached the features of the reference text, it can divide the features of the reference text into multiple feature fragments according to a pre-specified method. For example, the K matrix of the reference text can be divided into multiple sub-K matrices, and each sub-K matrix includes vectors corresponding to a fixed number of tokens (e.g., vectors corresponding to 64 or 128 tokens). Similarly, the V matrix of the reference text can be divided into multiple sub-V matrices, and each sub-V matrix includes vectors corresponding to a fixed number of tokens. In this way, a sub-K matrix and a sub-V matrix constitute a feature fragment.

[0214] Step 7072: Calculate the problem features of the problem text and the attention score for each feature segment.

[0215] After dividing the text into multiple feature segments, the execution device can extract the problem features of the problem text through the target model, and calculate the attention score of each feature segment and the problem features of the problem text.

[0216] Step 7073: Select at least one feature segment with the highest attention score.

[0217] Finally, based on the attention score of each feature fragment, the execution device can select at least one feature fragment with the highest attention score as the feature fragment for subsequent processing of the problem text.

[0218] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0219] Please see Figure 11 , Figure 11 This is a schematic diagram of a reasoning acceleration device provided in this application. Figure 11 As shown, the reasoning acceleration device includes: an acquisition module 1101, used to acquire question text and target reference text, wherein the question text indicates the question to be answered and the target reference text is the text to be referenced when answering the question; a processing module 1102, used to determine at least one target feature segment from multiple feature segments based on the relevance of the feature segments among multiple feature segments to the question text; the multiple feature segments are obtained by performing feature extraction on multiple text segments by a target model respectively, and the multiple text segments are divided based on the target reference text; the processing module 1102 is also used to process the question text by a target model based on at least one target feature segment to obtain the answer text.

[0220] In one possible implementation, the processing module 1102 is further configured to: determine whether the storage medium stores features of the reference text, wherein the storage medium stores features of multiple reference texts, and the features of the target reference text include multiple feature fragments; if the storage medium already stores features of the target reference text, read at least one target feature fragment from the storage medium based on the degree of relevance between the feature fragments among the multiple feature fragments and the problem text.

[0221] In one possible implementation, the processing module 1102 is further configured to: perform feature extraction on the problem text through the target model to obtain problem features; determine at least one target feature segment based on the correlation between the problem features and feature segments among multiple feature segments stored in the storage medium; and read at least one target feature segment from the storage medium.

[0222] In one possible implementation, the problem text appears in the first session, and multiple feature fragments are obtained by feature extraction performed before the first session and stored in the storage medium.

[0223] In one possible implementation, the target reference text appeared in a second session prior to the first session, and multiple feature fragments were extracted and stored in the storage medium during the second session.

[0224] In one possible implementation, the processing module 1102 is further configured to: determine the degree of correlation between the first text fragment and the problem text and extract feature fragments of the first text fragment by means of a target model when the features of the target reference text are not stored in the storage medium, wherein the first text fragment is one of a plurality of text fragments; and determine at least one target feature fragment based on the degree of correlation between the plurality of text fragments and the problem text.

[0225] In one possible implementation, the processing module 1102 is further configured to: process the first text fragment and the question text through the target model to obtain an output result; and determine the degree of correlation between the first text fragment and the question text based on the confidence level of the output result.

[0226] In one possible implementation, the question text appears in the Nth round of dialogue in the first session, where the first session includes multiple rounds of dialogue and N is a positive integer; the acquisition module 1101 is further configured to acquire historical dialogue content or a target historical dialogue summary, where the historical dialogue content includes content from at least one round of the first target dialogue, and the target historical dialogue summary includes a summary of content from at least one round of the second target dialogue; the first target dialogue is the dialogue preceding the Nth round of dialogue in the first session, and the second target dialogue is the dialogue preceding the first target dialogue in the first session; the processing module 1102 is further configured to perform feature extraction on the historical dialogue summary or historical dialogue content using a target model to obtain historical interaction features; the processing module 1102 is further configured to process the question text using a target model based on the historical interaction features and at least one target feature fragment.

[0227] In one possible implementation, the target historical dialogue summary is selected based on the relevance of multiple historical dialogue summaries to the question text, with the multiple historical interaction summaries corresponding to question-and-answer interaction content in different historical time periods.

[0228] In one possible implementation, the target model is a large language model, with multiple feature fragments including a KV cache.

[0229] In one possible implementation, the sum of the lengths of the question text and the target reference text is greater than a preset length.

[0230] Both the acquisition module 1101 and the processing module 1102 can be implemented in software or in hardware. For example, the implementation of the processing module 1102 will be described below. Similarly, the implementation of the acquisition module 1101 can be referenced to that of the processing module 1102.

[0231] As an example of a software functional unit, processing module 1102 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 1102 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0232] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0233] As an example of a hardware functional unit, the processing module 1102 may include at least one computing device, such as a server. Alternatively, the processing module 1102 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0234] The processing module 1102 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1102 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1102 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0235] Please see Figure 12 , Figure 12 A schematic diagram of the structure of a computing device provided in this application. Figure 12 The computing device 1200 shown can be used to execute the inference acceleration method provided in this embodiment. For example... Figure 12 As shown, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0236] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1202 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1202 may include a path for transmitting information between various components of the computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0237] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0238] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0239] The memory 1206 stores executable program code, and the processor 1204 executes this executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the aforementioned inference acceleration method. That is, the memory 1206 stores instructions for executing the inference acceleration method.

[0240] The communication interface 1208 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0241] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0242] Please see Figure 13 , Figure 13 This is a schematic diagram of a computing device cluster provided in this application. Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same instructions for executing task processing methods.

[0243] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing task processing methods. In other words, a combination of one or more computing devices 1200 can jointly execute instructions for executing task processing methods.

[0244] It should be noted that the memory 1206 in different computing devices 1200 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 1206 of different computing devices 1200 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0245] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 14 One possible implementation method is shown. Figure 14 This is a schematic diagram of another computing device cluster structure provided in this application. Figure 14 As shown, in a computing device cluster, two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1206 in computing device 1200A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 1206 in computing device 1200B stores instructions for executing the functions of the processing module.

[0246] It should be understood that Figure 14 The functions of the computing device 1200A shown can also be performed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be performed by multiple computing devices 1200.

[0247] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0248] Please refer to Figure 15 , Figure 15 This is a schematic diagram of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in some embodiments, wherein the above-described... Figure 2 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0249] Figure 15 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0250] In one embodiment, the computer-readable storage medium 1500 is provided using a signal bearer medium 1501. The signal bearer medium 1501 may include one or more program instructions 1502, which, when executed by one or more processors, can provide the above-mentioned... Figure 2 The described function or part of the function.

[0251] In some examples, the signal carrying medium 1501 may include a computer-readable medium 1503, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0252] In some embodiments, the signal-bearing medium 1501 may comprise a computer-recordable medium 1504, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1501 may comprise a communication medium 1505, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1501 may be transmitted by a wireless communication medium 1505 (e.g., a wireless communication medium conforming to the IEEE 1202.X standard or other transmission protocols).

[0253] One or more program instructions 1502 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1502 conveyed to the computing device via a computer-readable medium 1503, a computer-recordable medium 1504, and / or a communication medium 1505.

[0254] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0255] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0256] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0257] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for accelerating inference, characterized in that, include: Obtain the question text and the target reference text, wherein the question text is used to indicate the question to be answered, and the target reference text is the text to be referenced when answering the question; Based on the relevance of a feature fragment among multiple feature fragments to the question text, at least one target feature fragment is determined from the multiple feature fragments; the multiple feature fragments are obtained by performing feature extraction on multiple text fragments respectively by a target model, and the multiple text fragments are divided based on the target reference text; Based on the at least one target feature fragment, the question text is processed by the target model to obtain the answer text.

2. The method according to claim 1, characterized in that, The step of determining at least one target feature segment from the plurality of feature segments based on the relevance of the feature segments to the question text includes: Determine whether the storage medium stores the features of the reference text, wherein the storage medium stores the features of multiple reference texts, and the features of the target reference text include the multiple feature fragments; If the features of the target reference text are already stored in the storage medium, at least one target feature fragment is read from the storage medium according to the degree of relevance between the feature fragments among the plurality of feature fragments and the problem text.

3. The method according to claim 2, characterized in that, The step of reading at least one target feature segment from the storage medium based on the relevance of the feature segments among the plurality of feature segments to the question text includes: The problem features are obtained by performing feature extraction on the problem text using the target model. The at least one target feature segment is determined based on the correlation between the problem features and the feature segments among the plurality of feature segments stored in the storage medium, and the at least one target feature segment is read from the storage medium.

4. The method according to claim 2 or 3, characterized in that, The question text appears in the first session, and the multiple feature fragments are obtained by feature extraction performed before the first session and stored in the storage medium.

5. The method according to claim 4, characterized in that, The target reference text appeared in a second session prior to the first session, and the plurality of feature fragments were extracted and stored in the storage medium during the second session.

6. The method according to any one of claims 2-5, characterized in that, The step of determining at least one target feature segment from the plurality of feature segments based on the relevance of the feature segments to the question text includes: In the absence of features of the target reference text stored in the storage medium, the relevance of the first text fragment to the problem text is determined by the target model, and feature fragments of the first text fragment are extracted. The first text fragment is one of the plurality of text fragments. The at least one target feature segment is determined based on the degree of relevance between the plurality of text segments and the question text.

7. The method according to claim 6, characterized in that, The step of determining the relevance between the first text fragment and the question text using the target model includes: The target model processes the first text fragment and the question text to obtain the output result; Based on the confidence level of the output results, the degree of relevance between the first text fragment and the question text is determined.

8. The method according to any one of claims 1-7, characterized in that, The question text appears in the Nth round of dialogue in the first session, where the first session includes multiple rounds of dialogue, and N is a positive integer; the method further includes: Obtain historical dialogue content or target historical dialogue summary, wherein the historical dialogue content includes content from at least one round of the first target dialogue, and the target historical dialogue summary includes a summary of content from at least one round of the second target dialogue; the first target dialogue is the dialogue prior to the Nth round of dialogue in the first session, and the second target dialogue is the dialogue prior to the first target dialogue in the first session; The target model is used to extract features from the historical dialogue summary or the historical dialogue content to obtain historical interaction features. The process of processing the problem text using the target model based on the at least one target feature fragment includes: The question text is processed by the target model based on the historical interaction features and the at least one target feature fragment.

9. The method according to claim 7, characterized in that, The target historical dialogue summary is selected based on the relevance of multiple historical dialogue summaries to the question text, and the multiple historical interaction summaries correspond to the question-and-answer interaction content in different historical time periods.

10. The method according to any one of claims 1-9, characterized in that, The target model is a large language model, and the multiple feature fragments include a key-value cache (KV Cache).

11. The method according to any one of claims 1-10, characterized in that, The combined length of the question text and the target reference text is greater than the preset length.

12. The method according to any one of claims 2-11, characterized in that, The storage medium includes persistent storage media.

13. A reasoning acceleration device, characterized in that, include: The acquisition module is used to acquire question text and target reference text. The question text indicates the question to be answered, and the target reference text is text to be referenced when answering the question. The processing module is configured to determine at least one target feature segment from the multiple feature segments based on the relevance of the feature segments among the multiple feature segments to the question text; the multiple feature segments are obtained by performing feature extraction on the multiple text segments respectively by the target model, and the multiple text segments are divided based on the target reference text; The processing module is further configured to process the question text based on the at least one target feature fragment using the target model to obtain the answer text.

14. The apparatus according to claim 13, characterized in that, The processing module is further configured to: Determine whether the storage medium stores the features of the reference text, wherein the storage medium stores the features of multiple reference texts, and the features of the target reference text include the multiple feature fragments; If the features of the target reference text are already stored in the storage medium, at least one target feature fragment is read from the storage medium according to the degree of relevance between the feature fragments among the plurality of feature fragments and the problem text.

15. The apparatus according to claim 14, characterized in that, The processing module is further configured to: The problem features are obtained by performing feature extraction on the problem text using the target model. The at least one target feature segment is determined based on the correlation between the problem features and the feature segments among the plurality of feature segments stored in the storage medium, and the at least one target feature segment is read from the storage medium.

16. The apparatus according to claim 14 or 15, characterized in that, The question text appears in the first session, and the multiple feature fragments are obtained by feature extraction performed before the first session and stored in the storage medium.

17. The apparatus according to claim 16, characterized in that, The target reference text appeared in a second session prior to the first session, and the plurality of feature fragments were extracted and stored in the storage medium during the second session.

18. The apparatus according to any one of claims 14-17, characterized in that, The processing module is further configured to: In the absence of features of the target reference text stored in the storage medium, the relevance of the first text fragment to the problem text is determined by the target model, and feature fragments of the first text fragment are extracted. The first text fragment is one of the plurality of text fragments. The at least one target feature segment is determined based on the degree of relevance between the plurality of text segments and the question text.

19. The apparatus according to claim 18, characterized in that, The processing module is further configured to: The target model processes the first text fragment and the question text to obtain the output result; Based on the confidence level of the output results, the degree of relevance between the first text fragment and the question text is determined.

20. The apparatus according to any one of claims 13-19, characterized in that, The question text appears in the Nth round of dialogue in the first session, where the first session includes multiple rounds of dialogue and N is a positive integer. The acquisition module is further configured to acquire historical dialogue content or target historical dialogue summary, wherein the historical dialogue content includes content from at least one round of the first target dialogue, and the target historical dialogue summary includes a summary of content from at least one round of the second target dialogue; the first target dialogue is the dialogue prior to the Nth round of dialogue in the first session, and the second target dialogue is the dialogue prior to the first target dialogue in the first session; The processing module is further configured to perform feature extraction on the historical dialogue summary or the historical dialogue content through the target model to obtain historical interaction features; The processing module is further configured to process the question text using the target model based on the historical interaction features and the at least one target feature fragment.

21. The apparatus according to claim 19, characterized in that, The target historical dialogue summary is selected based on the relevance of multiple historical dialogue summaries to the question text, and the multiple historical interaction summaries correspond to the question-and-answer interaction content in different historical time periods.

22. A computing device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the computing device performs the method as described in any one of claims 1 to 12.

23. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 12.

24. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 12.

25. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 12.

Images