Inference acceleration method based on calculation with check and related device

Abstract

The invention relates to a reasoning acceleration method based on a check substitution algorithm, which is applied to a reasoning process of an acceleration model. The reasoning acceleration method comprises the following steps: aiming at an obtained problem text, firstly obtaining feature abstracts corresponding to a plurality of feature fragments obtained by respectively carrying out feature extraction on text fragments in advance; by comparing the correlation degree between the feature abstracts and the features of the problem text, part of the feature fragments can be selected from the stored feature fragments to serve as the features of the reference text to process the problem text. According to the scheme, the memory of the large model is stored on the basis of AI native storage, secondary reasoning of the large model is not needed, and the characteristics can be used and taken by searching instead of calculation, so that the reasoning efficiency of the large model is improved. Moreover, the feature abstracts with smaller data volume are adopted to represent the feature fragments, so that the feature fragments are screened based on the feature abstracts, the operand in the feature processing process can be effectively reduced, and the reasoning speed of the target model is further improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a reasoning acceleration method and related apparatus based on query-based computation. Background Technology

[0002] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data. They can not only generate natural language text, but also deeply understand the meaning of text and handle various natural language tasks, such as text summarization, intelligent question answering, and text translation.

[0003] With the development of large language models, their growth in scale is particularly remarkable, with the number of parameters jumping from billions to trillions. This increase in parameters allows large language models to capture the subtleties of human language more precisely and to gain a deeper understanding of its complexity.

[0004] However, due to the sheer number of parameters in large language models, each use incurs significant computational and time overhead due to the complexity of parameter calculation. This is especially true in long-sequence reasoning scenarios (such as question-answering), where the input to a large language model is a long sequence of text. Large language models often require a substantial amount of time to execute the reasoning process, resulting in slow response times and difficulty in meeting user needs. Summary of the Invention

[0005] This application provides a reasoning acceleration method based on lookup-based computation, which is applied to accelerate the model's reasoning process by finding stored features instead of recalculating them.

[0006] Firstly, a reasoning acceleration method is provided for scenarios involving question answering based on a target model. This method includes: the execution device first acquiring question text, which can be a user-provided question indicating the question to be answered.

[0007] In response to acquiring the problem text, the execution device obtains multiple feature summaries corresponding to multiple feature fragments. These multiple feature fragments are obtained by performing feature extraction on multiple text fragments using a target model. Each feature fragment has a corresponding feature summary, meaning there is a one-to-one correspondence between the multiple feature fragments and the multiple feature summaries. The first feature summary corresponding to the first feature fragment is obtained by performing feature extraction on the first feature fragment, and the data size of the first feature summary is smaller than the data size of the first feature fragment. In other words, the feature summary is used to represent the corresponding feature fragment to facilitate rapid subsequent feature fragment filtering. Specifically, the feature summary may be, for example, information such as the features or fingerprints of the corresponding feature fragment.

[0008] Then, based on the correlation between multiple feature summaries and question features, a target feature segment is determined from multiple feature segments. The question features are obtained by performing feature extraction on the question text using the target model. For example, the execution device can actually filter out the feature segments most relevant to the question text from multiple feature segments based on the feature summaries.

[0009] Finally, based on the target feature fragments, the execution device processes the question text through the target model to obtain the answer text. That is, the execution device can use the first feature fragment as a feature of the reference text to assist the target model in processing the question text, thereby obtaining the answer text.

[0010] In this scheme, for the acquired question text, feature summaries corresponding to multiple feature segments obtained by pre-extracting features from each text fragment are first acquired. By comparing the correlation between the feature summaries and the features of the question text, some feature segments can be selected from the stored multiple feature segments as features of the reference text to process the question text. This effectively reduces the amount of features that the target model needs to process during inference, realizing lookup-based computation in AI storage (i.e., replacing recalculation of feature segments by finding relevant feature segments), thus improving the inference speed of the target model. Furthermore, since the feature summaries corresponding to each feature segment are small in size and used to represent the feature segment, filtering feature segments based on feature summaries can effectively reduce the computational load of feature processing, further improving the inference speed of the target model.

[0011] In one possible implementation, multiple text fragments are obtained by segmenting a first reference text, which is the text used for reference when answering the question indicated by the question text. That is, multiple text fragments corresponding to multiple feature fragments are all obtained by segmenting the same reference text.

[0012] In this scheme, by selecting some feature segments from the feature segments corresponding to the same reference text as feature inputs to the target model, it is possible to filter out feature segments that can provide effective information for processing the problem text from multiple feature segments corresponding to the same reference text, and remove feature segments with redundant information, thereby reducing the number of feature segments that the target model needs to process, thus improving the efficiency of the target model in performing inference and ensuring the inference accuracy of the target model.

[0013] In one possible implementation, multiple feature fragments and their corresponding feature summaries are pre-stored on a storage medium. Therefore, the execution device may read the multiple feature summaries from the storage medium, determine a target feature fragment among the multiple feature fragments based on the correlation between the multiple feature summaries and the problem feature, and then read the target feature fragment from the storage medium. Here, the correlation between the multiple feature summaries and the problem feature may refer to the correlation between each feature summary and the problem feature.

[0014] In other words, both the feature fragments and their corresponding feature summaries are pre-stored on the storage medium. The execution device calculates the correlation between the feature summaries and the problem features to locate the corresponding feature fragments (i.e., the target feature fragments mentioned above) from the storage medium, and then reads and reuses the corresponding feature fragments, thereby avoiding recalculation of the feature fragments and improving the inference speed of the target model.

[0015] In one possible implementation, the storage medium stores a set of feature fragments and a feature summary corresponding to each feature fragment in the set. The set of feature fragments includes multiple feature fragments. That is, the storage medium also stores other feature fragments and feature summaries; the aforementioned multiple feature fragments are only a portion of the feature fragments stored in the storage medium.

[0016] In one possible implementation, the problem text appears in the first session. Multiple feature fragments are obtained by the target model in the second session by performing feature extraction on multiple text fragments and stored in the storage medium. Multiple feature summaries are also obtained in the second session based on the multiple feature fragments and stored in the storage medium. The second session occurs before the first session. That is, the multiple feature fragments are obtained by the execution device extracting text fragments that appear in the second session before the first session. By extracting feature fragments corresponding to text fragments that have already appeared in the session, the stored feature fragments can be directly reused when the corresponding text fragments continue to appear in subsequent sessions, thereby achieving feature lookup instead of computation and improving the inference speed of the model.

[0017] In one possible implementation, the multiple text fragments are obtained by segmenting a first reference text, which is the text used for reference when answering the question indicated by the question text, and appears in the second session. In another possible implementation, when processing the question text through the target model, the execution device acquires a beginning feature fragment and an end feature fragment. The beginning feature fragment is obtained by performing feature extraction on the first text fragment of the first reference text, and the end feature fragment is obtained by performing feature extraction on the last text fragment of the first reference text. That is, the aforementioned multiple text fragments are actually text fragments located between the first and last text fragments.

[0018] Then, the execution device processes the question text using the target model based on the beginning feature fragment, target feature fragment, and end feature fragment to obtain the answer text. That is, the execution device may concatenate the beginning feature fragment, target feature fragment, and end feature fragment as the features of the entire first reference text, thereby assisting the target model in processing the question text and obtaining the answer text output by the target model.

[0019] In this scheme, by isolating the feature segments corresponding to the beginning and end of the text segments, and combining the isolated feature segments with the feature segments corresponding to the selected middle text segments to form the features of the reference text, it is ensured that the feature segments input to the target model must include the feature segments corresponding to the beginning and end of the reference text. This ensures that important information from the reference text can be input into the target model, effectively guaranteeing the accuracy of model inference.

[0020] In one possible implementation, multiple feature fragments and their corresponding feature summaries are pre-stored on a storage medium. The execution device then reads the beginning and end feature fragments from this storage medium.

[0021] In one possible implementation, the first feature segment includes multiple sub-feature segments corresponding to different tokens. The first feature summary is a sub-feature segment selected from the multiple sub-feature segments based on their similarity to the final feature segment. Each token may correspond to one sub-feature segment. Based on the similarity between each sub-feature segment and the final feature segment, the execution device can determine one or more sub-feature segments with the highest similarity as the first feature summary.

[0022] In this approach, since the last text fragment is usually the text fragment closest to the question text in the input of the target model, it often provides more information to the target model. Therefore, by selecting some sub-feature fragments with high similarity to the last feature fragment as feature summaries from multiple sub-feature fragments, a representative representation of the entire feature fragment can be effectively constructed.

[0023] In one possible implementation, a single feature segment among multiple feature segments includes multiple sub-feature segments corresponding to different tokens, and the feature summary is obtained by taking the mean, maximum, and / or minimum values ​​of the multiple sub-feature segments.

[0024] In one possible implementation, the target model includes a first neural network layer and M adjacent second neural network layers, where M is a positive integer (e.g., 1, 2, or 3). Multiple feature segments are obtained by the first neural network layer performing feature extraction on multiple text segments, and the target feature segment is the first target feature segment.

[0025] When processing problem text through the target model, the execution device first determines the second target feature fragment based on the position of the first target feature fragment in multiple feature fragments, and then determines the second target feature fragment from multiple feature fragments obtained by the second neural network layer performing feature extraction on multiple text fragments.

[0026] Then, based on the first target feature fragment and the second target feature fragment, the execution device processes the question text through the target model. That is, the execution device actually processes the question text through the target model based on feature fragments extracted from different neural network layers in the target model.

[0027] In this scheme, when the target model includes a large number of neural network layers, in order to improve the efficiency of feature fragment selection, the execution device only performs the selection process on the feature fragments extracted by a portion of the neural network layers, and reuses the obtained selection results to select feature fragments extracted by other neural network layers, thereby reducing the amount of computation when performing feature fragment selection and improving the inference speed of the target model.

[0028] In one possible implementation, M is determined based on the characteristics of the target model. That is, the size of M is related to the target model itself; for example, the size of the neural network layers in the target model and the specific structure of those layers will affect the actual value of M. In practical applications, a batch of data can be used to test the target model to determine which neural network layers adjacent to the first neural network layer will have the same or similar feature fragment retrieval results as the first neural network layer, thereby determining the value of M.

[0029] In one possible implementation, the multiple feature segments are obtained by performing feature extraction on multiple text segments in the first reference text, and the target feature segment is the first target feature segment.

[0030] In this method, the execution device can also acquire multiple feature fragments of the second reference text, which are obtained by performing feature extraction on multiple text fragments in the second reference text.

[0031] Then, based on the degree of correlation between the feature summaries corresponding to multiple feature fragments of the second reference text and the problem features, the execution device determines the second target feature fragment among the multiple feature fragments of the second reference text.

[0032] Finally, based on the splicing features, the execution device processes the problem text through the target model, where the splicing features are obtained based on the first target feature fragment and the second target feature fragment.

[0033] In one possible implementation, the splicing features are obtained based on a first target feature fragment, a first positional encoding, a second target feature fragment, and a second positional encoding. The first positional encoding is used to indicate the position of the first target feature fragment corresponding to the first reference text and the position of the text fragment corresponding to the first target feature fragment in the first reference text. The second positional encoding is used to indicate the position of the second target feature fragment corresponding to the second reference text and the position of the text fragment corresponding to the second target feature fragment in the second reference text.

[0034] In this scheme, by setting corresponding position codes for feature segments corresponding to different reference texts, and the position codes can simultaneously indicate the reference text corresponding to the feature segment and its corresponding position in the reference text, the target model can effectively distinguish and perceive the input feature segments, which is beneficial to improving the accuracy of the target model.

[0035] In one possible implementation, after obtaining the first target feature segment and the second target feature segment, the execution device first determines a first sub-feature segment from the first target feature segment and the second target feature segment. The first sub-feature segment includes sub-feature segments located in the first target feature segment and / or sub-feature segments located in the second target feature segment. Furthermore, the first sub-feature segment is the feature segment in the first target feature segment and the second target feature segment that needs to be recalculated.

[0036] Then, the execution device, based on the first target feature fragment and the second target feature fragment, re-extracts the second sub-feature fragment corresponding to the target sub-text fragment through the target model. The target sub-text fragment is the text fragment corresponding to the first sub-feature fragment. In other words, after determining the first sub-feature fragment that needs to be calculated, the execution device performs the recalculation of the feature fragment based on the first target feature fragment and the second target feature fragment, instead of performing the calculation of the feature fragment based on the first reference text and the second reference text respectively.

[0037] Secondly, in the first target feature segment and the second target feature segment, the execution device updates the first sub-feature segment to the second sub-feature segment, so as to obtain the updated first target feature segment and the updated second target feature segment;

[0038] Finally, the execution device obtains the spliced ​​features based on the updated first target feature fragment and the updated second target feature fragment.

[0039] In this scheme, in order to improve the accuracy of feature fragments and ensure the efficiency of inference, the execution device recalculates some key feature fragments in the selected feature fragments, thereby improving the inference accuracy of the target model.

[0040] In one possible implementation, after obtaining the question text, the execution device first obtains text summaries of multiple pre-stored reference texts and determines the similarity between the text summaries of the multiple reference texts and the question text. Then, the execution device identifies the first and second reference texts among the multiple reference texts whose text summaries have the highest similarity to the question text.

[0041] In this scheme, by pre-storing text summaries of multiple reference texts and calculating the similarity between the text summaries of each reference text and the problem text, one or more reference texts that are most relevant to the problem text can be selected from multiple reference texts to assist the target model in processing the problem text, which is beneficial to improving the accuracy of the target model in processing the problem text.

[0042] In one possible implementation, multiple feature fragments and multiple feature summaries corresponding to the multiple feature fragments are pre-stored on a storage medium, including a persistent storage medium.

[0043] In this scheme, a storage medium is used to pre-store the features of the reference text. This allows the required features to be read directly from the storage medium when specific features of the reference text are needed, without having to recalculate the features of the reference text. This achieves feature lookup instead of calculation and improves the inference speed of the target model.

[0044] For example, by offloading the KV cache of hot documents to a storage medium for persistence, the KV cache can be reused in subsequent sessions through similarity retrieval, thus avoiding the recalculation of the KV cache and improving the inference speed of the model.

[0045] Secondly, a reasoning acceleration device is provided, comprising: an acquisition module for acquiring question text, the question text indicating a question to be answered; the acquisition module is further configured to acquire multiple feature summaries corresponding to multiple feature fragments, the multiple feature fragments being obtained by performing feature extraction on the multiple text fragments using a target model, a first feature summary corresponding to a first feature fragment among the multiple feature fragments being obtained by performing feature extraction on the first feature fragment, and the data volume of the first feature summary being less than the data volume of the first feature fragment; a processing module for determining a target feature fragment among the multiple feature fragments based on the correlation between the multiple feature summaries and question features, the question features being obtained by performing feature extraction on the question text using a target model; and the processing module is further configured to process the question text using a target model based on the target feature fragment to obtain answer text.

[0046] In one possible implementation, the multiple text fragments are obtained by segmenting a first reference text, which is the text used for reference when answering the question indicated by the question text.

[0047] In one possible implementation, multiple feature fragments and multiple feature summaries corresponding to the multiple feature fragments are pre-stored on a storage medium; the acquisition module is further configured to read the multiple feature summaries from the storage medium; the processing module is further configured to determine the target feature fragment among the multiple feature fragments based on the degree of correlation between the multiple feature summaries and the problem features, and read the target feature fragment from the storage medium.

[0048] In one possible implementation, the acquisition module is further configured to acquire the beginning feature fragment and the end feature fragment, wherein the beginning feature fragment is obtained by performing feature extraction on the first text fragment of the first reference text, and the end feature fragment is obtained by performing feature extraction on the last text fragment of the first reference text; the processing module is further configured to process the question text through the target model based on the beginning feature fragment, the target feature fragment, and the end feature fragment to obtain the answer text.

[0049] In one possible implementation, the first feature fragment includes multiple sub-feature fragments corresponding to different tokens, and the first feature summary is a sub-feature fragment selected from multiple sub-feature fragments based on the similarity between the multiple sub-feature fragments and the final feature fragment.

[0050] In one possible implementation, the first feature fragment includes multiple sub-feature fragments corresponding to different tokens, and the first feature summary is obtained by taking the mean, maximum, and / or minimum values ​​of the multiple sub-feature fragments.

[0051] In one possible implementation, the target model includes a first neural network layer and M second neural network layers adjacent to the first neural network layer, where M is a positive integer. Multiple feature segments are obtained by the first neural network layer performing feature extraction on multiple text segments, and the target feature segment is the first target feature segment. The processing module is further configured to: determine the second target feature segment from the multiple feature segments obtained by the second neural network layer performing feature extraction on multiple text segments based on the position of the first target feature segment among the multiple feature segments; and process the problem text through the target model based on the first target feature segment and the second target feature segment.

[0052] In one possible implementation, M is determined based on the characteristics of the target model.

[0053] In one possible implementation, the multiple feature fragments are obtained by performing feature extraction on multiple text fragments in the first reference text, and the target feature fragment is the first target feature fragment; the acquisition module is further used to acquire multiple feature fragments of the second reference text, which are obtained by performing feature extraction on multiple text fragments in the second reference text; the processing module is further used to determine the second target feature fragment among the multiple feature fragments of the second reference text based on the correlation between the feature summaries corresponding to the multiple feature fragments of the second reference text and the question features; the processing module is further used to process the question text through the target model based on the spliced ​​features, which are obtained based on the first target feature fragment and the second target feature fragment.

[0054] In one possible implementation, the processing module is further configured to: determine a first sub-feature segment in the first target feature segment and the second target feature segment, the first sub-feature segment including sub-feature segments located in the first target feature segment and / or sub-feature segments located in the second target feature segment; re-extract a second sub-feature segment using the target model based on the first target feature segment and the second target feature segment; update the first sub-feature segment to the second sub-feature segment in the first target feature segment and the second target feature segment to obtain an updated first target feature segment and an updated second target feature segment; and obtain a concatenated feature based on the updated first feature segment and the updated third feature segment.

[0055] In one possible implementation, the acquisition module is further configured to acquire text summaries of multiple pre-stored reference texts; the processing module is further configured to determine the similarity between the text summaries of the multiple reference texts and the question text; the processing module is further configured to determine a first reference text and a second reference text among the multiple reference texts based on the similarity between the text summaries of the multiple reference texts and the question text.

[0056] In one possible implementation, multiple feature fragments and multiple feature summaries corresponding to the multiple feature fragments are pre-stored on a storage medium, including a persistent storage medium.

[0057] Thirdly, an inference acceleration device is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the inference acceleration device to perform the method of any of the above aspects.

[0058] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods described in any of the preceding aspects.

[0059] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0060] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0061] In a seventh aspect, a computing device is provided, the computing device including an inference acceleration device of the third aspect or a chip system of the sixth aspect, wherein the inference acceleration device or the chip system in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0062] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0063] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0064] Figure 1 A schematic diagram of a system architecture provided for this application;

[0065] Figure 2 A flowchart illustrating a reasoning acceleration method provided in this application;

[0066] Figure 3 A flowchart illustrating a process for processing problem text using a target model, as provided in this application;

[0067] Figure 4 A flowchart illustrating another approach to processing problem text using a target model, as provided in this application;

[0068] Figure 5 This application provides a schematic diagram of filtering feature fragments extracted from different neural network layers;

[0069] Figure 6 This application provides a schematic diagram of performing filtering on feature fragments corresponding to different reference texts;

[0070] Figure 7 This application provides a flowchart illustrating the processing of user-provided question text in a question-and-answer scenario.

[0071] Figure 8 A schematic diagram illustrating the processing flow of a problem text provided in this application;

[0072] Figure 9 A flowchart illustrating a process for determining the features of reference text for an input model based on question text, as provided in this application;

[0073] Figure 10 A flowchart illustrating a process for determining feature segments of an input target model, as provided in this application;

[0074] Figure 11 A schematic diagram of a reasoning acceleration device provided in this application;

[0075] Figure 12 A schematic diagram of the structure of a computing device provided in this application;

[0076] Figure 13 This application provides a schematic diagram of the structure of a computing device cluster;

[0077] Figure 14 This application provides a schematic diagram of another computing device cluster structure.

[0078] Figure 15 This is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0080] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0081] To facilitate understanding, some technical terms used in this application will be introduced below.

[0082] (1) Large Language Model

[0083] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0084] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0085] Currently, large language models are mainly composed of Transformer networks.

[0086] (2) Transformer network

[0087] Transformer networks are powerful sequence models, but the computation time and memory required increase quadratically with sequence length, significantly increasing the hardware's storage and computing power demands. Essentially, Transformer networks employ a self-attention mechanism. Self-attention is a mechanism that associates different positions within a single sequence to compute a representation of the same sequence, playing a crucial role in machine reading, abstract summarization, and image description generation.

[0088] Taking the Transformer network applied to natural language processing as an example, the Transformer network processes input data of arbitrary length and generates new feature representations of the input data, which are then converted into target words. The self-attention network layer in the Transformer network uses an attention mechanism to capture the relationships between all other words, thereby generating new feature representations for each word. The advantage of the Transformer network's self-attention network is that the attention mechanism can directly capture the relationships between all words in a sentence without considering word positions.

[0089] (3) Key-Value Cache (KV Cache)

[0090] When using Transformer networks for natural language processing tasks, they typically employ a self-attention mechanism to process the input sequence. In this mechanism, the Transformer network generates a corresponding key (K) vector, value (V) vector, and query (Q) vector for each token in the input sequence. Further, the Transformer network calculates the degree of matching between each query vector and all key vectors, usually achieved through a dot product. Then, using the matching degree between query and key vectors as weights, it calculates a weighted sum of all value vectors to obtain the final result.

[0091] When the Transformer network processes the input sequence, the key vector generated by the Transformer network for each word in the input sequence can be stored in a single matrix, and the value vector generated by the Transformer network for each word in the input sequence can be stored in another matrix. Therefore, KV Cache refers to the key matrix and value matrix generated by the Transformer network for the words in the input sequence, and these key matrices and value matrices are cached.

[0092] (4) Token

[0093] A lexical is the smallest semantic unit in natural language processing, typically derived from the decomposition of long texts such as words, sentences, paragraphs, and articles. Generally, lexical units can be composed of letters, words, or symbols; for example, they can be composed of one or more letters or one or more words. Simply put, a lexical unit can be understood as a "fragment" of a word or phrase—the smallest indivisible semantic unit. For example, "waterfall" can be broken down into two lexical units: "water" and "fall." Additionally, punctuation marks can also be decomposed into lexical units because they affect the semantic understanding of the entire text. For example, "I don't know." can be decomposed into five lexical units: "I," "don," "'t," "know," and ".".

[0094] (5) Session

[0095] A session is a concept of a temporary interactive state established between a client and a server, used to track user actions over a period of time. Sessions are typically used to maintain user state and information, such as login status and conversation content.

[0096] Specifically, the conversation can be explained from the following aspects.

[0097] Continuous interaction: A session refers to a series of requests and responses between a user and a server over a period of time.

[0098] Lifecycle: A session typically begins when a user logs in or accesses the application and ends when the user logs out or the session times out. The timeout period is usually set by the server; for example, 30 minutes of inactivity will invalidate the session.

[0099] Information storage: The server stores user state data, such as user identity, permissions, and personalized settings. The client informs the server which session it is associated with through some identifier (such as a Session ID).

[0100] Session ID: To identify different users' sessions, the server assigns a unique identifier (Session ID) to each session. This Session ID is usually stored on the client and sent to the server with each request.

[0101] In general, in the human-computer dialogue scenario of this application, a session refers to the interaction between a user and an execution device (such as a server) over a period of time. Therefore, a session can include one or more rounds of dialogue between the user and the execution device.

[0102] Currently, in some long-sequence reasoning scenarios, large language models (MLMs) are inputted with lengthy sequences of text. These MLMs often spend a significant amount of time executing the reasoning process, resulting in slow response times and difficulty meeting user needs. For example, in question-answering scenarios, to ensure high-quality answers from MLMs, reference texts (such as papers, user manuals, and search results) are often provided simultaneously. This allows the MLM to provide targeted answers based on the content of the reference texts. In some cases, the reference texts may contain a lot of information, leading to a long input text for the MLM, which can negatively impact its reasoning speed.

[0103] In view of this, this application provides an inference acceleration method. For the acquired question text, it first obtains feature summaries corresponding to multiple feature segments obtained by pre-extracting features from the text fragments. By comparing the correlation between the feature summaries and the features of the question text, it can select some feature segments as features of the reference text to process the question text, effectively reducing the number of features the target model needs to process during inference and improving the inference speed of the target model. Furthermore, since the feature summaries corresponding to each feature segment are relatively small in size and used to represent the feature segments, selecting feature segments based on feature summaries can effectively reduce the computational load of the feature processing process, further improving the inference speed of the target model.

[0104] Please see Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided for this application. (For example...) Figure 1As shown, in this system architecture, the execution device 10 can be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster). The execution device 10 includes an accelerator 101 and a processor 102. The accelerator 101 is used to run the target model (e.g., a large language model) to process the inference tasks passed by the processor 102. The processor 102 is used to obtain task requests from clients (e.g., requests to answer a question) and schedule the accelerator 101 to process the specified inference task based on the task request from the client.

[0105] Optionally, the execution device 10 can be used in conjunction with other computing devices, such as data storage devices, load balancers, etc.; the execution device 10 can be deployed on a single physical site or distributed across multiple physical sites.

[0106] In addition, the system architecture also includes a data storage system 11, which is used to store data such as problem text, reference text, KV cache, or program code.

[0107] Optionally, for persistent data storage, the data storage system 11 can be located external to the execution device 10 and exchange data with the execution device 10 via a network. Alternatively, if the execution device 10 is a physical host, the data storage system 11 can also be located internally to the execution device 10, such as exchanging data with the processor via a bus. In this case, the data storage system 11 functions as a hard disk. With the data storage system 11, the execution device 10 can use data in the data storage system 11 (such as KV Cache) or call program code in the data storage system to implement the inference acceleration method provided in this application.

[0108] Optionally, users can interact with execution device 10 using their respective local devices. For example, a client 121 is deployed on local device 12, and users interact with the execution device through client 121 on local device 12. Local device 12 can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, laptop, or smart car.

[0109] Local device 12 can interact with execution device 10 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0110] Optionally, during the implementation of the inference acceleration method by the execution device 10, the local device 12 can provide the execution device 10 with task requests or prompts, so that the execution device 10 can process the input text through the model to complete the task. Furthermore, after the execution device 10 executes the inference acceleration method and obtains the output result, it can feed the output result back to the local device 12.

[0111] Please see Figure 2 , Figure 2 This is a flowchart illustrating a reasoning acceleration method provided in this application. Figure 2 As shown, the execution flow of this inference acceleration method includes the following steps 201-204.

[0112] Step 201: Obtain the question text, which indicates the question to be answered.

[0113] In this application, the inference acceleration method can be applied to scenarios where question answering is performed based on a target model. In a question answering scenario, a user interacts with an execution device deployed with the target model via a client on their local device, sending question text to the execution device in exchange for an answer. Therefore, the execution device can obtain the question text from the user, which indicates the question requiring a response. For example, the question text could be, for instance, "How many goals did Messi score in the World Cup?" or "Who is the author of Harry Potter?"—any question composed of text.

[0114] Step 202: Obtain multiple feature summaries corresponding to multiple feature fragments. The multiple feature fragments are obtained by performing feature extraction on multiple text fragments by the target model. The first feature summary corresponding to the first feature fragment among the multiple feature fragments is obtained by performing feature extraction on the first feature fragment, and the data volume of the first feature summary is less than the data volume of the first feature fragment.

[0115] Generally, in order to enable the target model to answer the question text provided by the user more effectively, the execution device often provides the target model with reference text when processing the question text. This allows the target model to refer to the content of the reference text to answer the question, thereby improving the accuracy of the target model's answer.

[0116] Therefore, this step involves pre-extracting features from multiple text segments to obtain and store the corresponding feature segments. The execution device can then select some feature segments from the pre-stored feature segments as features of the reference text to assist the target model in processing the problem text. There can be a one-to-one correspondence between the multiple text segments and the multiple feature segments. Of course, one text segment can also correspond to multiple feature segments; this application does not impose any specific limitations on this.

[0117] Specifically, for multiple pre-divided text segments, the execution device first performs feature extraction on the text segments using a target model, thereby obtaining multiple feature segments corresponding to the text segments. That is, there is a one-to-one correspondence between the multiple text segments and the multiple feature segments. Furthermore, for each feature segment, the execution device sets a corresponding feature summary. The feature summary is extracted based on the corresponding feature segment, and the data size of the feature summary is smaller than that of the corresponding feature segment (i.e., the feature summary is used to represent the corresponding feature segment), in order to facilitate rapid selection of feature segments.

[0118] Optionally, the multiple text fragments mentioned above are obtained by segmenting a first reference text, which is the text used for reference when answering the question indicated by the question text. That is, the multiple text fragments corresponding to the multiple feature fragments are all obtained by segmenting the same reference text.

[0119] Furthermore, the aforementioned multiple text fragments can also be obtained by dividing different reference texts. That is, among the multiple text fragments, different text fragments may belong to the same reference text or different reference texts.

[0120] The first reference text can be provided by the user who provided the question text. For example, when a user asks an academic question, the user can provide some academic papers as reference text for the target model. Similarly, when a user asks a current affairs question, the user can provide some web search results as reference text for the target model. In general, the reference text includes text content related to the question text, and the reference text can be provided in the form of a document or a webpage; this application does not impose specific limitations on this.

[0121] Furthermore, the first reference text can also be retrieved by the execution device based on the question text provided by the user. That is, the user does not provide a corresponding reference text when providing the question text. After obtaining the question text, the execution device can retrieve the first reference text related to the question text from the knowledge base. The knowledge base pre-stores a large number of reference texts for the execution device to retrieve based on the actual question text to be processed. For example, in a question-and-answer scenario in the financial field, the knowledge base stores a large number of user manuals. When the question text provided by the user is related to credit cards, the execution device can retrieve the credit card user manual from the knowledge base as a reference text.

[0122] That is, the first reference text corresponding to the problem text can be provided by the user who provided the problem text, or it can be retrieved by the execution device from the knowledge base based on the problem text. This application does not make any specific limitations on this.

[0123] Optionally, the target model is a large language model, and the multiple feature fragments include a key-value cache (KV Cache) extracted based on the target model. That is, each feature fragment includes a key matrix and a value matrix, and each key matrix includes one or more K vectors (corresponding to one or more tokens), and each value matrix includes one or more V vectors (corresponding to one or more tokens).

[0124] Generally, when the target model is a large language model, the first reference text can be considered as a long sequence of text composed of multiple tokens. For each token in the first reference text, the neural network layer in the target model can extract a K vector and a V vector. Therefore, when the first reference text is divided into multiple text segments, a text segment actually includes one or more tokens, and the features of the reference text can be divided into multiple feature segments, with each feature segment corresponding to a text segment.

[0125] Step 203: Based on the correlation between multiple feature summaries and problem features, determine the target feature fragment among multiple feature fragments. The problem features are obtained by performing feature extraction on the problem text through the target model.

[0126] Understandably, in most cases, not all textual content included in the reference text is highly relevant to the question text. The reference text often contains redundant content with low relevance to the question text, and this redundant content typically fails to provide valuable reference for the target model's response. Therefore, during the target model's processing of the question text, the features corresponding to the redundant content in the reference text with low relevance to the question text often fail to provide effective information for the target model.

[0127] Based on this, this step determines the target feature segment from multiple feature segments based on the relevance between multiple feature summaries and the question feature (e.g., based on the relevance between each feature summary and the question feature). Specifically, the target feature segment may include at least one feature segment from the multiple feature segments. That is, this step may involve selecting only the feature segments most relevant to the question text from multiple feature segments. The number of at least one feature segment selected from the multiple feature segments can be determined based on the number of feature segments, such as selecting 10% or 20% of the feature segments as the first feature segment. Alternatively, the number of at least one selected feature segment can be a pre-set fixed value, such as selecting 5 or 10 feature segments as the first feature segment. This application does not specifically limit the number of feature segments included in the selected first feature segment.

[0128] Furthermore, to improve the efficiency of feature segment selection among multiple feature segments and reduce the computational load on the execution device, this step does not directly calculate the correlation between feature segments and problem features. Instead, it calculates the correlation between the feature summary used to represent the feature segments and the problem features. Since the data volume of the feature summary is much smaller than that of the feature segments, and the feature summary itself is extracted from the feature segments and can represent the semantics expressed by the feature segments themselves, the calculated correlation between the feature summary and the problem features can be used as the correlation between the feature segments and the problem features. In this way, based on the correlation between the feature summary and the problem features, the aforementioned first feature segment can be selected from multiple feature segments.

[0129] The relevance between feature fragments and question features can be understood as the similarity between them. When the target model is a large language model, the relevance can specifically be represented by the attention score. Generally, a higher attention score indicates a higher similarity between the question features and the feature fragment, and thus a higher relevance between the feature fragment and the question text. Therefore, the execution device can consider the target feature fragment corresponding to the feature summary with the highest attention score as the feature fragment with the highest relevance to the question features.

[0130] Specifically, when the target model is a large language model, the question features extracted by the execution device can include a query matrix, and the feature fragments can include a key matrix. The feature summary corresponding to the feature fragment is a smaller matrix extracted based on the key matrix. In one possible example, the attention score between the question features and the feature summary can be obtained by multiplying the query matrix included in the question features with the feature summary. Of course, the attention score between the question features and the feature summary can also be obtained in other ways, as long as the obtained attention score can represent the similarity between the question features and the feature summary. This application does not impose any specific limitations here.

[0131] Step 204: Based on the target feature fragments, process the question text using the target model to obtain the answer text.

[0132] In related technologies, the execution device typically uses reference text and question text as input to the model, and sequentially performs feature extraction on the tokens in the reference text and question text. When extracting features corresponding to subsequent tokens, it is necessary to use the features corresponding to previous tokens.

[0133] In this step, the identified target feature fragments are used as the features corresponding to the reference text. Based on these target feature fragments, the target model is used to extract features from the question text, thus obtaining the features of the question text. Furthermore, the execution device further processes the extracted target feature fragments and question text features using the target model, outputting the tokens of the answer text one by one, ultimately obtaining the answer text. Specifically, the answer text can be the text directly returned to the user in the dialogue. Alternatively, the answer text can be one or more texts to be recommended generated by the target model. The execution device can process one or more texts to be recommended before returning them to the user in the dialogue (e.g., the execution device selects one text to return to the user).

[0134] For example, please refer to Figure 3 , Figure 3This application provides a flowchart illustrating the process of processing problem text using a target model. For example... Figure 3 As shown, the reference text can be divided into N text segments, from text segment 1 to text segment N. Correspondingly, text segment 1 to text segment N correspond to feature segments 1 to feature segments N, respectively. Furthermore, each feature segment from feature segment 1 to feature segment N has a corresponding feature summary; therefore, feature segment 1 to feature segment N correspond to feature summary 1 to feature summary N, respectively. Based on the problem text, the execution device can extract the problem features and determine at least one feature summary from the N feature summaries that has the highest relevance to the problem text (e.g., ...). Figure 3 Feature summaries 2 and 5 in the text. Thus, based on the feature fragments corresponding to at least one determined feature summary (e.g., ... Figure 3 Feature fragments 2 and 5 in the model can be used to process the question text through the target model to obtain the answer text output by the target model.

[0135] Optionally, when the aforementioned multiple text fragments are obtained by performing feature extraction on the first reference text, the execution device can first obtain the beginning feature fragment and the end feature fragment when processing the problem text through the target model. The beginning feature fragment is obtained by performing feature extraction on the first text fragment of the first reference text, and the end feature fragment is obtained by performing feature extraction on the last text fragment of the first reference text. That is, the aforementioned multiple text fragments are actually text fragments located between the first and last text fragments.

[0136] Then, based on the beginning feature fragment, target feature fragment, and end feature fragment, the execution device processes the question text through the target model to obtain the answer text. That is, the execution device may concatenate the beginning feature fragment, target feature fragment, and end feature fragment as the features of the entire first reference text, thereby assisting the target model in processing the question text and obtaining the answer text output by the target model.

[0137] In simple terms, for the first reference text, the execution device first isolates the first text segment (i.e., the opening text segment) and the last text segment (i.e., the closing text segment) from the first reference text, and then selects some feature segments from the feature segments corresponding to the multiple text segments in between. Finally, the execution device combines the feature segments corresponding to the opening and closing text segments with the selected feature segments to form the features of the first reference text, ensuring that the feature segments input to the target model necessarily include the feature segments corresponding to the opening and closing text segments of the first reference text.

[0138] Optionally, the aforementioned multiple feature fragments and their corresponding feature summaries are pre-stored on a storage medium. The execution device then reads the beginning and end feature fragments from this storage medium.

[0139] In this scheme, a storage medium is used to pre-store the features of the reference text. This allows the required features to be read directly from the storage medium when specific features of the reference text are needed, without having to recalculate the features of the reference text. This achieves feature lookup instead of calculation and improves the inference speed of the target model.

[0140] For example, by offloading the KV cache of hot documents to a storage medium for persistence, the KV cache can be reused in subsequent sessions through similarity retrieval, thus avoiding the recalculation of the KV cache and improving the inference speed of the model.

[0141] For example, please refer to Figure 4 , Figure 4 This application provides another flowchart illustrating the processing of problem text using a target model. For example... Figure 4 As shown, the reference text can be divided into N+2 text segments, from text segment 0 to text segment N+1, where text segment 0 is the first text segment and text segment N+1 is the last text segment. By performing feature extraction on each text segment using the target model, the corresponding feature segments for each text segment can be obtained. Specifically, text segment 0 corresponds to feature segment 0, text segments 1 through N correspond to feature segments 1 through N respectively, and text segment N+1 corresponds to feature segment N+1.

[0142] Furthermore, each feature fragment from feature fragment 1 to feature fragment N has a corresponding feature summary; therefore, feature fragment 1 to feature fragment N correspond to feature summary 1 to feature summary N, respectively. Based on the question text, the execution device can extract the question features and determine at least one feature summary from the N feature summaries that has the highest relevance to the question text (e.g., ...). Figure 4 Feature summaries 2 and 5 in the dataset are used to determine the corresponding feature segments (e.g., feature summaries 2 and 5 in the dataset). Figure 4 The initial feature fragment (i.e., feature fragment 0), the determined feature fragments 2 and 5, and the final feature fragment (i.e., feature fragment N+1) are then combined as features of the reference text. These features are then used to process the question text through the target model, thereby obtaining the answer text output by the target model.

[0143] Generally, the opening text segment of a reference text often contains a summary of the entire reference text, while the closing text segment often contains a concluding summary and is the segment closest to the question text. Therefore, both the opening and closing text segments of a reference text are very helpful for the target model to identify key information in the reference text. Thus, in this approach, by isolating the feature segments corresponding to the opening and closing text segments and combining these isolated feature segments with the feature segments corresponding to the selected intermediate text segments to construct the features of the reference text, it is ensured that the feature segments input to the target model necessarily include the feature segments corresponding to the opening and closing text segments of the reference text. This guarantees that important information from the reference text can be input into the target model, effectively ensuring the accuracy of model inference.

[0144] In this application, each feature segment in the multiple feature segments corresponds to a text segment that may include multiple tokens, such as 80 or 100 tokens. This application does not limit the number of tokens included in a text segment. When performing feature extraction on the text segment, each token in the text segment can have its corresponding feature extracted. Therefore, each feature segment corresponding to a text segment may include multiple sub-feature segments corresponding to different tokens, that is, each sub-feature segment corresponds to one token.

[0145] Based on this, when a feature segment includes multiple sub-feature segments, and each sub-feature segment corresponds to a token, there are multiple ways to determine the feature summary corresponding to the feature segment in this application.

[0146] In implementation method 1, the feature summary is obtained by averaging multiple sub-feature segments in the feature segment.

[0147] Specifically, since each sub-feature segment corresponds to a token, a sub-feature segment is actually a feature matrix corresponding to a single token. For multiple sub-feature segments (i.e., multiple feature matrices) corresponding to different tokens within a feature segment, the feature summary is obtained by averaging these multiple feature matrices. In this case, the feature summary is essentially a single feature matrix obtained by averaging multiple feature matrices. That is, a feature segment is a large matrix composed of multiple small matrices corresponding to tokens, while the feature summary is a small matrix corresponding to a single token. For example, if the text segment corresponding to a feature segment includes 80 tokens, the feature segment actually includes matrices corresponding to 80 tokens, while the feature summary can be understood as a matrix corresponding to a single token; therefore, the data size of the feature summary is 1 / 80th of that of the feature segment.

[0148] In implementation method 2, the feature summary is obtained by finding the minimum value of multiple sub-feature segments in the feature segment.

[0149] Similarly, when the sub-feature fragment is the feature matrix corresponding to a single token, a feature matrix can be obtained by finding the minimum value of the feature matrices corresponding to multiple tokens in each channel. This feature matrix obtained by finding the minimum value can then be used as a feature summary, and its size is the same as the size of the feature matrix corresponding to a single token; that is, the data size of the feature summary is the same as the data size of the feature matrix corresponding to a single token.

[0150] In implementation method 3, the feature summary is obtained by taking the maximum value of multiple sub-feature segments in the feature segment.

[0151] Similarly, when the sub-feature fragment is the feature matrix corresponding to a single token, a feature matrix can be obtained by finding the maximum value of the feature matrices corresponding to multiple tokens in each channel. This feature matrix obtained by finding the maximum value can then be used as a feature summary, and its size is the same as the feature matrix corresponding to a single token.

[0152] Furthermore, in some possible implementations, the feature summary can also be obtained by finding the minimum and maximum values ​​of multiple sub-feature segments within a feature fragment. Specifically, after finding the minimum value of multiple sub-feature segments, a feature matrix related to the minimum value can be obtained; after finding the maximum value of multiple sub-feature segments, a feature matrix related to the maximum value can be obtained. Thus, by concatenating the feature matrix related to the minimum value and the feature matrix related to the maximum value, a new feature matrix can be obtained as the feature summary. In this case, the data size of the feature summary is the sum of the data sizes of the feature matrices corresponding to the two tokens.

[0153] In implementation method 4, when multiple sub-feature fragments correspond to a single reference text, the feature summary is the sub-feature fragment with the highest similarity to the last feature fragment among the multiple sub-feature fragments.

[0154] For example, when multiple sub-feature fragments correspond to the first reference text, the execution device can perform text fragment segmentation on the first reference text to obtain a beginning text fragment, multiple middle text fragments, and a ending text fragment. The beginning feature fragment is obtained by performing feature extraction on the first text fragment of the first reference text, and the ending feature fragment is obtained by performing feature extraction on the last text fragment of the first reference text.

[0155] In the conventional processing of the target model, the reference text and the question text are concatenated and input simultaneously. Therefore, the last text segment, being the closest to the question text, often provides more information to the target model. Based on this, this implementation selects the sub-feature segments with the highest similarity to the last feature segment from among multiple sub-feature segments as the feature summary, thus using a subset of sub-feature segments as a representative representation of the entire feature segment. The number of sub-feature segments included in the feature summary is not specifically limited; for example, it can be one or three.

[0156] Optionally, the target model includes multiple neural network layers, such as a first neural network layer and M second neural network layers adjacent to the first neural network layer, where M is a positive integer (e.g., 1, 2, or 3). The aforementioned multiple feature segments are obtained by the first neural network layer performing feature extraction on multiple text segments. Specifically, the aforementioned target feature segment is the first target feature segment. That is, the aforementioned multiple feature segments are merely features extracted by one neural network layer for multiple text segments; other neural network layers in the target model will also extract corresponding features for multiple text segments. Furthermore, when processing problem text using the target model, it is often necessary to base the analysis on the features extracted by all neural network layers in the target model for the text segments.

[0157] Based on this, when processing problem text through the target model, the execution device first determines the second target feature segment from the multiple feature segments obtained by the second neural network layer performing feature extraction on multiple text segments based on the position of the first target feature segment in multiple feature segments;

[0158] Then, based on the first and second target feature fragments, the execution device processes the problem text through the target model. That is, the execution device actually processes the problem text through the target model based on feature fragments extracted from different neural network layers. Specifically, the position of the feature fragments included in the second target feature fragment among the multiple feature fragments extracted by the second neural network layer is the same as the position of the feature fragments included in the first target feature fragment among the multiple feature fragments extracted by the first neural network layer. For example, if the feature fragments included in the first target feature fragment are the 2nd and 5th feature fragments among the multiple feature fragments extracted by the first neural network layer, then the feature fragments included in the second target feature fragment are also the 2nd and 5th feature fragments among the multiple feature fragments extracted by the second neural network layer.

[0159] Here, M can be an integer greater than 1, in which case the execution device can reuse the filtering results on the feature fragments extracted from multiple neural network layers. For example, the execution device filters the feature fragments extracted from the first neural network layer (i.e., the first neural network layer mentioned above) based on the above steps, and then reuses the filtering results to determine the feature fragments to be filtered from the feature fragments extracted from the second and third neural network layers (i.e., the two second neural network layers mentioned above).

[0160] Optionally, M is determined based on the characteristics of the target model. That is, the size of M is related to the target model itself. For example, the size of the neural network layers in the target model and the specific structure of the neural network layers in the target model will affect the actual value of M. In practical applications, a batch of data can be used to test the target model to determine which neural network layers adjacent to the first neural network layer will have the same or similar feature fragment retrieval results as the first neural network layer, thereby determining the value of M.

[0161] Specifically, the applicant's research found that when performing feature segment selection based on steps 201-204 above, the selection results between adjacent neural network layers are often the same or very similar. Therefore, when the target model includes a large number of neural network layers, in order to improve the efficiency of feature segment selection, this application may perform the selection process only on the feature segments extracted by a portion of the neural network layers, and reuse the obtained selection results to select feature segments extracted by other neural network layers, thereby reducing the amount of computation when performing feature segment selection and improving the inference speed of the target model.

[0162] It should be noted that the above only describes how to select a subset of feature fragments from the multiple feature fragments extracted by the first and second neural network layers as input to the target model. In practical applications, the execution device needs to select a subset of feature fragments from the multiple feature fragments extracted by all neural network layers as input to the target model. The specific selection method can refer to the method described above for selecting a subset of feature fragments from the multiple feature fragments extracted by the first and second neural network layers, and will not be repeated here. For example, the execution device can refer to steps 201-204 above to select a subset of feature fragments from the multiple feature fragments extracted by the third neural network layer, and based on the position of the selected subset of feature fragments among the multiple feature fragments, directly determine a subset of feature fragments from the multiple feature fragments extracted by the fourth neural network layer, without needing to select a subset of feature fragments from the multiple feature fragments extracted by the fourth neural network layer again based on steps 201-204 above.

[0163] For example, please refer to Figure 5, Figure 5 This is a schematic diagram illustrating the filtering of feature fragments extracted from different neural network layers, as provided in this application. Figure 5 As shown, each neural network layer in the target model extracts N feature fragments. For the N feature fragments extracted from neural network layer 1, the execution device, based on steps 201-204 above, filters some feature fragments based on feature summaries, thereby obtaining the 2nd and 5th feature fragments. For neural network layer 2 adjacent to neural network layer 1, the execution device does not perform feature fragment filtering based on steps 201-204 above, but directly reuses the filtering results of neural network layer 1, determining the 2nd and 5th feature fragments as the selected feature fragments from the N feature fragments extracted from neural network layer 2. Similarly, for the N feature fragments extracted from neural network layer 3, the execution device, based on steps 201-204 above, filters some feature fragments based on feature summaries, thereby obtaining the 1st and 4th feature fragments. For neural network layer 4 adjacent to neural network layer 3, the execution device directly reuses the filtering results of neural network layer 3, determining the 1st and 4th feature fragments as the selected feature fragments from the N feature fragments extracted from neural network layer 4.

[0164] The above describes how the execution device assists the target model in processing problem text based on the features of a single reference text. In some cases, the execution device may also acquire multiple reference texts, thus requiring the target model to assist in processing problem text based on the features of multiple reference texts.

[0165] For example, the multiple feature fragments are obtained by performing feature extraction on multiple text fragments in the first reference text, and the target feature fragment is the first target feature fragment. Based on the above steps 201-204, the execution device can also obtain multiple feature fragments of the second reference text, which are obtained by performing feature extraction on multiple text fragments in the second reference text.

[0166] Then, based on the degree of correlation between the feature summaries corresponding to multiple feature fragments of the second reference text and the problem features, the execution device determines the second target feature fragment among the multiple feature fragments of the second reference text.

[0167] Finally, based on the splicing features, the execution device processes the problem text through the target model, where the splicing features are obtained based on the first target feature fragment and the second target feature fragment.

[0168] For example, please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating the filtering of feature fragments corresponding to different reference texts, as provided in this application. Figure 6 As shown, for the first reference text, the execution device extracts N+2 feature segments, from feature segment 0 to feature segment N+1, where feature segment 0 is the beginning feature segment and feature segment N+1 is the end feature segment. For the second reference text, the execution device extracts M+2 feature segments, from feature segment 0 to feature segment M+1, where feature segment 0 is the beginning feature segment and feature segment M+1 is the end feature segment. At this point, the execution device selects feature segments 2 and 5 (i.e., the aforementioned first feature segment) from feature segments 1-N corresponding to the first reference text, and feature segments 1 and 4 (i.e., the aforementioned third feature segment) from feature segments 1-M corresponding to the second reference text. Thus, the execution device can concatenate the selected feature segments corresponding to the first and second reference texts to obtain concatenated features, which can then be used to process the problem text using the target model.

[0169] Optionally, the splicing features are obtained based on a first target feature fragment, a first position code, a second target feature fragment, and a second position code. The first position code is used to indicate the position of the first target feature fragment corresponding to the first reference text and the position of the text fragment corresponding to the first target feature fragment in the first reference text. The second position code is used to indicate the position of the second target feature fragment corresponding to the second reference text and the position of the text fragment corresponding to the second target feature fragment in the second reference text.

[0170] In other words, based on the first position encoding and the second position encoding, the target model can perceive which reference text the first target feature fragment and the second target feature fragment of the current input come from, and where the corresponding text fragment is located in the reference text. This enables the target model to process the problem text more effectively based on the input feature fragments, thereby improving the accuracy of the target model.

[0171] The first positional encoding can be, for example, a matrix. It can be superimposed on the first target feature segment by addition, i.e., adding the first target feature segment to the first positional encoding to obtain a feature segment containing positional encoding information. Similarly, the second positional encoding can also be superimposed on the third feature segment by addition. Specifically, the values ​​of the first and second positional encodings can be determined according to the positional encoding method, and this application does not impose specific limitations on this.

[0172] In this scheme, by setting corresponding position codes for feature segments corresponding to different reference texts, and the position codes can simultaneously indicate the reference text corresponding to the feature segment and its corresponding position in the reference text, the target model can effectively distinguish and perceive the input feature segments, which is beneficial to improving the accuracy of the target model.

[0173] Understandably, when the target model is a large language model, when performing feature extraction on the reference text, for any given text segment in the reference text, the target model generates corresponding feature segments based on that text segment and other text segments preceding it. Therefore, for the multiple feature segments corresponding to the first reference text, subsequent feature segments are generated based on the corresponding text segments and the preceding text segments. Consequently, when the execution device selects a subset of feature segments from these multiple feature segments to input into the target model, these selected feature segments actually correspond to only a portion of the reference text. Therefore, when performing feature extraction based on these selected text segments, the resulting feature segments may differ from the currently selected subset.

[0174] Based on this, in order to improve the accuracy of feature fragments and ensure the efficiency of inference, this application may recalculate some key feature fragments in the selected feature fragments, so that cross-attention can be established between feature fragments from different reference texts, making it easier for the target model to understand and process directly reused feature fragments, thereby improving the inference accuracy of the target model.

[0175] For example, after obtaining the first feature fragment and the third feature fragment described above, the execution device can generate splicing features based on the following steps.

[0176] After obtaining the first target feature segment and the second target feature segment, the execution device first determines a first sub-feature segment from the first target feature segment and the second target feature segment. The first sub-feature segment includes sub-feature segments located in the first target feature segment and / or sub-feature segments located in the second target feature segment. Furthermore, the first sub-feature segment is the feature segment in the first target feature segment and the second target feature segment that needs to be recalculated.

[0177] Then, the execution device, based on the first target feature fragment and the second target feature fragment, re-extracts the second sub-feature fragment corresponding to the target sub-text fragment through the target model. The target sub-text fragment is the text fragment corresponding to the first sub-feature fragment. In other words, after determining the first sub-feature fragment that needs to be calculated, the execution device performs the recalculation of the feature fragment based on the first target feature fragment and the second target feature fragment, instead of independently performing the calculation of the feature fragment based on the first reference text and the second reference text respectively.

[0178] Secondly, the execution device updates the first sub-feature fragment to the second sub-feature fragment in the first target feature fragment and the second target feature fragment, so as to obtain the updated first target feature fragment and the updated second target feature fragment;

[0179] Finally, the execution device obtains the spliced ​​features based on the updated first target feature fragment and the updated second target feature fragment. Alternatively, before generating the spliced ​​features, the execution device can add the updated first target feature fragment to the first position code, and add the updated second target feature fragment to the second position code, thereby assigning corresponding position code information to the feature fragments.

[0180] It is understandable that the first and second target feature segments mentioned above are extracted independently based on their respective reference texts. Since the target model calculates the features of subsequent tokens based on the features of all preceding tokens in the input text during the conventional feature extraction process, directly concatenating the features extracted independently from different reference texts (i.e., the first and second target feature segments) can easily lead to a lack of cross-attention between features, making it difficult for the target model to understand directly reused features. Therefore, after determining that the first sub-feature segment needs to be recalculated, the corresponding features of the first sub-feature segment can be recalculated based on both the first and second target feature segments to obtain the second sub-feature segment, thereby achieving the recalculation of some key tokens. In this way, the second sub-feature segment is actually calculated based on all features preceding the second sub-feature segment in both the first and second target feature segments, rather than being calculated based on independent reference texts. That is, the second sub-feature fragment is actually calculated by combining feature fragments from different reference texts. This ensures that the second sub-feature fragment can build cross-attention between feature fragments from different reference texts, so that there will be no cross-attention loss between the updated first target feature fragment and the updated second target feature fragment. This makes it easier for the target model to understand the reused features and improves the accuracy of the target model.

[0181] In this application, the execution device can determine the first sub-feature fragment that needs to be recalculated in several ways. For example, the execution device can randomly determine a portion of the feature fragments (e.g., 10% of the feature fragments) from the first target feature fragment and the second target feature fragment as the first sub-feature fragment. As another example, the execution device can calculate attention scores between the feature fragments corresponding to each token in the first target feature fragment and the end feature fragment of the first reference text, thereby selecting the feature fragments with the highest attention scores as the first sub-feature fragment; and it can also calculate attention scores between the feature fragments corresponding to each token in the second target feature fragment and the end feature fragment of the second reference text, thereby selecting the feature fragments with the highest attention scores as the first sub-feature fragment.

[0182] Optionally, for the aforementioned first reference text and second reference text, the execution device may perform a search among multiple pre-stored texts based on the problem text to obtain the first reference text and second reference text that are more relevant to the problem text.

[0183] For example, after obtaining the problem text, the execution device obtains a text summary of a plurality of pre-stored reference texts, which is used to indicate the summary content of the reference texts.

[0184] Then, the execution device determines the similarity between the text summaries of multiple reference texts and the question text. For example, the execution device calculates the vector representations of the text summaries and the question text using a neural network model, and then determines the similarity between the text summaries and the question text by calculating the distance between the vector representations of the text summaries and the question text.

[0185] In this way, the execution device can identify the first and second reference texts among multiple reference texts that have the highest similarity between the text summary and the question text.

[0186] In this scheme, by pre-storing text summaries of multiple reference texts and calculating the similarity between the text summaries of each reference text and the problem text, one or more reference texts that are most relevant to the problem text can be selected from multiple reference texts to assist the target model in processing the problem text, which is beneficial to improving the accuracy of the target model in processing the problem text.

[0187] To facilitate understanding, the reasoning acceleration method provided in this application will be explained in detail below with specific examples.

[0188] Please see Figure 7 , Figure 7 This application provides a flowchart illustrating the process of processing user-provided question text in a question-and-answer scenario. For example... Figure 7As shown, in a question-and-answer scenario, the process of processing the user-provided question text includes the following steps 701-7012. Steps 701-704 are pre-executed steps used to extract relevant feature fragments from the reference text in the knowledge base and to build retrieval-friendly indexes for these feature fragments. Steps 705-7012 are the steps executed normally in the question-and-answer scenario, used to output the corresponding answer text based on the target model for the user-provided question text.

[0189] Step 701: Perform feature extraction on the reference texts in the knowledge base using the target model to obtain the KV Cache for each reference text.

[0190] Specifically, the knowledge base stores a large amount of pre-collected reference texts, which provide information for the target model when answering questions. For example, in a question-and-answer scenario in the financial field, the knowledge base stores a large number of user manuals, which provide reference information within the financial domain. Similarly, in a question-and-answer scenario in the technology field, the knowledge base stores a large number of scientific papers, which document various technical solutions within the technology sector.

[0191] For each reference text in the knowledge base, the reference text can be input into the target model, which can then extract the corresponding key-value cache for each reference text. Specifically, the target model can be a large language model.

[0192] Step 702: Segment the KV Cache of each reference text to obtain the beginning feature segment, multiple middle feature segments, and the end feature segment.

[0193] Specifically, the KV Cache for each reference text is composed of a matrix combination corresponding to each token in the reference text. Therefore, the execution device can segment the KV Cache for each reference text from the perspective of tokens. For example, the execution device can segment the KV Cache for each reference text into an initial feature segment, multiple middle feature segments, and an ending feature segment, with 80 tokens as the segmentation granularity. Each feature segment corresponds to 80 tokens.

[0194] Step 703: Extract feature summaries for each intermediate feature segment to construct an index for the feature segments.

[0195] In addition to the beginning and end feature segments, each reference text also has multiple feature segments. Therefore, the execution device can extract a smaller feature summary for each feature segment corresponding to the reference text, which facilitates the subsequent construction of the feature segment index. The method by which the execution device extracts the feature summary for each feature segment can refer to Implementation Methods 1-4 above, and will not be repeated here.

[0196] Step 704: Construct an index that facilitates retrieval based on feature summaries.

[0197] Although smaller feature summaries have been extracted for each feature fragment, calculating and comparing the similarity of each feature summary with the question text individually during the feature retrieval stage would still incur significant computational overhead. Therefore, to improve the efficiency of feature retrieval, the execution device can build an index for each feature summary based on the similarity between them. This allows feature summaries with high similarity to be clustered together, facilitating the rapid retrieval of feature summaries with high similarity to the question text.

[0198] For example, please refer to Figure 8 and Figure 9 , Figure 8 A schematic diagram illustrating the processing flow of a problem text provided in this application; Figure 9 This application provides a flowchart illustrating the process of determining the features of reference text for an input model based on question text. For example... Figure 8 and Figure 9 As shown, steps 702-704 above can be performed by a differential representation index building module deployed in the execution device. For each KV cache of reference text in the knowledge base, this differential representation index building module can segment the KV cache, extract feature summaries from the segmented feature fragments, and build an easily searchable index for each feature summary.

[0199] Step 705: Obtain the question text provided by the user.

[0200] Step 706: Determine whether the text answering the current question depends on reference text in the knowledge base.

[0201] Understandably, in some cases, the question text provided by the user may be general phrases, such as "Hello" or "What's the date today?". In such cases, answering the user's question text does not need to rely on reference text in the knowledge base. Therefore, in this step, to improve the efficiency of answering question texts, after obtaining the user's question text, it can be pre-determined whether the answer depends on reference text in the knowledge base. For example, the execution device can input the question text into a pre-trained natural language processing model with a small number of parameters, and the natural language processing model can output whether the answer depends on reference text in the knowledge base.

[0202] Step 707: Use the target model to directly process the problem text.

[0203] If the answer to the current question text does not depend on the reference text in the knowledge base, the execution device will directly input the question text into the target model to obtain the answer text output by the target model.

[0204] Step 708: Retrieve at least one relevant reference text from the knowledge base.

[0205] If answering the current question text requires relying on reference texts in a knowledge base, the execution device retrieves at least one relevant reference text from the knowledge base for that question text. Specifically, the execution device may obtain text summaries of each reference text and calculate the similarity between the text summaries of each reference text and the question text, thereby selecting one or more reference texts that are most relevant to the question text from among multiple reference texts.

[0206] Step 709: Perform selective retrieval based on interlayer awareness for the feature segments of the reference text.

[0207] For example, such as Figure 8As shown, step 709 can be performed by a hierarchical adaptive retrieval module deployed in the execution device, thereby retrieving some feature fragments for input to the target model from multiple feature fragments of the reference text. Specifically, the hierarchical adaptive retrieval module independently performs feature fragment retrieval for different reference texts (i.e., hierarchical retrieval), thus retrieving some feature fragments for input to the target model from multiple feature fragments corresponding to each reference text. Furthermore, for feature fragments extracted from different neural network layers of the same reference text, the hierarchical adaptive retrieval module also independently performs retrieval (i.e., hierarchical retrieval), thus retrieving some feature fragments for input to the target model from multiple feature fragments extracted by each neural network layer. Moreover, for feature fragments extracted from different neural network layers, the retrieval results of adjacent neural network layers can be reused (i.e., retrieval based on inter-layer awareness), thereby reducing the resource and time overhead of retrieval.

[0208] Specifically, the process of the hierarchical adaptive retrieval module in retrieving feature fragments can be referred to the process of filtering feature fragments based on feature summaries described above, and will not be repeated here.

[0209] Step 7010 involves concatenating the beginning and end feature segments of the retrieved feature segments and performing position encoding on the feature segments.

[0210] like Figure 9 As shown, after the execution device retrieves the corresponding partial feature fragments for each reference text, it concatenates the corresponding beginning and end feature fragments into these partial feature fragments, thus using them as features of the reference text for the target model to be output. Furthermore, after concatenating the feature fragments, the execution device can also perform positional encoding on the feature fragments, enabling the feature fragments to simultaneously provide positional information (i.e., the reference text to which the feature fragment belongs and the position of the corresponding text fragment within the reference text).

[0211] For example, please refer to Figure 10 , Figure 10 This application provides a flowchart illustrating a process for determining feature segments of an input target model. For example... Figure 10As shown, for reference texts 1-3, the execution device retrieves partial feature segments from multiple feature segments in the middle of each reference text, and concatenates these partial feature segments with the beginning and end feature segments. Then, the execution device further concatenates the concatenated feature segments corresponding to each reference text according to the chronological order of the reference texts, thus ensuring that the input to the target model includes concatenated feature segments from multiple reference texts. Furthermore, the execution device performs positional encoding optimization on each feature segment, for example, by adding each feature segment to its corresponding positional encoding, enabling the feature segment to simultaneously carry its own positional information.

[0212] Step 7011: Perform key segment identification and recalculation for the feature segments.

[0213] After concatenating feature fragments from multiple reference texts, the execution device can, based on... Figure 8 The key segment selective recalculation module, as shown, identifies key segments in the feature segments of each reference text and recalculates the key segments based on the text segments corresponding to these feature segments, thereby updating the key segments. Specifically, the process of the execution device identifying and recalculating key segments can refer to the process of determining the first sub-feature segment and recalculating the second sub-feature segment described above, and will not be repeated here.

[0214] Step 7012: Based on the feature fragments, process the question text using the target model to obtain the answer text.

[0215] It should be noted that when inputting feature fragments into the target model and processing the problem text based on these fragments, a mask can be set for the feature fragments. This ensures that the target model, when processing the feature fragments corresponding to each reference text, only focuses on the feature fragments preceding the current reference text and the feature fragment to be processed, ignoring the feature fragments of other reference texts. For example... Figure 10 As shown, by setting a mask for the feature fragments of multiple reference texts, the target model can only see the current feature fragment and the previous feature fragments within the same reference text when processing the feature fragments of any reference text, and cannot see the feature fragments of other reference texts. This constrains the target model to focus only on each reference text itself and avoids mutual influence between reference texts.

[0216] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0217] Please see Figure 11 , Figure 11 This is a schematic diagram of a reasoning acceleration device provided in this application. Figure 11As shown, the inference acceleration device includes: an acquisition module 1101, used to acquire question text, which indicates a question to be answered; the acquisition module 1101 is also used to acquire multiple feature summaries corresponding to multiple feature fragments, which are obtained by performing feature extraction on multiple text fragments using a target model, wherein a first feature summary corresponding to a first feature fragment is obtained by performing feature extraction on the first feature fragment, and the data volume of the first feature summary is less than the data volume of the first feature fragment; a processing module 1102, used to determine a target feature fragment among the multiple feature fragments based on the correlation between the multiple feature summaries and the question features, wherein the question features are obtained by performing feature extraction on the question text using a target model; and the processing module 1102 is also used to process the question text using a target model based on the target feature fragment to obtain the answer text.

[0218] In one possible implementation, the multiple text fragments are obtained by segmenting a first reference text, which is the text used for reference when answering the question indicated by the question text.

[0219] In one possible implementation, multiple feature fragments and multiple feature summaries corresponding to the multiple feature fragments are pre-stored on a storage medium; the acquisition module 1101 is further configured to read multiple feature summaries from the storage medium; the processing module 1102 is further configured to determine a target feature fragment among the multiple feature fragments based on the degree of correlation between the multiple feature summaries and the problem features, and read the target feature fragment from the storage medium.

[0220] In one possible implementation, the acquisition module 1101 is further configured to acquire the beginning feature fragment and the end feature fragment, wherein the beginning feature fragment is obtained by performing feature extraction on the first text fragment of the first reference text, and the end feature fragment is obtained by performing feature extraction on the last text fragment of the first reference text; the processing module 1102 is further configured to process the question text through the target model based on the beginning feature fragment, the target feature fragment, and the end feature fragment to obtain the answer text.

[0221] In one possible implementation, the first feature fragment includes multiple sub-feature fragments corresponding to different tokens, and the first feature summary is a sub-feature fragment selected from multiple sub-feature fragments based on the similarity between the multiple sub-feature fragments and the final feature fragment.

[0222] In one possible implementation, the first feature fragment includes multiple sub-feature fragments corresponding to different tokens, and the first feature summary is obtained by taking the mean, maximum, and / or minimum values ​​of the multiple sub-feature fragments.

[0223] In one possible implementation, the target model includes a first neural network layer and M second neural network layers adjacent to the first neural network layer, where M is a positive integer. Multiple feature segments are obtained by the first neural network layer performing feature extraction on multiple text segments, and the target feature segment is the first target feature segment. The processing module 1102 is further configured to: determine the second target feature segment among the multiple feature segments obtained by the second neural network layer performing feature extraction on multiple text segments based on the position of the first target feature segment among the multiple feature segments; and process the problem text through the target model based on the first target feature segment and the second target feature segment.

[0224] In one possible implementation, M is determined based on the characteristics of the target model.

[0225] In one possible implementation, the multiple feature fragments are obtained by performing feature extraction on multiple text fragments in the first reference text, and the target feature fragment is the first target feature fragment; the acquisition module 1101 is also used to acquire multiple feature fragments of the second reference text, which are obtained by performing feature extraction on multiple text fragments in the second reference text; the processing module 1102 is also used to determine the second target feature fragment among the multiple feature fragments of the second reference text based on the correlation between the feature summary corresponding to the multiple feature fragments of the second reference text and the question feature; the processing module 1102 is also used to process the question text through the target model based on the spliced ​​feature, which is obtained based on the first target feature fragment and the second target feature fragment.

[0226] In one possible implementation, the processing module 1102 is further configured to: determine a first sub-feature segment in the first target feature segment and the second target feature segment, the first sub-feature segment including sub-feature segments located in the first target feature segment and / or sub-feature segments located in the second target feature segment; re-extract a second sub-feature segment using the target model based on the first target feature segment and the second target feature segment; update the first sub-feature segment to the second sub-feature segment in the first target feature segment and the second target feature segment to obtain an updated first target feature segment and an updated second target feature segment; and obtain a spliced ​​feature based on the updated first feature segment and the updated third feature segment.

[0227] In one possible implementation, the acquisition module 1101 is further configured to acquire text summaries of multiple pre-stored reference texts; the processing module 1102 is further configured to determine the similarity between the text summaries of the multiple reference texts and the question text; the processing module 1102 is further configured to determine a first reference text and a second reference text among the multiple reference texts based on the similarity between the text summaries of the multiple reference texts and the question text.

[0228] In one possible implementation, multiple feature fragments and multiple feature summaries corresponding to the multiple feature fragments are pre-stored on a storage medium, including a persistent storage medium.

[0229] Both the acquisition module 1101 and the processing module 1102 can be implemented in software or in hardware. For example, the implementation of the processing module 1102 will be described below. Similarly, the implementation of the acquisition module 1101 can be referenced to that of the processing module 1102.

[0230] As an example of a software functional unit, processing module 1102 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 1102 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0231] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0232] As an example of a hardware functional unit, the processing module 1102 may include at least one computing device, such as a server. Alternatively, the processing module 1102 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0233] The processing module 1102 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1102 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1102 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0234] Please see Figure 12 , Figure 12 A schematic diagram of the structure of a computing device provided in this application. Figure 12 The computing device 1200 shown can be used to execute the inference acceleration method provided in this embodiment. For example... Figure 12 As shown, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0235] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1202 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1202 may include a path for transmitting information between various components of the computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0236] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0237] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0238] The memory 1206 stores executable program code, and the processor 1204 executes this executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the aforementioned inference acceleration method. That is, the memory 1206 stores instructions for executing the inference acceleration method.

[0239] The communication interface 1208 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0240] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0241] Please see Figure 13 , Figure 13 This is a schematic diagram of a computing device cluster provided in this application. Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same instructions for executing task processing methods.

[0242] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store a portion of the instructions for executing the inference acceleration method. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions for executing the inference acceleration method.

[0243] It should be noted that the memory 1206 in different computing devices 1200 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 1206 of different computing devices 1200 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0244] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 14 One possible implementation method is shown. Figure 14 This is a schematic diagram of another computing device cluster structure provided in this application. Figure 14 As shown, in a computing device cluster, two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1206 in computing device 1200A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 1206 in computing device 1200B stores instructions for executing the functions of the processing module.

[0245] It should be understood that Figure 14 The functions of the computing device 1200A shown can also be performed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be performed by multiple computing devices 1200.

[0246] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit can be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit can be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0247] Please refer to Figure 15 , Figure 15 This is a schematic diagram of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the above-described... Figure 2 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0248] Figure 15 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0249] In one embodiment, the computer-readable storage medium 1500 is provided using a signal bearer medium 1501. The signal bearer medium 1501 may include one or more program instructions 1502, which, when executed by one or more processors, can provide the above-mentioned... Figure 2 The described function or part of the function.

[0250] In some examples, the signal carrying medium 1501 may include a computer-readable medium 1503, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0251] In some embodiments, the signal-bearing medium 1501 may comprise a computer-recordable medium 1504, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1501 may comprise a communication medium 1505, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1501 may be transmitted by a wireless communication medium 1505 (e.g., a wireless communication medium conforming to the IEEE 1202.X standard or other transmission protocols).

[0252] One or more program instructions 1502 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1502 conveyed to the computing device via a computer-readable medium 1503, a computer-recordable medium 1504, and / or a communication medium 1505.

[0253] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0254] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0255] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0256] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for accelerating inference, characterized in that, include: Obtain the question text, which indicates the question to be answered; Multiple feature summaries corresponding to multiple feature segments are obtained. The multiple feature segments are obtained by performing feature extraction on multiple text segments by a target model. The first feature summary corresponding to the first feature segment in the multiple feature segments is obtained by performing feature extraction on the first feature segment, and the data volume of the first feature summary is less than the data volume of the first feature segment. Based on the correlation between the multiple feature summaries and the question features, a target feature segment is determined from the multiple feature segments, wherein the question features are obtained by performing feature extraction on the question text using the target model. Based on the target feature fragments, the question text is processed by the target model to obtain the answer text.

2. The method according to claim 1, characterized in that, The plurality of text fragments are obtained by dividing a first reference text, which is a text used for reference when answering the question indicated by the question text.

3. The method according to claim 1 or 2, characterized in that, The plurality of feature fragments and the plurality of feature summaries corresponding to the plurality of feature fragments are pre-stored on a storage medium; The step of obtaining multiple feature summaries corresponding to multiple feature fragments includes: Read the plurality of feature summaries from the storage medium; The step of determining the target feature segment from the multiple feature segments based on the correlation between the multiple feature summaries and the problem features includes: Based on the degree of correlation between the plurality of feature summaries and the problem features, the target feature fragment is determined from the plurality of feature fragments, and the target feature fragment is read from the storage medium.

4. The method according to claim 2 or 3, characterized in that, The process of processing the problem text based on the target feature fragment and using the target model includes: Obtain the beginning feature segment and the end feature segment, wherein the beginning feature segment is obtained by performing feature extraction on the first text segment of the first reference text, and the end feature segment is obtained by performing feature extraction on the last text segment of the first reference text; Based on the beginning feature fragment, the target feature fragment, and the end feature fragment, the question text is processed by the target model to obtain the answer text.

5. The method according to any one of claims 1-4, characterized in that, The first feature segment includes multiple sub-feature segments corresponding to different tokens, and the first feature summary is a sub-feature segment selected from the multiple sub-feature segments based on the similarity between the multiple sub-feature segments and the final feature segment.

6. The method according to any one of claims 1-4, characterized in that, The first feature segment includes multiple sub-feature segments corresponding to different tokens, and the first feature summary is obtained by calculating the mean, maximum and / or minimum values ​​of the multiple sub-feature segments.

7. The method according to any one of claims 1-6, characterized in that, The target model includes a first neural network layer and M second neural network layers adjacent to the first neural network layer, where M is a positive integer. The multiple feature segments are obtained by the first neural network layer performing feature extraction on the multiple text segments, and the target feature segment is the first target feature segment. The process of processing the problem text based on the target feature fragment and using the target model includes: Based on the position of the first target feature fragment among multiple feature fragments, the second target feature fragment is determined from multiple feature fragments obtained by the second neural network layer performing feature extraction on the multiple text fragments; The problem text is processed by the target model based on the first target feature fragment and the second target feature fragment.

8. The method according to claim 7, characterized in that, M is determined based on the characteristics of the target model.

9. The method according to any one of claims 1-8, characterized in that, The plurality of feature segments are obtained by performing feature extraction on the plurality of text segments in the first reference text, and the target feature segment is the first target feature segment. The method further includes: Multiple feature fragments of the second reference text are obtained, wherein the multiple feature fragments of the second reference text are obtained by performing feature extraction on multiple text fragments in the second reference text; Based on the correlation between the feature summaries corresponding to multiple feature fragments of the second reference text and the problem features, a second target feature fragment is determined among the multiple feature fragments of the second reference text. The step of processing the problem text based on the first feature fragment using the target model includes: The problem text is processed by the target model based on the splicing features, which are obtained based on the first target feature fragment and the second target feature fragment.

10. The method according to claim 9, characterized in that, The method further includes: A first sub-feature segment is determined in the first target feature segment and the second target feature segment, wherein the first sub-feature segment includes a sub-feature segment located in the first target feature segment and / or a sub-feature segment located in the second target feature segment; Based on the first target feature fragment and the second target feature fragment, the second sub-feature fragment is re-extracted using the target model; In the first target feature segment and the second target feature segment, the first sub-feature segment is updated to the second sub-feature segment to obtain the updated first target feature segment and the updated second target feature segment; The spliced ​​features are obtained based on the updated first feature fragment and the updated third feature fragment.

11. The method according to claim 9 or 10, characterized in that, The method further includes: Obtain a text summary of multiple pre-stored reference texts; Determine the similarity between the text summaries of the plurality of reference texts and the question text; Based on the similarity between the text summaries of the plurality of reference texts and the question text, the first reference text and the second reference text are determined from the plurality of reference texts.

12. The method according to any one of claims 1-11, characterized in that, The plurality of feature fragments and the plurality of feature summaries corresponding to the plurality of feature fragments are pre-stored on a storage medium, the storage medium including a persistent storage medium.

13. A reasoning acceleration device, characterized in that, include: The acquisition module is used to acquire the question text, which indicates the question to be answered; The acquisition module is further configured to acquire multiple feature summaries corresponding to multiple feature segments. The multiple feature segments are obtained by performing feature extraction on multiple text segments by a target model. The first feature summary corresponding to the first feature segment in the multiple feature segments is obtained by performing feature extraction on the first feature segment, and the data volume of the first feature summary is less than the data volume of the first feature segment. The processing module is used to determine a target feature fragment among the multiple feature fragments based on the degree of correlation between the multiple feature summaries and the question features, wherein the question features are obtained by performing feature extraction on the question text through the target model; The processing module is further configured to process the question text based on the target feature fragments using the target model to obtain the answer text.

14. The apparatus according to claim 13, characterized in that, The plurality of text fragments are obtained by dividing a first reference text, which is a text used for reference when answering the question indicated by the question text.

15. The apparatus according to claim 13 or 14, characterized in that, The plurality of feature fragments and the plurality of feature summaries corresponding to the plurality of feature fragments are pre-stored on a storage medium; The acquisition module is further configured to read the plurality of feature summaries from the storage medium; The processing module is further configured to determine the target feature fragment among the plurality of feature fragments based on the degree of correlation between the plurality of feature summaries and the problem features, and to read the target feature fragment from the storage medium.

16. The apparatus according to claim 14 or 15, characterized in that, The acquisition module is further configured to acquire a beginning feature segment and an end feature segment, wherein the beginning feature segment is obtained by performing feature extraction on the first text segment of the first reference text, and the end feature segment is obtained by performing feature extraction on the last text segment of the first reference text; The processing module is further configured to process the question text using the target model based on the beginning feature fragment, the target feature fragment, and the end feature fragment to obtain the answer text.

17. The apparatus according to any one of claims 13-16, characterized in that, The first feature fragment includes multiple sub-feature fragments corresponding to different tokens, and the first feature summary is a sub-feature fragment selected from the multiple sub-feature fragments based on the similarity between the multiple sub-feature fragments and the last feature fragment.

18. The apparatus according to any one of claims 13-16, characterized in that, The first feature segment includes multiple sub-feature segments corresponding to different tokens, and the first feature summary is obtained by calculating the mean, maximum and / or minimum values ​​of the multiple sub-feature segments.

19. The apparatus according to any one of claims 13-18, characterized in that, The target model includes a first neural network layer and M second neural network layers adjacent to the first neural network layer, where M is a positive integer. The multiple feature segments are obtained by the first neural network layer performing feature extraction on the multiple text segments, and the target feature segment is the first target feature segment. The processing module is further configured to: Based on the position of the first target feature fragment among multiple feature fragments, the second target feature fragment is determined from multiple feature fragments obtained by the second neural network layer performing feature extraction on the multiple text fragments; The problem text is processed by the target model based on the first target feature fragment and the second target feature fragment.

20. The apparatus according to claim 19, characterized in that, M is determined based on the characteristics of the target model.

21. The apparatus according to any one of claims 13-20, characterized in that, The plurality of feature segments are obtained by performing feature extraction on the plurality of text segments in the first reference text, and the target feature segment is the first target feature segment; The acquisition module is further configured to acquire multiple feature fragments of the second reference text, wherein the multiple feature fragments of the second reference text are obtained by performing feature extraction on multiple text fragments in the second reference text; The processing module is further configured to determine a second target feature segment among the multiple feature segments of the second reference text based on the degree of correlation between the feature summaries corresponding to the multiple feature segments of the second reference text and the problem features; The processing module is further configured to process the problem text based on the splicing features through the target model, wherein the splicing features are obtained based on the first target feature fragment and the second target feature fragment.

22. The apparatus according to claim 21, characterized in that, The processing module is further configured to: A first sub-feature segment is determined in the first target feature segment and the second target feature segment, wherein the first sub-feature segment includes a sub-feature segment located in the first target feature segment and / or a sub-feature segment located in the second target feature segment; Based on the first target feature fragment and the second target feature fragment, the second sub-feature fragment is re-extracted using the target model; In the first target feature segment and the second target feature segment, the first sub-feature segment is updated to the second sub-feature segment to obtain the updated first target feature segment and the updated second target feature segment; The spliced ​​features are obtained based on the updated first feature fragment and the updated third feature fragment.

23. The apparatus according to claim 21 or 22, characterized in that, The acquisition module is also used to acquire text summaries of multiple pre-stored reference texts; The processing module is also used to determine the similarity between the text summaries of the plurality of reference texts and the question text; The processing module is further configured to determine the first reference text and the second reference text among the plurality of reference texts based on the similarity between the text summaries of the plurality of reference texts and the question text.

24. The apparatus according to any one of claims 13-23, characterized in that, The plurality of feature fragments and the plurality of feature summaries corresponding to the plurality of feature fragments are pre-stored on a storage medium, the storage medium including a persistent storage medium.

25. A computing device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the computing device performs the method as described in any one of claims 1 to 12.

26. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 12.

27. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 12.

28. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 12.

Images