Large language model reasoning optimization method and device, terminal equipment and storage medium
By constructing a knowledge document graph and optimizing the GPU memory allocation of large language models through attention mechanisms, the problem of high GPU memory overhead in edge deployment is solved, and efficient reasoning is achieved in scenarios with long contexts and external knowledge retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HONG KONG UNIV OF SCI & TECH (GUANGZHOU)
- Filing Date
- 2026-01-21
- Publication Date
- 2026-04-24
AI Technical Summary
Existing technologies suffer from high memory overhead in the deployment of large pre-trained language models on the edge, resulting in low inference throughput and unstable operation. In particular, they are difficult to focus on key information in scenarios with long context input and external knowledge retrieval.
By constructing a knowledge document graph, semantically relevant target knowledge documents are filtered and divided into semantic blocks. The importance of semantic blocks is calculated by combining graph neural networks and attention mechanisms, and memory resources are dynamically allocated to optimize key-value data management.
It effectively reduces GPU memory overhead, improves inference speed and throughput, ensures stable model operation in resource-constrained environments, and can identify high-value documents relevant to queries.
Smart Images
Figure CN121920533A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing, and in particular to a method, apparatus, terminal device and storage medium for optimizing reasoning in large language models. Background Technology
[0002] With the widespread application of large-scale pre-trained language models in natural language processing tasks, the demand for edge deployment is increasing. However, due to the limited memory and computing power of edge devices, when the inference process needs to combine external knowledge documents and process long context inputs, the model needs to maintain a large amount of key-value data (Key / Value Cache). As the context length increases linearly, the GPU memory occupied by the K / V cache also increases rapidly, further reducing the inference throughput and causing the model to be unable to run stably in resource-constrained environments.
[0003] To reduce the memory overhead of long contextual reasoning, existing techniques typically employ fixed window pruning, heuristic truncation, compressed attention, and sparse attention to retain only the key-value data of a portion of the text tokens. However, these methods generally suffer from the following technical problems: First, most existing methods rely on fixed rules, such as fixed window length, preset sparsity patterns, or uniform compression ratios, without differentiating or weighing the importance of the information content. When introducing external retrieval knowledge, they often indiscriminately concatenate all document content into the context. This not only increases the memory burden but also makes it difficult for the model to focus on key information. Second, although some methods introduce attention distributions to filter tokens, they still operate at the individual token level, resulting in significant computational overhead and making them unsuitable for external knowledge retrieval scenarios.
[0004] Therefore, how to improve the inference speed of large language models while reducing their memory overhead is a technical problem that needs to be solved. Summary of the Invention
[0005] This application provides a method, apparatus, terminal device, and storage medium for optimizing large language model inference, which can solve the problem in the prior art of how to improve the inference speed of large language models while reducing the memory overhead of large language models.
[0006] This application provides a method for optimizing reasoning in large language models in some embodiments, including: The system receives and searches for several target knowledge documents from a preset set of knowledge documents based on a user query, divides each target knowledge document into several first semantic blocks; wherein each first semantic block includes several minimum text units, each minimum text unit corresponds to a key-value data; the key-value data includes a key vector used to match the user query; Based on the average value of the key value data of the several smallest text units, determine the first key value data corresponding to each of the first semantic blocks; The first attention value corresponding to each of the first semantic blocks is calculated by combining the key vectors of the several smallest text units through the attention mechanism. When the large language model performs inference, the memory allocation of each of the first key-value data is allocated according to the first attention value.
[0007] Compared to existing technologies, the above embodiments have the following beneficial effects: Firstly, this application filters semantically relevant target knowledge documents from a preset knowledge document set based on user queries, and further divides them into several first semantic blocks. This prevents external knowledge from directly participating in reasoning as fine-grained tokens (i.e., the smallest text units), but rather manages it as structured semantic units. Subsequently, by aggregating the key-value data of the smallest text units within the first semantic block, first key-value data representing the overall content of the first semantic block is obtained. Then, combined with an attention mechanism, the importance of the first semantic block relative to the user query is calculated, realizing a semantic block-based model reasoning process, effectively reducing the memory overhead during reasoning. Furthermore, this application dynamically allocates key-value memory during reasoning based on the first attention value of the first semantic block, concentrating memory resources on storing semantic blocks that contribute more to result generation, while discarding or reducing the memory usage of low-relevance semantic blocks. This mechanism can significantly reduce the cache size of key-value data without compromising reasoning quality, reduce memory overhead, and improve the throughput and response speed of the model reasoning stage.
[0008] Furthermore, the step of searching for several target knowledge documents from a preset set of knowledge documents based on a user query includes: Construct a first knowledge document graph by using each of the knowledge documents as nodes and the semantic relationships between the knowledge documents as edges. The first knowledge document graph is input into a preset graph neural network model to perform node feature aggregation operations, thereby obtaining the second knowledge document graph; Sequentially calculate the semantic similarity between the user query and each node in the second knowledge document graph, obtain a number of first nodes with the highest semantic similarity, and use the knowledge document corresponding to each first node as the target knowledge document.
[0009] Compared with existing technologies, the above embodiments have the following beneficial effects: By establishing a structured association graph among knowledge documents and propagating semantic features related to user queries through graph neural networks, each document node can obtain a more accurate semantic representation after aggregating neighborhood semantics. Thus, when calculating the semantic similarity between user queries and document nodes, the model not only focuses on the content of the document itself, but also utilizes its contextual information in the knowledge graph structure, thereby more reliably identifying the target knowledge document that is truly related to the query. At the same time, it makes the documents that subsequently enter the inference stage more refined and focused, effectively reducing the visibility burden caused by irrelevant documents, reducing the scale of subsequent semantic block partitioning and key-value caching, and creating preconditions for memory allocation strategies.
[0010] Furthermore, after taking the knowledge document corresponding to each of the first nodes as the target knowledge document, the method further includes: Extract and concatenate the first features of multiple dimensions of each target knowledge document to obtain fused features; wherein, the first features include: structural score features, keyword matching features, time freshness features, source credibility features, and semantic relevance features; Each of the fused features is input into the pre-trained learning ranking model to obtain a comprehensive score for each target knowledge document, and the target knowledge documents with the highest comprehensive scores are retained.
[0011] Compared with existing technologies, the above embodiments have the following beneficial effects: by integrating information such as structure score, keyword matching degree, time freshness, source credibility and semantic relevance in multiple dimensions, the document ranking results can simultaneously reflect semantic value, information timeliness and source reliability. Then, based on the learning ranking model, the comprehensive score of each target knowledge document is evaluated, and the target knowledge documents are retained according to the comprehensive score, so that the final retained documents have higher knowledge effectiveness, further reducing irrelevant or low-quality documents from entering the semantic block partitioning stage, reducing the scale of subsequent key-value cache from the source, and improving the efficiency of video memory allocation.
[0012] Further, the step of calculating the first attention value corresponding to each of the first semantic blocks by combining the key vectors of the several smallest text units through the attention mechanism includes: For each of the minimum text units, the dot product similarity between the key vector of the minimum text unit and the user query is calculated, and the dot product similarity is input into the normalized exponential function to obtain the second attention value of the minimum text unit; For each of the first semantic blocks, the average of the second attention values of each of the smallest text units corresponding to the first semantic block is calculated to obtain the first attention value.
[0013] Compared to existing technologies, the above embodiments have the following advantages: First, by calculating the dot product similarity for each smallest text unit, the fine-grained relevance between content and user queries is reflected at the smallest text unit level. Then, a second attention value is obtained through a normalized exponential function, and the average is taken at the semantic block level to form a first attention value, obtaining an importance index at the semantic block level, thus simultaneously preserving the accuracy of token granularity and the stability of semantic block granularity. Finally, the obtained first attention value can be directly used to guide the allocation of GPU memory at the semantic block level, prioritizing GPU memory resources for semantic blocks that contribute more to inference, thereby achieving structured pruning of the key-value cache and providing a quantifiable basis for GPU memory optimization.
[0014] Furthermore, the formula for calculating the first attention value is as follows: in, For the first semantic block First attention value; For the first semantic block The smallest set of text units; The smallest text unit key vector The second attention value, For user queries; for The transpose of the matrix; It is a normalized exponential function; As a representative and Dimensions.
[0015] Further, the step of allocating memory for each of the first key-value data according to the first attention value includes: From all the first semantic blocks, determine a number of second semantic blocks with the first attention value first, and retain the first key value data of each second semantic block; According to the first attention value of each second semantic block, allocate corresponding video memory for the corresponding first key value data.
[0016] Compared to existing technologies, the above embodiments have the following advantages: By selecting the second semantic block with the highest attention value, it ensures that GPU memory resources are concentrated on high-value content, rather than being evenly distributed or fixedly pruned, thus preventing key semantics from being compressed or discarded by irrelevant content from the source. Secondly, allocating GPU memory according to the proportion of attention values forms a continuous resource allocation mechanism, which not only maintains the complete key-value information of important semantic blocks but also allows for controlled compression of less important semantic blocks, forming a hierarchical caching structure. This method minimizes GPU memory usage while ensuring inference quality, enabling the model to operate efficiently in long-context inference scenarios or external knowledge retrieval scenarios, improving response speed and throughput.
[0017] Furthermore, the large language model includes several Transformer groups, and each Transformer group includes several Transformer layers; when the large language model performs inference, it further includes: whenever the inference process enters the first Transformer layer of each Transformer group, recalculating the first attention value of each first semantic block, so as to reallocate the memory allocation of each first key-value data according to the first attention value.
[0018] Compared to existing technologies, the above embodiments have the following advantages: Traditional pruning schemes typically determine the caching strategy all at once before inference begins, making it impossible to update according to changes in the focus of attention during deep inference, resulting in the premature pruning of some important semantics in the later stages. This application, however, by recalculating attention in key layers of multiple Transformer groups, can capture the redistribution of attention at different inference stages, enabling memory resources to be migrated in real time to more contributing semantic blocks. This not only improves the matching accuracy of memory allocation but also enhances the stability and quality of the model in deep inference.
[0019] Another embodiment of this application also provides a large language model inference optimization device, including: a semantic block partitioning module, a semantic block key-value data determination module, an attention calculation module, and a key-value data memory allocation module; The semantic block segmentation module is used to receive and search for several target knowledge documents from a preset set of knowledge documents according to the user query, segment each target knowledge document, and obtain several first semantic blocks; wherein each first semantic block includes: several minimum text units, each minimum text unit corresponds to a key-value data; the key-value data includes: a key vector used to match the user query; The semantic block key value data determination module is used to determine the first key value data corresponding to each of the first semantic blocks based on the average value of the key value data of the plurality of smallest text units. The attention calculation module is used to calculate the first attention value corresponding to each of the first semantic blocks by combining the key vectors of the several smallest text units through an attention mechanism. The key-value data memory allocation module is used to allocate memory for each of the first key-value data according to the first attention value when the large language model is performing inference.
[0020] Another embodiment of this application also provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps of the large language model inference optimization method of this application.
[0021] Another embodiment of this application also provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of the large language model inference optimization method of this application. Attached Figure Description
[0022] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0023] Figure 1 This is a flowchart illustrating a large language model inference optimization method provided in some embodiments of this application; Figure 2 This is a schematic diagram of the structure of a large language model inference optimization device provided in some embodiments of this application. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.
[0026] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.
[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0028] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0029] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).
[0030] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.
[0031] To reduce the memory overhead of long contextual reasoning, existing techniques typically employ fixed window pruning, heuristic truncation, compressed attention, and sparse attention to retain only the key-value data of a portion of the text tokens. However, these methods generally suffer from the following technical problems: First, most existing methods rely on fixed rules, such as fixed window length, preset sparsity patterns, or uniform compression ratios, without differentiating or weighing the importance of the information content. When introducing external retrieval knowledge, they often indiscriminately concatenate all document content into the context. This not only increases the memory burden but also makes it difficult for the model to focus on key information. Second, although some methods introduce attention distributions to filter tokens, they still operate at the individual token level, resulting in significant computational overhead and making them unsuitable for external knowledge retrieval scenarios.
[0032] Please refer to Figure 1 To address the problem in existing technologies of how to improve the inference speed of large language models while reducing their memory overhead, this application provides an optimization method for large language model inference, comprising steps S101 to S104, specifically: S101: Receive and search for several target knowledge documents from a preset set of knowledge documents according to the user query, divide each target knowledge document into several first semantic blocks; wherein, each first semantic block includes: several minimum text units, each minimum text unit corresponds to a key-value data; the key-value data includes: a key vector used to match the user query.
[0033] Preferably, in some embodiments of this application, the key-value data further includes a value vector for responding to user queries. It is understood that the key-value data (Key / Value, K / V) is data computed using the attention mechanism in a Transformer / self-attention mechanism, obtained by linear transformation of the embedded representation of each smallest text unit.
[0034] Preferably, in some embodiments of this application, the step of dividing each target knowledge document into several first semantic blocks includes: dividing the target knowledge document into several first semantic blocks (i.e., chunks) according to a fixed length, with the smallest text unit as the unit, wherein the smallest text unit in each first semantic block is semantically continuous in order to maintain syntactic and semantic integrity.
[0035] Furthermore, in some embodiments of this application, the step of searching for several target knowledge documents from a preset set of knowledge documents based on a user query includes: Construct a first knowledge document graph by using each of the knowledge documents as nodes and the semantic relationships between the knowledge documents as edges. The first knowledge document graph is input into a preset graph neural network model to perform node feature aggregation operations, thereby obtaining the second knowledge document graph; Sequentially calculate the semantic similarity between the user query and each node in the second knowledge document graph, obtain a number of first nodes with the highest semantic similarity, and use the knowledge document corresponding to each first node as the target knowledge document.
[0036] Preferably, in some embodiments of this application, the construction process of the first knowledge document graph includes: converting existing knowledge documents into knowledge graph nodes through entityification, using semantic relationships such as references, similarities, and co-occurrences between knowledge documents as edges, and combining time decay and confidence to form weighted relationships. The corresponding mathematical modeling of the first knowledge graph is as follows: ,in, A set of nodes; The set of nodes represents a knowledge document, and the edges represent semantic relationships such as references, similarities, and co-occurrences between knowledge documents. Each edge carries a weight, which is determined through confidence and time decay. Further, each node... initial embedding vector Represented as: ,in, ; Representative node Corresponding knowledge documents; For encoders, it can be , Or lightweight This application does not impose any restrictions on the encoder.
[0037] Preferably, in some embodiments of this application, the step of inputting the first knowledge document graph into a preset graph neural network model for node feature aggregation to obtain a second knowledge document graph includes: The constructed first knowledge document graph is input into the graph neural network model, where multiple layers of graph convolution operations are performed to achieve message passing and relationship aggregation. The process of each graph convolution layer operation is as follows: in, Representative node go through Embedded vectors after layer graph convolutional layer operations; The activation function can be either ReLU or GELU; this application does not impose any restrictions on the activation function. for The weight parameter matrix during layer graph convolution; for Bias vector during layer graph convolution; The aggregation function can be used to aggregate features, and can be performed by averaging, weighted summing, or attention weighting. This application does not limit the aggregation function. For nodes The set of neighbors; For nodes The neighboring nodes; Neighboring nodes go through Embedsion vectors after multi-layer graph convolution operations. After multiple layers of graph convolution operations, the embedding vector of each node incorporates neighbor relationships and semantic features.
[0038] Preferably, in some embodiments of this application, calculating the semantic similarity between the user query and each node in the second knowledge document graph includes: First, the embedding vector corresponding to the user query is determined using the following formula: in, Query the corresponding embedding vector for the user; For user queries.
[0039] Furthermore, semantic similarity is calculated using the following formula: in, For semantic similarity; The formula for calculating semantic similarity can be a dot product operation, a cosine similarity operation, or a relevance score output by a multilayer perceptron. This application does not limit the calculation process of semantic similarity.
[0040] Preferably, after calculating the semantic similarity between the user query and each node in the knowledge graph, the knowledge documents can be directly sorted according to the semantic similarity, thereby filtering out the final target knowledge documents.
[0041] This application establishes a structured association graph among knowledge documents and propagates semantic features related to user queries through a graph neural network. This enables each document node to obtain a more accurate semantic representation after aggregating neighborhood semantics. Thus, when calculating the semantic similarity between user queries and document nodes, the model not only focuses on the content of the document itself but also utilizes its contextual information in the knowledge graph structure. This allows for more reliable identification of truly query-related target knowledge documents. At the same time, it makes the documents entering the inference stage more refined and focused, effectively reducing the visibility burden caused by irrelevant documents, reducing the scale of subsequent semantic block partitioning and key-value caching, and creating preconditions for memory allocation strategies.
[0042] Furthermore, in some embodiments of this application, after taking the knowledge document corresponding to each of the first nodes as the target knowledge document, it further includes: Extract and concatenate the first features of multiple dimensions of each target knowledge document to obtain fused features; wherein, the first features include: structural score features, keyword matching features, time freshness features, source credibility features, and semantic relevance features; Each of the fused features is input into the pre-trained learning ranking model to obtain a comprehensive score for each target knowledge document, and the target knowledge documents with the highest comprehensive scores are retained.
[0043] Preferably, in some embodiments of this application, after initially screening target knowledge documents based on semantic similarity, the target knowledge documents are further screened based on the extracted first features in multiple dimensions.
[0044] Preferably, in some embodiments of this application, the structural score feature is obtained based on the structural information of the second knowledge graph, such as the degree of the node corresponding to the target knowledge document or the shortest path and path confidence with the query node. This is used to measure the importance of the document in the graph and its structural relationship with the query.
[0045] Preferably, in some embodiments of this application, the keyword matching features are determined by statistically analyzing the degree of matching between the target knowledge document and the user query at the keyword level, such as using TF-IDF, BM25, or weighted keyword matching. Emphasis is placed on the direct coverage of the literal meaning of the user query.
[0046] Preferably, in some embodiments of this application, the time freshness feature is determined by assigning a value based on the publication time decay function of the target knowledge document, specifically as follows: in, For target knowledge documents Time freshness characteristics; and These are the current time and the publication time of the target knowledge document, respectively. This represents the time decay coefficient. Newer documents receive higher scores. This feature is suitable for retrieval scenarios involving news, academic papers, and documents with high timeliness requirements.
[0047] Preferably, in some embodiments of this application, the source credibility feature is determined by measuring the source quality of the target knowledge document (e.g., journal impact factor, website credibility, authoritative institution identification, etc.), specifically by using a predefined credibility rating or training a classifier to judge credibility.
[0048] Preferably, in some embodiments of this application, the semantic relevance feature adopts the semantic similarity between the target indicator document and the user query.
[0049] Preferably, in some embodiments of this application, the learning ranking model may employ a linear / logistic regression model, a gradient boosting tree model (such as LambdaMART and RankNet), or a Transformer-based ranking model (such as BERT Ranker, MonoT5, etc.).
[0050] Preferably, in some embodiments of this application, when the target knowledge documents to be finally selected are determined, the selection can be based solely on semantic similarity. Alternatively, after preliminary selection based on semantic similarity, the final selection can be made by extracting multidimensional features and relying on a learning ranking model.
[0051] This application integrates information such as structure score, keyword matching degree, time freshness, source credibility, and semantic relevance from multiple dimensions, so that the document ranking results can simultaneously reflect semantic value, information timeliness, and source reliability. Then, based on the learning ranking model, the comprehensive score of each target knowledge document is evaluated, and the target knowledge documents are retained according to the comprehensive score, so that the final retained documents have higher knowledge effectiveness, further reducing irrelevant or low-quality documents from entering the semantic block partitioning stage, reducing the scale of subsequent key-value cache from the source, and improving the efficiency of memory allocation.
[0052] S102: Determine the first key value data corresponding to each of the first semantic blocks based on the average value of the key value data of the plurality of smallest text units.
[0053] Preferably, in some embodiments of this application, the first key-value data of the first semantic block is calculated and obtained in the following manner: in, Represents the first semantic block The key vector; For the first semantic block The number of the first smallest text units contained within; For the first semantic block The set of the first smallest text units; The smallest text unit The key vector; Represents the first semantic block The value vector; The smallest text unit The value vector.
[0054] S103: Calculate the first attention value corresponding to each of the first semantic blocks by combining the key vectors of the several smallest text units through the attention mechanism.
[0055] Furthermore, in some embodiments of this application, the step of calculating the first attention value corresponding to each of the first semantic blocks by combining the key vectors of the plurality of smallest text units through an attention mechanism includes: For each of the minimum text units, the dot product similarity between the key vector of the minimum text unit and the user query is calculated, and the dot product similarity is input into the normalized exponential function to obtain the second attention value of the minimum text unit; For each of the first semantic blocks, the average of the second attention values of each of the smallest text units corresponding to the first semantic block is calculated to obtain the first attention value.
[0056] Preferably, in some embodiments of this application, after the first attention value of the first semantic block is calculated, the top-K second semantic blocks can be directly selected according to the first attention value, the key-value data of the second semantic block is retained, and the key-value data of the remaining first semantic blocks are discarded or delayed in loading.
[0057] This application first calculates the dot product similarity for each smallest text unit, thus reflecting the fine-grained relevance between content and user queries at the smallest text unit level. It then obtains a second attention value through a normalized exponential function, and averages these values at the semantic block level to form a first attention value, obtaining an importance index at the semantic block level. This approach simultaneously preserves the accuracy of token granularity and the stability of semantic block granularity. The final first attention value can be directly used to guide memory allocation at the semantic block level, prioritizing memory resources for semantic blocks that contribute more to inference. This achieves structured pruning of the key-value cache, providing a quantifiable basis for memory optimization.
[0058] Furthermore, in some embodiments of this application, the formula for calculating the first attention value is specifically as follows: in, For the first semantic block First attention value; For the first semantic block The smallest set of text units; The smallest text unit key vector The second attention value, For user queries; for The transpose of the matrix; It is a normalized exponential function; As a representative and The dimensions are as follows. It can be seen that the higher the first attention value of the first semantic block, the stronger its influence on the global context, and the more worthwhile the corresponding key-value data is to be retained.
[0059] S104: When the large language model performs inference, allocate the memory of each of the first key-value data according to the first attention value.
[0060] Furthermore, in some embodiments of this application, the step of allocating memory for each of the first key-value data according to the first attention value includes: From all the first semantic blocks, determine a number of second semantic blocks with the first attention value first, and retain the first key value data of each second semantic block; According to the first attention value of each second semantic block, allocate corresponding video memory for the corresponding first key value data.
[0061] Preferably, in some embodiments of this application, the step of allocating corresponding video memory to the corresponding first key value data according to the first attention value of each second semantic block includes: firstly determining and reserving the minimum video memory required for each second semantic block, and then allocating the remaining video memory proportionally according to the first attention value of the second semantic block.
[0062] This application ensures that GPU memory resources are concentrated on high-value content by selecting the second semantic block with the highest attention value, rather than being evenly distributed or fixed-priority pruning. This prevents key semantics from being compressed or discarded by irrelevant content from the outset. Secondly, allocating GPU memory proportionally based on attention values creates a continuous resource allocation mechanism. This not only preserves the complete key-value information of important semantic blocks but also allows for controlled compression of less important semantic blocks, forming a hierarchical caching structure. This method minimizes GPU memory usage while maintaining inference quality, enabling the model to operate efficiently in long-context inference scenarios or external knowledge retrieval scenarios, improving response speed and throughput.
[0063] Furthermore, in some embodiments of this application, the large language model includes several Transformer groups, and each Transformer group includes several Transformer layers; when the large language model performs inference, it further includes: whenever the inference process enters the first Transformer layer of each Transformer group, recalculating the first attention value of each first semantic block, so as to reallocate the memory allocation of each first key-value data according to the first attention value.
[0064] Traditional pruning schemes typically determine the caching strategy all at once before inference begins, failing to update it according to changes in focus during deep inference. This leads to the premature pruning of some important semantic blocks in the later stages. This application, however, recalculates attention at key layers across multiple Transformer groups, capturing the redistribution of attention at different inference stages. This allows memory resources to be migrated in real-time to more contributing semantic blocks. This not only improves the matching accuracy of memory allocation but also enhances the stability and quality of the model during deep inference. Furthermore, this method avoids recalculating attention for semantic blocks at every Transformer layer, significantly reducing compression computation overhead and improving inference throughput.
[0065] In summary, the large language model inference optimization method provided in this application has the following advantages compared to existing technologies: First, based on user queries, this application filters semantically relevant target knowledge documents from a preset knowledge document set and further divides them into several first semantic blocks. This prevents external knowledge from directly participating in inference as fine-grained tokens (i.e., the smallest text units), but instead manages it as structured semantic units. Then, by aggregating the key-value data of the smallest text units within the first semantic block, first key-value data representing the overall content of the first semantic block is obtained. Combined with an attention mechanism, the importance of the first semantic block relative to the user query is calculated, realizing a model inference process with semantic blocks as the granularity, effectively reducing the memory overhead during inference. Furthermore, this application dynamically allocates key-value memory during inference based on the first attention value of the first semantic block, concentrating memory resources on storing semantic blocks that contribute more to result generation, while discarding or reducing the memory usage of low-relevance semantic blocks. This mechanism can significantly reduce the cache size of key-value data without compromising inference quality, reduce memory overhead, and improve the throughput and response speed of the model inference stage.
[0066] like Figure 2 As shown, based on the above-described method embodiments, this application provides a large language model inference optimization device, including: a semantic block partitioning module 201, a semantic block key-value data determination module 202, an attention calculation module 203, and a key-value data memory allocation module 204.
[0067] Further, in some embodiments of this application, the semantic block segmentation module 201 is used to receive and search for several target knowledge documents from several preset knowledge documents according to user queries, segment each target knowledge document to obtain several first semantic blocks; wherein, each first semantic block includes: several minimum text units, each minimum text unit corresponds to a key value data; the key value data includes: a key vector for matching the user query; the semantic block key value data determination module 202 is used to determine the first key value data corresponding to each first semantic block according to the mean of the key value data of the several minimum text units; the attention calculation module 203 is used to calculate the first attention value corresponding to each first semantic block through an attention mechanism combined with the key vector of the several minimum text units; the key value data memory allocation module 204 is used to allocate memory allocation for each first key value data according to the first attention value when the large language model performs inference.
[0068] Further, in some embodiments of this application, the semantic block partitioning module 201 includes: a first knowledge document graph construction unit, a graph convolution unit, and a semantic similarity calculation unit; the semantic block partitioning module 201 is used to search for several target knowledge documents from a preset number of knowledge documents according to a user query, including: the first knowledge document graph construction unit is used to construct a first knowledge document graph by taking each knowledge document as a node and the semantic relationship between each knowledge document as an edge; the graph convolution unit is used to input the first knowledge document graph into a preset graph neural network model to perform node feature aggregation operation to obtain a second knowledge document graph; the semantic similarity calculation unit is used to sequentially calculate the semantic similarity between the user query and each node in the second knowledge document graph, obtain several first nodes with the highest semantic similarity, and take the knowledge document corresponding to each first node as the target knowledge document.
[0069] Furthermore, in some embodiments of this application, the semantic block partitioning module 201 further includes: a feature extraction unit and a learning and ranking model invocation unit; the semantic block partitioning module 201, after using the knowledge document corresponding to each first node as the target knowledge document, further includes: the feature extraction unit, used to extract and concatenate multiple dimensions of first features of each target knowledge document to obtain fused features; wherein, the first features include: structural score features, keyword matching features, time freshness features, source credibility features, and semantic relevance features; the learning and ranking model invocation unit is used to input each of the fused features into a pre-trained learning and ranking model to obtain a comprehensive score corresponding to each target knowledge document, and retain a number of target knowledge documents with the highest comprehensive scores.
[0070] Further, in some embodiments of this application, the attention calculation module 203 includes: a first calculation unit and a second calculation unit; the attention calculation module 203 is used to calculate a first attention value corresponding to each of the first semantic blocks by combining the key vectors of the plurality of minimum text units through an attention mechanism, including: the first calculation unit is used to calculate the dot product similarity between the key vector of the minimum text unit and the user query for each of the minimum text units, and input the dot product similarity into a normalized exponential function to obtain a second attention value of the minimum text unit; the second calculation unit is used to calculate the mean of the second attention values of each of the minimum text units corresponding to the first semantic block for each of the first semantic blocks, to obtain the first attention value.
[0071] Furthermore, in some embodiments of this application, the formula for calculating the first attention value is specifically as follows: in, For the first semantic block First attention value; For the first semantic block The smallest set of text units; The smallest text unit key vector The second attention value, For user queries; for The transpose of the matrix; It is a normalized exponential function; As a representative and Dimensions.
[0072] Further, in some embodiments of this application, the key-value data memory allocation module 204 includes: a semantic block filtering unit and a memory allocation unit; the key-value data memory allocation module 204 is used to allocate memory allocation for each of the first key-value data according to the first attention value, including: the semantic block filtering unit is used to determine a plurality of second semantic blocks with the first attention value preceding them from all the first semantic blocks, and retain the first key-value data of each second semantic block; the memory allocation unit is used to allocate corresponding memory for the corresponding first key-value data according to the first attention value of each second semantic block.
[0073] Furthermore, in some embodiments of this application, the large language model includes several Transformer groups, and each Transformer group includes several Transformer layers; when the large language model performs inference, it further includes: whenever the inference process enters the first Transformer layer of each Transformer group, recalculating the first attention value of each first semantic block, so as to reallocate the memory allocation of each first key-value data according to the first attention value.
[0074] It is understood that the above-described device embodiments correspond to the method embodiments of this application, and can implement the large language model inference optimization method provided by any of the above-described method embodiments of this application.
[0075] In summary, the large language model inference optimization device provided in this application has the following advantages compared to the prior art: Firstly, based on user queries, this application filters semantically relevant target knowledge documents from a preset knowledge document set and further divides them into several first semantic blocks. This prevents external knowledge from directly participating in inference as fine-grained tokens (i.e., the smallest text units), but instead manages it as structured semantic units. Subsequently, by aggregating the key-value data of the smallest text units within the first semantic block, first key-value data representing the overall content of the first semantic block is obtained. Then, combined with an attention mechanism, the importance of the first semantic block relative to the user query is calculated, realizing a model inference process with semantic blocks as the granularity, effectively reducing the memory overhead during inference. Furthermore, this application dynamically allocates key-value memory during inference based on the first attention value of the first semantic block, concentrating memory resources on storing semantic blocks that contribute more to result generation, while discarding or reducing the memory usage of low-relevance semantic blocks. This mechanism can significantly reduce the cache size of key-value data without compromising inference quality, reduce memory overhead, and improve the throughput and response speed of the model inference stage.
[0076] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0077] Based on the above embodiments of the large language model inference optimization method, another embodiment of this application provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the large language model inference optimization method of any embodiment of this application.
[0078] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete this application. The one or more module units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.
[0079] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.
[0080] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.
[0081] Based on the above-described method embodiments, another embodiment of this application provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the large language model inference optimization method described in any of the above-described method embodiments of this application.
[0082] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.
Claims
1. A method for optimizing reasoning in a large language model, characterized in that, include: The system receives and searches for several target knowledge documents from a preset set of knowledge documents based on a user query, divides each target knowledge document into several first semantic blocks; wherein each first semantic block includes several minimum text units, each minimum text unit corresponds to a key-value data; the key-value data includes a key vector used to match the user query; Based on the average value of the key value data of the several smallest text units, determine the first key value data corresponding to each of the first semantic blocks; The first attention value corresponding to each of the first semantic blocks is calculated by combining the key vectors of the several smallest text units through the attention mechanism. When the large language model performs inference, the memory allocation of each of the first key-value data is allocated according to the first attention value.
2. The large language model reasoning optimization method as described in claim 1, characterized in that, The step of searching for several target knowledge documents from a preset set of knowledge documents based on a user query includes: Construct a first knowledge document graph by using each of the knowledge documents as nodes and the semantic relationships between the knowledge documents as edges. The first knowledge document graph is input into a preset graph neural network model to perform node feature aggregation operations, thereby obtaining the second knowledge document graph; Sequentially calculate the semantic similarity between the user query and each node in the second knowledge document graph, obtain a number of first nodes with the highest semantic similarity, and use the knowledge document corresponding to each first node as the target knowledge document.
3. The large language model reasoning optimization method as described in claim 2, characterized in that, After taking the knowledge document corresponding to each of the first nodes as the target knowledge document, the method further includes: Extract and concatenate the first features of multiple dimensions of each target knowledge document to obtain fused features; wherein, the first features include: structural score features, keyword matching features, time freshness features, source credibility features, and semantic relevance features; Each of the fused features is input into the pre-trained learning ranking model to obtain a comprehensive score for each target knowledge document, and the target knowledge documents with the highest comprehensive scores are retained.
4. The large language model reasoning optimization method as described in claim 1, characterized in that, The step of calculating the first attention value for each of the first semantic blocks by combining the key vectors of the several smallest text units through an attention mechanism includes: For each of the minimum text units, the dot product similarity between the key vector of the minimum text unit and the user query is calculated, and the dot product similarity is input into the normalized exponential function to obtain the second attention value of the minimum text unit; For each of the first semantic blocks, the average of the second attention values of each of the smallest text units corresponding to the first semantic block is calculated to obtain the first attention value.
5. The large language model inference optimization method as described in claim 4, wherein the formula for calculating the first attention value is as follows: in, For the first semantic block First attention value; For the first semantic block The smallest set of text units; The smallest text unit key vector The second attention value, For user queries; for The transpose of the matrix; It is a normalized exponential function; As a representative and Dimensions.
6. The large language model reasoning optimization method as described in claim 1, characterized in that, The step of allocating memory for each of the first key-value data according to the first attention value includes: From all the first semantic blocks, determine a number of second semantic blocks with the first attention value first, and retain the first key value data of each second semantic block; According to the first attention value of each second semantic block, allocate corresponding video memory for the corresponding first key value data.
7. The large language model inference optimization method as described in any one of claims 1 to 6, characterized in that, The large language model includes several Transformer groups, and each Transformer group includes several Transformer layers; when the large language model performs inference, it further includes: whenever the inference process enters the first Transformer layer of each Transformer group, recalculating the first attention value of each first semantic block, so as to reallocate the memory allocation of each first key-value data according to the first attention value.
8. A large language model reasoning optimization device, characterized in that, include: The module includes a semantic block partitioning module, a semantic block key-value data determination module, an attention calculation module, and a key-value data memory allocation module. The semantic block segmentation module is used to receive and search for several target knowledge documents from a preset set of knowledge documents according to the user query, segment each target knowledge document, and obtain several first semantic blocks; wherein each first semantic block includes: several minimum text units, each minimum text unit corresponds to a key-value data; the key-value data includes: a key vector used to match the user query; The semantic block key value data determination module is used to determine the first key value data corresponding to each of the first semantic blocks based on the average value of the key value data of the plurality of smallest text units. The attention calculation module is used to calculate the first attention value corresponding to each of the first semantic blocks by combining the key vectors of the several smallest text units through an attention mechanism. The key-value data memory allocation module is used to allocate memory for each of the first key-value data according to the first attention value when the large language model is performing inference.
9. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement a large language model inference optimization method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform a large language model inference optimization method as described in any one of claims 1 to 7.