Large language model reasoning method and system based on cascaded sparse attention mechanism and medium

Abstract

The application belongs to the technical field of large language model reasoning acceleration and long context processing, and particularly relates to a large language model reasoning method, system and medium based on a cascaded sparse attention mechanism, which comprises the following steps: dividing the key-value cache of a large language model into fixed-size pages, extracting statistical features in each page to calculate global importance scores, sorting and screening all pages according to the scores to construct a page set, further dividing the page set into fixed-size data blocks, constructing a multi-branch parallel extraction structure to extract a global key block set and a local block set and then merging and deduplicating the two sets to obtain a data block set; performing sparse attention calculation based on the data block set to output a reasoning result; and gradually realizing sparse calculation from a macro page to a micro data block, effectively overcoming the secondary calculation and memory complexity bottleneck of the attention mechanism, and realizing efficient long-context large language model reasoning.

Description

Technical Field

[0001] This invention belongs to the field of large language model reasoning acceleration and long context processing technology, specifically involving large language model reasoning methods, systems and media based on cascaded sparse attention mechanisms. Background Technology

[0002] Long-context large-scale language model inference is of paramount importance in various fields such as multi-turn dialogue systems, long code understanding, and document analysis. Currently, the standard Transformer architecture achieves sequence modeling through a self-attention mechanism, requiring attention to the entire historical key-value (KV) cache during the decoding stage of generating each token. This method uses a dense attention matrix to calculate the dot product of all query vectors and key vectors, thereby achieving the interaction and extraction of global contextual features.

[0003] However, this method has the following problems: The standard self-attention mechanism has time and space complexity. In long context inference, the size of the KV cache grows linearly or even quadratically with the sequence length, resulting in extremely high memory consumption. For example, a 7B model with 32K context requires up to 16GB of memory bandwidth. Memory access bandwidth has become a serious performance bottleneck, resulting in extremely high cost and huge latency for long sequence inference.

[0004] Existing sparse attention or cache eviction methods often rely on fixed sparse patterns, such as sliding windows, or blindly and permanently discard parts of the key-value cache. These methods ignore the dynamic nature of token importance changing with the current query, leading to the loss of crucial contextual information during long-range dependency inference and significantly reducing model accuracy. Furthermore, existing coarse-grained page selection methods lack fine-grained precise control, while purely fine-grained methods face enormous computational overhead, making it difficult to strike a balance between the two. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a large language model reasoning method, system, and medium based on a cascaded sparse attention mechanism. It performs query-aware page selection, multi-branch fine-grained block selection, dynamic fusion, and sparse attention computation, effectively mining the core contextual information in the large language model and achieving a reasoning process that balances extremely high efficiency with lossless accuracy.

[0006] To achieve the above objectives, the first aspect of this invention proposes a large language model reasoning method based on a cascaded sparse attention mechanism, comprising the following steps: S1. Page selection for query awareness: Obtain a long text sequence containing multiple lexical units, input it into a large language model to generate a key-value cache and store it; The key-value cache of the large language model is divided into pages. A global importance score for each page is calculated by extracting statistical features from all key vectors within that page. Based on these global importance scores, all pages are sorted and filtered to construct a page set. ; S2, Multi-branch fine-grained block selection: The page collection The data is divided into blocks, and a multi-branch parallel strategy is used to filter the data blocks: the first branch uses statistical features to calculate the importance score of each data block and filter the global key block set. The second branch selects the contiguous data blocks closest to the current query position to form a local block set. ; S3, Dynamic Fusion and Sparse Attention Computation: The global key block set With the local block set Merging and deduplication are performed to obtain a set of data blocks. The data block set Perform sparse attention computation and output the inference result.

[0007] Furthermore, the key-value cache of the large language model is divided into pages. A global importance score for each page is calculated by extracting statistical features from all key vectors within that page. Based on this global importance score, all pages are sorted and filtered to construct a page set. ,include: S101. Divide the key-value cache of the large language model into pages of fixed size, and extract the maximum, minimum, mean and variance of all key vectors in each feature dimension within each page. S102. Using each component of the given query vector, perform a dot product operation with the statistical features of the corresponding dimension, and extract the maximum value of the product result under the corresponding dimension as the maximum interaction contribution value. S103. Sum the maximum interaction contribution values ​​across all feature dimensions to obtain the global importance score of the page. S104. Sort all pages according to the global importance score and filter the top pages with the highest scores. Each page is the top-K most relevant pages, and a page set is constructed based on these top-K most relevant pages. .

[0008] Furthermore, the first branch uses statistical features to calculate the importance score of each data block and then filters the global key block set. ,include: S201. The first branch is the selection branch, which extracts the maximum, minimum, mean and variance of all key vectors in each data block in each feature dimension. S202. Using each component of the given query vector, perform dot product operations with the block-level statistical features of the corresponding dimension, and extract the maximum value of the product results in the corresponding dimension as the maximum interaction contribution value. S203. Sum the maximum interaction contribution values ​​across all feature dimensions to obtain the block-level importance score; S204, Page Collection The union of data blocks contained in each page is used to obtain a global candidate data block set. The top P data blocks with the highest block-level importance scores are extracted from the global candidate data block set to form a global key data block set. , represented as: ; in, Indicates the current reasoning step Next, select the set of global key data blocks extracted from the selected branch; This indicates the extraction of the top block-level importance scores from the global candidate data block set. One data block operation; The parameter for the preset number of critical data blocks to retain; This represents the block-level importance score of data block b. Represents a collection of pages single page The collection of all data blocks contained therein; This represents the union operation, which is the set of pages. All data blocks from all pages are merged into a unified global candidate block set.

[0009] Furthermore, the second branch is a local window branch, which selects the contiguous data blocks closest to the current query position to form a local block set. , represented as: ; in, Relative to the current query position Last The most recent data block, Indicates the number of consecutive data blocks.

[0010] Furthermore, the global key block set With the local block set Merging and deduplication are performed to obtain a set of data blocks. The data block set Perform sparse attention computation and output the inference result. ,include: S301, the global key block set With the local block set pass The operation performs merging and deduplication to obtain a set of data blocks. ; S302, According to the data block set Through data collection operations, sparse key matrices and sparse value matrices are extracted from the original key-value matrix, represented as follows: ; ; in, Indicates data collection operation, Represents the key matrix, Represents a value matrix, Represents the sparse bond matrix. This represents the coefficient value matrix.

[0011] S303. Limit the number of data blocks retained in each step to a preset range to control the total number of tokens and complete the sparse pruning operation of the key-value cache; S304. Based on the sparse key matrix and the sparse value matrix, perform attention calculation on the pruned key-value cache to obtain the processed key-value cache. , represented as: ; in, Indicates the current step The query feature vector; This represents the transpose of the sparse bond matrix; The size of the feature dimension; This represents the normalized exponential function; This is the corresponding sparse value matrix; Key-value cache The text used for subsequent decoding to generate the current word is then output as the reasoning result.

[0012] A second aspect of this invention proposes a large language model inference system based on a cascaded sparse attention mechanism, comprising: The acquisition module is used to acquire long text sequences containing multiple lexical units, input them into a large language model to generate key-value caches and store them; The query-aware page selection module divides the key-value cache of the large language model into pages. It calculates the global importance score of each page by extracting statistical features of all key vectors within that page, and then sorts and filters all pages based on these global importance scores to construct a page set. ; A multi-branch fine-grained block selection module is used to select the page set. The data is divided into blocks, and a multi-branch parallel strategy is used to filter the data blocks: the first branch uses statistical features to calculate the importance score of each data block and filter the global key block set. The second branch selects the contiguous data blocks closest to the current query position to form a local block set. ; The dynamic fusion and sparse attention computation module is used to process the global key block set. With the local block set Merging and deduplication are performed to obtain a set of data blocks. The data block set Perform sparse attention computation and output the inference result.

[0013] A third aspect of the present invention provides a computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the steps of the large language model inference method based on the cascaded sparse attention mechanism as described in the first aspect.

[0014] Compared with the prior art, the advantages of this invention are: This invention designs a query-aware coarse-grained page selection mechanism. By dividing the key-value cache into fixed-size pages and pre-extracting four-dimensional lightweight statistical features, these statistics can be directly used with the query vector to estimate the upper limit score of the interaction during inference. This strategy cleverly avoids the massive full matrix dot product operation, and can quickly and accurately filter out redundant contextual information at the macro level, significantly reducing the candidate search space and memory access overhead. Thus, it significantly improves the inference speed of large language models while preserving core global semantic information. This invention also proposes a novel multi-branch fine-grained block selection and dynamic fusion architecture. Within the retained key pages, it combines Top-p key block selection based on statistical scores with the nearest key block retention strategy based on a sliding window to ensure that short-distance local dependencies are not lost.

[0015] This progressive cascaded sparsity design, from macroscopic pages to microscopic data blocks, overcomes the shortcomings of traditional coarse-grained methods that are prone to losing details, and also makes up for the computational cost of purely fine-grained methods. It successfully breaks through the secondary complexity bottleneck of standard attention mechanisms and achieves efficient large-scale language model inference while ensuring that the inference accuracy of long context tasks is almost undamaged. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] In the attached diagram: Figure 1 This is the overall architecture diagram of the large language model inference method based on the cascaded sparse attention mechanism in this specific embodiment; Figure 2 This is a flowchart illustrating the specific process architecture of the large language model reasoning method based on the cascaded sparse attention mechanism in this embodiment. Figure 3 This is a line graph showing the compression ratios of the large language model inference method based on the cascaded sparse attention mechanism in this specific implementation on the qa_quality dataset. Figure 4 This is a line graph showing the compression ratios of the large language model inference method based on the cascaded sparse attention mechanism in this specific implementation on the qa_squad dataset. Figure 5 The image shows a line graph of the compression ratio of the large language model inference method based on the cascaded sparse attention mechanism in this specific implementation on the qa_toefl dataset. Figure 6 This is the overall architecture diagram of the large language model reasoning system based on the cascaded sparse attention mechanism in this specific embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0019] Example 1: Please see Figure 1 and Figure 2 This invention provides a large language model inference method based on a cascaded sparse attention mechanism, comprising the following steps: S1. Page selection for query awareness: Obtain a long text sequence containing multiple lexical units, input it into a large language model to generate a key-value cache and store it; The key-value cache of the large language model is divided into pages. A global importance score for each page is calculated by extracting statistical features from all key vectors within that page. Based on these global importance scores, all pages are sorted and filtered to construct a page set. ; S2, Multi-branch fine-grained block selection: The page collection The data is divided into blocks, and a multi-branch parallel strategy is used to filter the data blocks: the first branch uses statistical features to calculate the importance score of each data block and filter the global key block set. The second branch selects the contiguous data blocks closest to the current query position to form a local block set. ; S3, Dynamic Fusion and Sparse Attention Computation: The global key block set With the local block set Merging and deduplication are performed to obtain a set of data blocks. The data block set Perform sparse attention computation and output the inference result.

[0020] As a preferred implementation, a long text sequence including multiple lexical units is obtained, input into a large language model to generate a key-value cache, and then stored.

[0021] Specifically, when a user inputs text into a large language model, the text is first lexicalized, breaking it down into multiple words to obtain an input sequence. This input sequence, containing multiple words, is then fed into the large language model all at once. During the pre-filling stage, the large language model processes these words in parallel: for each word, the model converts it into a vector representation through an embedding layer, and then calculates the corresponding key vector K and value vector V using a linear transformation matrix in a multi-head attention layer. After calculation, the key and value vectors corresponding to all words are organized into key vector matrices and value vector matrices, and stored in physical memory, forming a key-value cache.

[0022] In a preferred embodiment, step S1 includes: S101. Divide the key-value cache of the large language model into fixed-size blocks. The page, and extract all key vectors in the feature dimension of each page. The four-dimensional statistical characteristics include: maximum value Minimum value mean and variance ; S102. Using each component of the given query vector, perform a dot product operation with the statistical features of the corresponding dimension, and extract the maximum value of the product results in the corresponding dimension as the maximum interaction contribution value; for all feature dimensions... The global importance score of a page is obtained by summing the maximum interaction contribution values ​​on each page, and is expressed as follows: ; in, This indicates the page's overall importance score; Indicates the query vector at the th The values ​​of each dimension, i.e., the query vector components; Indicates the total number of feature dimensions; This represents a maximum value function used to filter out the maximum interaction contribution value for a corresponding dimension from four statistical products, thereby estimating the importance of a page while avoiding the overhead of full matrix multiplication. , , , These represent the key vectors within the page at the [number]th [position]. Maximum, minimum, mean, and variance in each dimension. Used to sum and aggregate the maximum interaction contribution values ​​across all dimensions of the page; S103. Sort all pages according to their global importance score and filter the top-scoring pages. Each page is the top-K most relevant pages, and a page set is constructed based on these top-K most relevant pages. , represented as: ; in, This represents the set of the Top-K most relevant pages retained after filtering by importance score; This means extracting the top-scoring pages from all pages in the large language model key-value cache. Operations on individual elements; page represents the page. This represents the total number of pages in the key-value cache. Budget parameters for the preset number of pages to retain.

[0023] As a specific implementation method, in step S2, the page set Further divided into sizes For non-overlapping data blocks, a multi-branch parallel strategy is used to filter the data blocks.

[0024] As a specific implementation method, in step S2, the first branch uses statistical features to calculate the importance score of each data block and filter the global key block set. Specifically: S201, The first branch, i.e., the selection branch, extracts all key vectors in each data block along the feature dimension. Block-level statistical characteristics, including: maximum value Minimum value mean and variance ; S202. Using each component of the given query vector, perform dot product operations with the block-level statistical features of the corresponding dimension, and extract the maximum value of the product results in the corresponding dimension as the maximum interaction contribution value. S203, All feature dimensions The block-level importance score is calculated by summing the maximum interaction contribution values ​​on each block, and is expressed as follows: ; in, This represents the block-level importance score of the data block, where b represents the data block. , , , These represent the key vectors within the data block at the th... Maximum, minimum, mean, and variance in each dimension; Used to sum and aggregate the maximum interaction contribution values ​​across all dimensions of a data block; S204, Page Collection The union of data blocks contained in each page is used to obtain a global candidate data block set. The top P data blocks with the highest block-level importance scores are extracted from the global candidate data block set to form a global key data block set. See Figure 2 The TOP-P block in the code is represented as: ; in, Indicates the current reasoning step Next, select the set of global key data blocks extracted from the selected branch; This indicates the extraction of the top block-level importance scores from the global candidate data block set. One data block operation; The parameter for the preset number of critical data blocks to retain; express Represents a collection of pages single page The collection of all data blocks contained therein; This represents the union operation, which is the set of pages. All data blocks from all pages are merged into a unified global candidate block set.

[0025] In one specific implementation, in step S2, the second branch is a local window branch, which selects the continuous data blocks closest to the current query position to form a local block set. See Figure 1 The most recent W block in the data is represented as: ; in, Relative to the current query position Last The most recent data block, Indicates the number of consecutive data blocks.

[0026] Specifically, step S3 includes: S301, Global Key Block Set With local block set pass The operation performs merging and deduplication to obtain a set of data blocks. See Figure 2 In this context, P~P+W is represented as: ; S302, According to the data block set Through data collection operations, sparse key matrices and sparse value matrices are extracted from the original key-value matrices (key matrix and value matrix), represented as follows: ; ; in, Indicates data collection operation, Represents the key matrix, Represents a value matrix, Represents the sparse bond matrix. This represents the coefficient value matrix.

[0027] S303, Limit the number of data blocks retained in each step to [number]. Number of data blocks, i.e., a maximum total number of data blocks to retain. 1 token, of which This represents the number of tokens contained in a single data block, thus completing the sparse pruning operation for the key-value cache. S304. Based on the sparse key matrix and the sparse value matrix, the query vector performs attention calculation on the pruned key-value cache to obtain the processed key-value cache. , represented as: ; in, Indicates the current step The query feature vector; This represents the transpose of the sparse bond matrix; The feature dimension size is used to scale and distribute the dot product score. This represents the normalized exponential function, used to convert the scaled inner product into attention weights; For the corresponding sparse value matrix; key-value cache The text used for subsequent decoding to generate the current word is then output as the reasoning result.

[0028] Example 2: See Figure 6 Based on the same inventive concept, this invention also provides a large language model reasoning system based on a cascaded sparse attention mechanism, comprising: The acquisition module is used to acquire long text sequences containing multiple lexical units, input them into a large language model to generate key-value caches and store them; The query-aware page selection module divides the key-value cache of the large language model into pages. It calculates the global importance score of each page by extracting statistical features of all key vectors within that page, and then sorts and filters all pages based on these global importance scores to construct a page set. ; A multi-branch fine-grained block selection module is used to select the page set. The data is divided into blocks, and a multi-branch parallel strategy is constructed to filter the data blocks. The first branch uses statistical features to calculate the importance score of each data block and filter the global key block set. The second branch selects the contiguous data blocks closest to the current query position to form a local block set. ; The dynamic fusion and sparse attention computation module is used to process the global key block set. With the local block set Merging and deduplication are performed to obtain a set of data blocks. The data block set Perform sparse attention computation and output the inference result.

[0029] Example 3: A computer-readable storage medium stores a computer program that, when executed by a processor, implements the large language model reasoning method based on a cascaded sparse attention mechanism as described in Embodiment 1 above. When the program runs on a computer processor, it achieves progressive sparsity from macro to micro, effectively breaking through the bottleneck of quadratic complexity in large models while maintaining high-precision long-context reasoning.

[0030] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should be protected by the present invention.

[0031] Example 4: This embodiment uses the open-source Qwen-8B model as the base model for evaluation. Specifically, this embodiment focuses on natural language understanding and reading comprehension tasks, selecting representative long-text question answering datasets: long document multiple-choice question answering (qa_quality), document-based extractive question answering (qa_squad), and TOEFL reading comprehension question answering (qa_toefl) as test datasets. These three datasets are all low-dispersion, low-span benchmarks, requiring the model to locate the correct answer amidst a large amount of noise. Due to their reliability and broad coverage, they are widely used for evaluating the capabilities of large models with long contexts.

[0032] In this embodiment, the input sequence length of the model is set to 16384 (16K) tokens. The model uses accuracy and F1 score as evaluation metrics. Accuracy measures the proportion of correct predictions; a value closer to 1 indicates higher prediction accuracy. The F1 score is the harmonic mean of precision and recall, used to measure the accuracy of the model in extracting text segments; a higher value indicates stronger ability to capture and interpret actual textual clues. Experiments were conducted on single or multiple high-performance GPUs, and evaluations were performed under different key-value cache compression ratios by controlling the application of sparsity strategies only during the decoding stage.

[0033] To demonstrate the effectiveness of this invention, dynamic cache eviction methods SnapKV and AdaSnapKV were selected for comparative experiments with the method proposed in this invention. Three representative cache compression ratios of 2.5x, 5.0x, and 10.0x were chosen for comparison. Detailed information on the experimental results is shown in Tables 1-3 below, which compare the accuracy of qa_quality, the F1 score of qa_squad, and the accuracy of qa_toefl, respectively.

[0034] Table 1. Comparison of accuracy between the present invention and existing methods on the qa_quality task. ; Table 2 Comparison of F1 scores between the present invention and existing methods on the qa_squad task. ; Table 3. Comparison of accuracy between the present invention and existing methods on the QA_TOEFL task.

[0035] Across all tasks and compression scales, the method of this invention consistently demonstrates extremely high robustness. Particularly on the qa_quality task at compression rates of 2.5x and 10.0x, the accuracy of this invention is 0.655 and 0.640, respectively, significantly outperforming the accuracy of SnapKV and AdaSnapKV. This indicates that the method of this invention has higher accuracy when facing long-range semantic extraction.

[0036] This invention also performs excellently on the F1 score of the qa_squad task, achieving F1 scores of 0.825 and 0.825 at compression ratios of 2.5x and 10.0x, respectively, maintaining overall stability and outperforming other baseline methods in most cases. This further demonstrates that the cascaded sparsity strategy of this invention effectively preserves the key local evidence needed to answer factual questions while reducing contextual redundancy.

[0037] In terms of the qa_toefl task metrics, the method of this invention also performs best across various compression scales, with accuracy consistently remaining between 0.885 and 0.890, demonstrating a good ability to explain variations in long text data. In contrast, the accuracy of other methods generally exhibits greater fluctuations, especially at medium compression rates (such as 5.0x), where SnapKV and AdaSnapKV show a significant downward trend.

[0038] See Figure 3 , Figure 4 , Figure 5 The accuracy performance of different methods at various compression ratios on different datasets (qa_quality, qa_squad, and qa_toefl) was compared. The red pentagrams represent the method of this invention, the blue circles represent the SnapKV method, and the green squares represent the AdaSnapKV method. As can be seen from the figure, with the increase in compression ratio (from 2.5x to 10.0x), the accuracy of all methods showed the expected decrease, but the prediction curve of this invention remained at the top. This demonstrates that the "page-first, data-block-later" hierarchical selection strategy of this invention can effectively preserve key local evidence even under extreme compression conditions, exhibiting stronger stability and reliability.

[0039] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should be protected by the present invention.

Claims

1. A large language model reasoning method based on cascaded sparse attention mechanism, characterized in that, Includes the following steps: S1. Page selection for query awareness: Obtain a long text sequence containing multiple lexical units, input it into a large language model to generate a key-value cache and store it; The key-value cache in the large language model is divided into pages. A global importance score for each page is calculated by extracting statistical features from all key vectors within that page. Based on these global importance scores, all pages are sorted and filtered to construct a page set. ; S2, Multi-branch fine-grained block selection: The page collection The data is divided into blocks, and a multi-branch parallel strategy is constructed to filter the data blocks: the first branch uses statistical features to calculate the importance score of each data block and filters the global key block set. The second branch selects the contiguous data blocks closest to the current query position to form a local block set. ; S3, Dynamic Fusion and Sparse Attention Computation: The global key block set With the local block set Merging and deduplication are performed to obtain a set of data blocks. The data block set Perform sparse attention computation and output the inference result.

2. The method according to claim 1, characterized in that, The key-value cache of the large language model is divided into pages. A global importance score for each page is calculated by extracting statistical features from all key vectors within that page. Based on these global importance scores, all pages are sorted and filtered to construct a page set. ,include: S101. Divide the key-value cache of the large language model into pages of fixed size, and extract the maximum, minimum, mean and variance of all key vectors in each feature dimension within each page. S102. Using each component of the given query vector, perform dot product operations with the statistical features of the corresponding dimension, and extract the maximum value of the product results under the corresponding dimension as the maximum interaction contribution value. S103. Sum the maximum interaction contribution values ​​across all feature dimensions to obtain the global importance score of the page; S104. Sort all pages according to the global importance score and filter the top pages with the highest scores. Each page is the top-K most relevant pages, and a page set is constructed based on these top-K most relevant pages. .

3. The method according to claim 1, characterized in that, The first branch uses statistical features to calculate the importance score of each data block and then filters the global set of key blocks. ,include: S201. The first branch is the selection branch, which extracts the maximum, minimum, mean and variance of all key vectors in each data block in each feature dimension. S202. Using each component of the given query vector, perform dot product operations with the block-level statistical features of the corresponding dimension, and extract the maximum value of the product results in the corresponding dimension as the maximum interaction contribution value. S203. Sum the maximum interaction contribution values ​​across all feature dimensions to obtain the block-level importance score; S204, Page Collection The union of data blocks contained in each page is used to obtain a global candidate data block set. The top P data blocks with the highest block-level importance scores are extracted from the global candidate data block set to form a global key data block set. , is represented as: ； in, Indicates the current reasoning step Next, select the set of global key data blocks extracted from the selected branch; This indicates the extraction of the top block-level importance scores from the global candidate data block set. One data block operation; The parameter for the preset number of critical data blocks to retain; This represents the block-level importance score of data block b. Represents a collection of pages single page The collection of all data blocks contained therein; This represents the union operation, which is the set of pages. All data blocks from all pages are merged into a unified global candidate block set.

4. The method according to claim 1, characterized in that, The second branch is a local window branch, which selects the contiguous data blocks closest to the current query position to form a local block set. , is represented as: ； in, Relative to the current query position Last The most recent data block, Indicates the number of consecutive data blocks.

5. The method according to claim 1, characterized in that, The global key block set With the local block set Merging and deduplication are performed to obtain a set of data blocks. The data block set Perform sparse attention computation and output the inference results, including: S301, the global key block set With the local block set pass The operation performs merging and deduplication to obtain a set of data blocks. ; S302, According to the data block set Through data collection operations, sparse key matrices and sparse value matrices are extracted from the original key-value matrix, represented as follows: ； ； in, Indicates data collection operation, Represents the key matrix. Represents a value matrix, Represents the sparse bond matrix. Represents the coefficient value matrix; S303. Limit the number of data blocks retained in each step to a preset range to control the total number of tokens and complete the sparse pruning operation of the key-value cache; S304. Based on the sparse key matrix and the sparse value matrix, perform attention calculation on the pruned key-value cache to obtain the processed key-value cache. , is represented as: ； in, Indicates the current step The query feature vector; This represents the transpose of the sparse bond matrix; The size of the feature dimension; This represents the normalized exponential function; This is the corresponding sparse value matrix; Key-value cache The text used for subsequent decoding to generate the current word is then output as the reasoning result.

6. A large language model reasoning system based on a cascaded sparse attention mechanism, characterized in that, include: The acquisition module is used to acquire long text sequences containing multiple lexical units, input them into a large language model to generate key-value caches and store them; The query-aware page selection module divides the key-value cache of the large language model into pages. It calculates the global importance score of each page by extracting statistical features of all key vectors within that page, and then sorts and filters all pages based on these global importance scores to construct a page set. ; A multi-branch fine-grained block selection module is used to select the page set. The data is divided into blocks, and a multi-branch parallel strategy is constructed to filter the data blocks: the first branch uses statistical features to calculate the importance score of each data block and filters the global key block set. The second branch selects the contiguous data blocks closest to the current query position to form a local block set. ; The dynamic fusion and sparse attention computation module is used to process the global key block set. With the local block set Merging and deduplication are performed to obtain a set of data blocks. The data block set Perform sparse attention computation and output the inference result.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it is used to implement the large language model reasoning method based on the cascaded sparse attention mechanism as described in any one of claims 1-5.

Images