Large model reasoning acceleration processing method and device, electronic equipment and storage medium

Abstract

The invention provides a large model reasoning acceleration processing method and device, electronic equipment and a storage medium, and belongs to the technical field of large model computation.The method comprises the steps that a query sequence is divided into a plurality of query groups, and a parallel computing unit is allocated to each query group, pre-loading the required key value cache in the local storage space of the parallel computing unit; in each parallel computing unit, a plurality of thread bundles are used for carrying out parallel computing on the query of the current token and the dot product of all historical keys, and a similarity vector is generated; performing normalization processing on each similarity vector, and outputting a plurality of attention weights; performing weighted summation on the attention weights and values to obtain a weighted sum, and temporarily storing the weighted sum in a local storage space of the parallel computing unit; under the condition that all the parallel computing units complete computing, the weighted sum of all the parallel computing units is merged, and a complete output result is written back into the global storage space. The large model reasoning delay is reduced, and the throughput and the bandwidth utilization rate are improved.

Description

Technical Field

[0001] This application relates to the field of large model computing technology, and in particular to a method, apparatus, electronic device and storage medium for accelerating large model inference. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in natural language processing, code generation, and intelligent dialogue, the performance bottleneck in their inference phase has become increasingly prominent. Especially when handling long contextual inputs, the computational complexity and memory overhead of the self-attention mechanism increase quadratically with sequence length, leading to a significant increase in inference latency and limited throughput. Among these, the key-value cache (KV cache), as the core data structure used in the attention mechanism to store intermediate results of historical context, is frequently accessed, constituting a major source of memory bandwidth pressure during inference. Therefore, how to efficiently manage the storage and access of the KV cache, reduce redundant memory transfers, and improve computational parallelism efficiency has become crucial for achieving low-latency, high-throughput inference for large models.

[0003] Current mainstream techniques for accelerating large-scale model inference mainly include FlashAttention, PagedAttention, and Group Query Attention (GQA). FlashAttention reduces global storage access and improves bandwidth utilization through operator fusion and tiling strategies; PagedAttention borrows from operating system paging to optimize the memory layout of the key-value cache to alleviate fragmentation; GQA reduces the total cache size and access frequency by grouping multiple query headers and sharing the same key-value cache. Although these methods alleviate memory pressure to some extent, they still have significant limitations: on the one hand, repeated accesses to the same key-value cache by multiple query headers are not effectively aggregated, resulting in the same data being loaded multiple times in the GPU memory hierarchy, causing redundant data movement; on the other hand, existing solutions generally lack the utilization of the temporal locality of memory access, making it difficult to fundamentally alleviate bandwidth bottlenecks. In addition, these methods focus on coarse-grained task partitioning at the parallel scheduling level, limiting the full utilization of GPU computing resources.

[0004] In summary, how to effectively eliminate redundant loading of KV cache, improve the temporal locality of memory access, and improve hardware efficiency while ensuring complete context modeling capabilities has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for accelerating large model inference, in order to solve the defects of redundant loading of KV cache, poor temporal locality, and low utilization of GPU computing resources in the prior art for large model inference.

[0006] This application provides a method for accelerating large model inference, including the following steps: The query sequence is divided into multiple query groups, where each query group contains multiple queries that share the same key-value cache; A parallel computing unit is allocated to each query group, and the key-value cache is preloaded in the local storage space of the parallel computing unit; In each of the parallel computing units, parallel computing resources are used to calculate the dot product of the current token query and all historical keys in parallel based on preset inter-query parallelism and intra-query parallelism to generate a similarity vector; wherein, the parallel computing is performed simultaneously within the same parallel computing unit: the calculation of different queries is processed in parallel at the same token position based on the inter-query parallelism, and the words at different positions are processed in parallel within a query based on the intra-query parallelism. The similarity vectors are normalized to output multiple attention weights that sum to 1; The multiple attention weights are summed with the value Value to obtain a weighted sum, which is then temporarily stored in the local storage space. Once all parallel computing units have completed their calculations, the weighted sums of the parallel computing units are combined to obtain the complete output result, which is then written back to the global storage space.

[0007] According to the large model inference acceleration method provided in this application, the inter-query parallelism and the intra-query parallelism are determined based on the following steps: Based on the range of values ​​for the inter-query parallelism and the intra-query parallelism corresponding to the inter-query parallelism, multiple parallelism candidate schemes are generated. Based on the measured throughput of each of the proposed parallelism candidate schemes, the final inter-query parallelism and intra-query parallelism are determined.

[0008] According to the large model inference acceleration method provided in this application, the intra-query parallelism corresponding to the inter-query parallelism is determined based on the following steps: The total number of thread bundles is determined based on the total number of threads on the GPU. The intra-query parallelism corresponding to the inter-query parallelism is determined based on the ratio of the total number of thread bundles to the inter-query parallelism.

[0009] According to the large model inference acceleration method provided in this application, determining the intra-query parallelism corresponding to the inter-query parallelism based on the ratio of the total number of thread bundles to the inter-query parallelism includes: Based on the ratio of the total number of thread bundles to the equivalent resource units, the intra-query parallelism corresponding to the inter-query parallelism is determined. The equivalent resource unit is the product of the inter-query parallelism and the protection coefficient, whereby the protection coefficient is used to conservatively reduce the theoretical hardware resources.

[0010] According to the large-model inference acceleration method provided in this application, the intra-query parallelism is calculated based on the following formula: intra_i=floor(W total / (inter_i×Protection_Value)); Where inter_i is the inter-query parallelism, intra_i is the maximum value of the intra-query parallelism corresponding to inter_i, and W total It represents the total number of thread bundles, Protection_Value is the protection factor, and floor() rounds down.

[0011] According to the large model inference acceleration method provided in this application, the step of determining the final inter-query parallelism and intra-query parallelism based on the measured throughput of each of the proposed parallelism candidate schemes includes: The candidate parallelism scheme with the highest measured throughput and that satisfies the memory access constraints was selected as the final inter-query parallelism and intra-query parallelism. The memory access constraint is that the local storage space required by the candidate parallelism scheme is less than or equal to the upper limit of the local storage space capacity of a parallel computing unit.

[0012] This application also provides a large model inference acceleration processing device, including the following modules: The sequence grouping module is used to divide a query sequence into multiple query groups, wherein each query group contains multiple queries that share the same key-value cache. The resource allocation module is used to: allocate a parallel computing unit for each query group and preload the key-value cache in the local storage space of the parallel computing unit; The parallel computing module is used to: in each of the parallel computing units, utilize parallel computing resources to calculate the dot product of the current token query and all historical keys in parallel based on preset inter-query parallelism and intra-query parallelism, and generate a similarity vector; wherein, the parallel computing is performed simultaneously within the same parallel computing unit: based on the inter-query parallelism, it processes the calculation of different queries in parallel at the same token position, and based on the intra-query parallelism, it processes words at different positions in parallel within a query; The normalization module is used to: normalize each of the similarity vectors and output multiple attention weights that sum to 1; The weighted summation module is used to: sum the multiple attention weights and the value Value in a weighted manner to obtain a weighted sum and temporarily store it in the local storage space; The result integration module is used to: merge the weighted sums of all parallel computing units after all parallel computing units have completed their calculations, obtain a complete output result, and write the complete output result back to the global storage space.

[0013] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the large model inference acceleration processing methods described above.

[0014] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the large model inference acceleration processing method as described above.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the large model inference acceleration processing methods described above.

[0016] The large-model inference acceleration method, apparatus, electronic device, and storage medium provided in this application place queries sharing the same KV cache in a parallel computing unit, enabling multiple queries to be processed in parallel within the same parallel computing unit, achieving inter-query parallelism. Simultaneously, within each parallel computing unit, multiple thread bundles are used to execute the dot product operation of the current token and all historical keys in parallel, achieving intra-query parallelism. This two-level parallel architecture breaks through the limitations of existing technologies that rely solely on single-dimensional parallelism, fully activating the fine-grained concurrency capabilities of the GPU, and significantly improving computational throughput, especially in long-context scenarios. Furthermore, all KV caches required by the query group are pre-loaded in the local storage space of the parallel computing unit. Since the local storage space has low latency and high bandwidth characteristics, and is shared by all threads within the parallel computing unit, when multiple threads need to access the same KV cache, it only needs to be loaded from the global storage space to the local storage space once. Subsequent accesses directly hit the on-chip storage, completely avoiding repeated global reads caused by the lack of locality in traditional solutions. This effectively solves the problem of multiple loadings of the same KV cache in existing technologies and significantly reduces memory bandwidth usage. Furthermore, operations such as similarity vector normalization and weighted summation of attention weights and values ​​are all performed within the parallel computing units. Intermediate results are temporarily stored in local storage space and only merged and written back to global storage space after all parallel computing units have completed their local computations. This reduces the number of global write-backs of intermediate data and optimizes the execution pipeline by overlapping computation and memory access, thereby improving overall efficiency. In summary, this application, through query grouping, parallel computing unit binding, local storage space preloading, and two-level parallel computing, fundamentally improves the access locality of the KV cache without changing the model structure, eliminates redundant data movement, and fully releases the parallel computing potential of the GPU, thereby achieving lower latency, higher throughput, and better memory bandwidth utilization. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the method for accelerating large model inference provided in this application; Figure 2 This is a schematic diagram of the parallel computing architecture provided in this application; Figure 3 This is a schematic diagram of the structure of the large model inference acceleration processing device provided in this application; Figure 4This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that in the description of the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, or integral connections; they can be mechanical connections or electrical connections; they can be direct connections or indirect connections through an intermediate medium; and they can be internal connections between two elements. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0021] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0022] The following is combined Figures 1-4 This application describes the large model inference acceleration processing method, apparatus, electronic device, and storage medium provided in the embodiments of this application.

[0023] Figure 1 This is a flowchart illustrating the method for accelerating large model inference provided in this application, as shown below. Figure 1 As shown, the method includes the following: S110, the query sequence is divided into multiple query groups, wherein each query group contains multiple queries that share the same key-value cache; S120, allocate a parallel computing unit for each query group, and preload the key-value cache in the local storage space of the parallel computing unit; S130, in each of the parallel computing units, parallel computing resources are used to calculate the dot product of the current token query and all historical keys in parallel based on preset inter-query parallelism and intra-query parallelism to generate a similarity vector; wherein, the parallel computing is performed simultaneously within the same parallel computing unit: the calculation of different queries is processed in parallel at the same token position based on the inter-query parallelism, and the words at different positions are processed in parallel within a query based on the intra-query parallelism; S140, normalize each of the similarity vectors and output multiple attention weights that sum to 1; S150, the multiple attention weights and values ​​are weighted and summed to obtain a weighted sum, which is then temporarily stored in the local storage space; S160, after all parallel computing units have completed their calculations, the weighted sums of the parallel computing units are combined to obtain a complete output result, and the complete output result is written back to the global storage space.

[0024] It should be noted that the execution subject of the large model inference acceleration processing method provided in this application embodiment can be a server, computer device, such as a mobile phone, tablet computer, laptop computer, handheld computer, vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0025] In S110, the query sequence is a batch of queries to be processed during the autoregressive generation process. Multiple queries in a batch are divided into several subsets according to the KV cache, and each subset constitutes a query group that shares the same KV cache.

[0026] In S120, each query group is assigned to an independent computing unit, namely a parallel computing unit. Each parallel computing unit reads the required historical key and value cache from the global storage space according to the query group it is responsible for, with a read length of L, i.e., the context length. This KV data is then loaded into the local storage space of the parallel computing unit at once, thereby avoiding each thread accessing the global storage space separately and making full use of the high bandwidth and low latency characteristics of the local storage space.

[0027] Taking the CUDA (Compute Unified Device Architecture) architecture as an example, a thread block is allocated for each query group, and the required key-value cache is preloaded in the shared memory of the thread block.

[0028] It is understandable that, in the specific implementation process, the large model inference acceleration processing method provided in this application can also be used in other architecture environments such as AMD platform, NPU / TPU, etc., and its operating principle is the same, so it will not be described in detail here.

[0029] In S130, a warp is the smallest unit of execution in parallel computing. Continuing with CUDA as an example, a warp, as the smallest unit of execution in CUDA, typically contains 32 threads, executing synchronously in a single-instruction multithreaded manner. Multiple warps form a single thread block.

[0030] Figure 2 This is a schematic diagram of the parallel computing architecture provided in this application, as shown below. Figure 2 As shown, inter-query parallelism involves performing attention computations on multiple independent queries in parallel at the same time step, i.e., generating the same output position. The degree of inter-query parallelism is the number of parallel processes in inter-query parallel computation. Intra-query parallelism involves splitting the calculation of the dot product between the current query and all historical keys (of length L) within a single query into multiple thread bundles for parallel execution to accelerate long context processing. The degree of intra-query parallelism is the number of parallel processes in intra-query parallel computation.

[0031] In this embodiment of the application, within each parallel computing unit, parallel dot product calculation is performed through inter-query parallel computing and intra-query parallel computing. Each query generates a similarity vector of length L. The overall output of the parallel computing unit is a similarity vector matrix containing all queries within the parallel computing unit.

[0032] In S140, the similarity vector is normalized by the Softmax function in the parallel computing unit, and the dot product result is converted into attention weights in the form of a probability distribution, satisfying that the weights sum to 1. That is, for the similarity vector matrix, the Softmax operation is performed row by row, and the original similarity score of each row is converted into an attention weight distribution with a sum of 1.

[0033] In S150, the preloaded Value cache is weighted and summed using attention weights. After the output vector (i.e. attention output) of each query is calculated, it is temporarily stored in the local storage space, waiting for subsequent merging. That is, the attention weight matrix obtained in S130 is multiplied with the Value matrix in the KV cache to obtain the weighted output vector of each token, and the result is temporarily stored in the local storage space.

[0034] In S160, after all parallel computing units complete their local computations, the host or a specific synchronization mechanism triggers result merging. The local outputs of each parallel computing unit are concatenated in the original query order to form a complete batch output. Finally, the complete output is written back to the global memory space for subsequent use. The global memory space, also known as GPU main memory, has a large capacity but high access latency and limited bandwidth.

[0035] Preferably, the complete output result is written back to the designated buffer in the GPU memory using non-blocking memory write operations. Coalesced write is employed to improve memory write efficiency, and synchronization barriers ensure the integrity of the write operation. Specifically, adjacent threads access contiguous and aligned memory addresses, thereby merging multiple small write requests into a minimum number of high-bandwidth transactions, maximizing DRAM utilization. The write operation is processed asynchronously by the hardware memory subsystem, without blocking subsequent scheduling of computing units, thus improving pipeline efficiency.

[0036] The large-model inference acceleration method provided in this application places queries sharing the same KV cache in a parallel computing unit, enabling multiple queries to be processed in parallel within the same unit, achieving inter-query parallelism. Simultaneously, within each parallel computing unit, multiple thread bundles are used to perform the dot product operation between the current token and all historical keys in parallel, achieving intra-query parallelism. This two-level parallel architecture overcomes the limitations of existing technologies that rely solely on single-dimensional parallelism, fully activating the fine-grained concurrency capabilities of the GPU, significantly improving computational throughput, especially in long-context scenarios. Furthermore, all KV caches required by the query group are pre-loaded in the local storage space of the parallel computing unit. Since the local storage space has low latency and high bandwidth characteristics and is shared by all threads within the parallel computing unit, when multiple threads need to access the same KV cache, it only needs to be loaded from the global storage space to the local storage space once. Subsequent accesses directly hit the on-chip storage, completely avoiding repeated global reads caused by the lack of locality in traditional solutions. This effectively solves the problem of multiple loadings of the same KV cache in existing technologies, significantly reducing memory bandwidth usage. Furthermore, operations such as similarity vector normalization and weighted summation of attention weights and values ​​are all performed within the parallel computing units. Intermediate results are temporarily stored in local storage space and only merged and written back to global storage space after all parallel computing units have completed their local computations. This reduces the number of global write-backs of intermediate data and optimizes the execution pipeline by overlapping computation and memory access, thereby improving overall efficiency. In summary, this application, through query grouping, parallel computing unit binding, local storage space preloading, and two-level parallel computing, fundamentally improves the access locality of the KV cache without changing the model structure, eliminates redundant data movement, and fully releases the parallel computing potential of the GPU, thereby achieving lower latency, higher throughput, and better memory bandwidth utilization.

[0037] In an optional embodiment, intra-query parallelism is the number of parallel processes in intra-query parallel computation, and inter-query parallelism is the number of parallel processes in inter-query parallel computation. The inter-query parallelism and the intra-query parallelism are determined based on the following steps: Based on the range of values ​​for the inter-query parallelism and the intra-query parallelism corresponding to the inter-query parallelism, multiple parallelism candidate schemes are generated. Based on the measured throughput of each of the proposed parallelism candidate schemes, the final inter-query parallelism and intra-query parallelism are determined.

[0038] Here, query parallelism refers to the number of threads or thread bundles used to compute the dot product of a single query with all historical key vectors in parallel when processing a single query, reflecting the granularity of splitting long contexts.

[0039] Here, the parallelism between queries refers to the number of independent queries processed in parallel within the same inference step. The maximum value can be set adaptively according to actual usage requirements.

[0040] Here, the parallelism candidate scheme is a combination of (inter-query parallelism, intra-query parallelism) that satisfies hardware constraints. For example, (4, 64) means that 4 queries are processed simultaneously, and the attention score of each query is computed in parallel using 64 thread bundles.

[0041] In this embodiment of the application, the i-th parallelism candidate scheme is set as (inter_i, intra_i), where inter_i is the inter-query parallelism, and the value range is [1, Group_size]. The intra-query parallelism intra_i corresponding to inter_i is determined, and the inter-query parallelism inter_i gradually increases from 1 to Group_size to generate multiple parallelism candidate schemes.

[0042] Here, the measured throughput is the number of tokens generated per unit time when running a candidate scheme with a specific degree of parallelism on real GPU hardware.

[0043] Furthermore, in this embodiment, a test kernel function is executed for each parallelism candidate scheme. The complete attention calculation is run on the actual KV cache data, including similarity dot product calculation, Softmax and PV product. The execution time of the kernel is recorded using CUDA events, the throughput of the scheme is calculated, and the parallelism candidate scheme with the highest throughput is selected.

[0044] The large model inference acceleration processing method provided in this application solves the suboptimal problem of one-size-fits-all parallelism configuration caused by fixed parallelism by selecting the optimal parallelism candidate scheme that matches the current load through actual testing. This ensures that hardware resources are accurately matched to computing needs and maximizes GPU resource utilization.

[0045] In an optional embodiment, the intra-query parallelism corresponding to the inter-query parallelism is determined based on the following steps: The total number of thread bundles is determined based on the total number of threads on the GPU. The intra-query parallelism corresponding to the inter-query parallelism is determined based on the ratio of the total number of thread bundles to the inter-query parallelism.

[0046] Here, the total number of GPU threads refers to the total number T of GPU threads available for performing attention computation in the current inference task. usable Under the CUDA architecture, this can be obtained by calling the CUDA runtime API, and other key hardware parameters of the target GPU device can also be obtained, including but not limited to the maximum number of threads per multiprocessor, memory bandwidth, and other complete hardware configuration information.

[0047] In the specific implementation process, the GetThread() function is called to obtain the total number of GPU threads, the thread bundle warp size is 32, and the total number of thread bundles W total via floor(T) usable / 32) is calculated, where floor() is the floor function.

[0048] The large model inference acceleration processing method provided in this application determines the corresponding intra-query parallelism based on the ratio of the total number of thread bundles to the inter-query parallelism. In other words, it forces the product of the inter-query parallelism and the intra-query parallelism to be approximately equal to the total number of thread bundles, so that the number of active thread bundles approaches the hardware limit and maximizes the utilization of GPU computing resources.

[0049] In an optional embodiment, determining the intra-query parallelism corresponding to the inter-query parallelism based on the ratio of the total number of thread bundles to the inter-query parallelism includes: Based on the ratio of the total number of thread bundles to the equivalent resource units, the intra-query parallelism corresponding to the inter-query parallelism is determined. The equivalent resource unit is the product of the inter-query parallelism and the protection coefficient, whereby the protection coefficient is used to conservatively reduce the theoretical hardware resources.

[0050] Here, the protection factor is an empirical scaling factor greater than 1 used to conservatively estimate available parallel resources, with the aim of avoiding kernel startup failure or performance degradation due to over-allocation of resources. Optionally, the protection factor is set to 4.

[0051] In this embodiment of the application, the equivalent resource unit ERU = inter_i × Protection_Value, where inter_i is the parallelism between queries and Protection_Value is the protection coefficient.

[0052] Furthermore, based on W total The ratio of / ERU determines the parallelism within the query.

[0053] It is understandable that methods that directly determine the parallelism within a query based on the ratio of the total number of thread bundles to the parallelism between queries assume that each query occupies only one resource. However, in reality, each query needs to exclusively occupy local storage space to store the KV cache, occupy a large number of registers, and consume scheduling metadata. When the parallelism between queries is large, even if the parallelism calculation within a query is valid, kernel startup failure may still occur due to register or local storage space exceeding limits. The large model inference acceleration processing method provided in this application explicitly amplifies the resource overhead between queries through a protection coefficient, prevents resource exceeding limits, reserves sufficient resources to ensure safe kernel startup, and guarantees system stability.

[0054] In an optional embodiment, the intra-query parallelism is calculated based on the following formula: intra_i=floor(W total / (inter_i×Protection_Value)); Where inter_i is the inter-query parallelism, intra_i is the maximum value of the intra-query parallelism corresponding to inter_i, and W total It represents the total number of thread bundles, Protection_Value is the protection factor, and floor() rounds down.

[0055] The large model inference acceleration processing method provided in this application ensures that the number of allocated resources does not exceed the available limit by rounding down, avoiding runtime errors and improving system stability. In addition, rounding down naturally outputs integers, which is consistent with the hardware scheduling model and conforms to the discrete nature of hardware resources.

[0056] In an optional embodiment, determining the final inter-query parallelism and intra-query parallelism based on the measured throughput of each of the candidate parallelism schemes includes: The candidate parallelism scheme with the highest measured throughput and that satisfies the memory access constraints was selected as the final inter-query parallelism and intra-query parallelism. The memory access constraint is that the local storage space required by the candidate parallelism scheme is less than or equal to the upper limit of the local storage space capacity of a parallel computing unit.

[0057] In this embodiment of the application, the measured throughput of all candidate parallelism schemes is compared, and the scheme with the highest throughput and that meets the memory access constraints is selected as the final (inter-query parallelism, intra-query parallelism) parameters.

[0058] In some embodiments, after selecting the scheme with the highest measured throughput, it is checked whether the scheme meets the memory access constraint, that is, the required local storage space size does not exceed the local storage space capacity limit of a single parallel computing unit of the GPU. If it does not meet the constraint, the schemes are selected from the suboptimal schemes in order of measured throughput until a feasible scheme that meets both the measured throughput requirement and the memory constraint is found.

[0059] In other embodiments, all candidate schemes that do not meet the memory access constraints are first filtered out from the remaining parallelism candidate schemes; then, the parallelism candidate scheme with the highest measured throughput is selected from the remaining parallelism candidate schemes as the final (inter-query parallelism, intra-query parallelism) parameters.

[0060] The large model inference acceleration method provided in this application ensures the physical feasibility of the solution through memory access constraints, prevents kernel startup failure, and realizes the safe and feasible parallel configuration of large model inference.

[0061] The following section uses the CUDA architecture as an example to illustrate the effectiveness of the large model inference acceleration method provided in this application, combined with specific test data.

[0062] This application dynamically probes multiple parallelism candidate schemes and selects the optimal parameters based on measured throughput, enabling multiple query heads sharing the KV cache to access the cache multiple times in a short period of time. This method significantly improves the cache hit rate and greatly reduces redundant data flow between GPU memory levels. As shown in Table 1, it reduces data movement by an average of 3.5 times, effectively improving memory bandwidth utilization efficiency.

[0063] Table 1 Comparison of video memory access This application combines intra-query parallelism and inter-query parallelism strategies, dynamically optimizing the parallelism parameter based on measured throughput. This enables multiple queries sharing a KV cache to be processed simultaneously, and multiple terms in a single query can also be processed in parallel. This combination of parallelism strategies maximizes hardware resource utilization. As shown in Table 2, it achieves an average performance improvement of 1.81 times compared to the vLLM benchmark system, with a maximum speedup of 1.95 times.

[0064] Table 2 Comparison of Acceleration Effects The large model inference acceleration processing apparatus provided in the embodiments of this application is described below. The large model inference acceleration processing apparatus described below can be referred to in correspondence with the large model inference acceleration processing method described above.

[0065] Figure 3 This is a schematic diagram of the structure of the large model inference acceleration processing device provided in this application, as shown below. Figure 3 As shown, the large model inference acceleration processing device may include, but is not limited to: Sequence grouping module 310 is used to: divide a query sequence into multiple query groups, wherein each query group contains multiple queries that share the same key-value cache; Resource allocation module 320 is used to: allocate a parallel computing unit for each query group and preload the key-value cache in the local storage space of the parallel computing unit; The parallel computing module 330 is used to: in each of the parallel computing units, utilize parallel computing resources to calculate the dot product of the current token query and all historical keys in parallel based on preset inter-query parallelism and intra-query parallelism, and generate a similarity vector; wherein the parallel computing is performed simultaneously within the same parallel computing unit: based on the inter-query parallelism, it processes the calculation of different queries in parallel at the same token position, and based on the intra-query parallelism, it processes words at different positions in parallel within a query; The normalization module 340 is used to: normalize each of the similarity vectors and output multiple attention weights that sum to 1; The weighted summation module 350 is used to: sum the multiple attention weights and the value Value in a weighted manner to obtain a weighted sum and temporarily store it in the local storage space; The result integration module 360 ​​is used to: merge the weighted sums of all the parallel computing units after all the parallel computing units have completed the calculation, obtain the complete output result, and write the complete output result back to the global storage space.

[0066] In an optional embodiment, the large model inference acceleration processing device further includes a parallelism determination module, used to determine the inter-query parallelism and intra-query parallelism based on the following steps: Based on the range of values ​​for the inter-query parallelism and the intra-query parallelism corresponding to the inter-query parallelism, multiple parallelism candidate schemes are generated. Based on the measured throughput of each of the proposed parallelism candidate schemes, the final inter-query parallelism and intra-query parallelism are determined. Wherein, the intra-query parallelism is the number of parallel processing operations in intra-query parallel computing, and the inter-query parallelism is the number of parallel processing operations in inter-query parallel computing.

[0067] In an optional embodiment, the parallelism determination module is further specifically used for: The total number of thread bundles is determined based on the total number of threads on the GPU. The intra-query parallelism corresponding to the inter-query parallelism is determined based on the ratio of the total number of thread bundles to the inter-query parallelism.

[0068] In an optional embodiment, the parallelism determination module is further specifically used for: Based on the ratio of the total number of thread bundles to the equivalent resource units, the intra-query parallelism corresponding to the inter-query parallelism is determined. The equivalent resource unit is the product of the inter-query parallelism and the protection coefficient, whereby the protection coefficient is used to conservatively reduce the theoretical hardware resources.

[0069] In an optional embodiment, the parallelism determination module specifically calculates the intra-query parallelism based on the following formula: intra_i=floor(W total / (inter_i×Protection_Value)); Where inter_i is the inter-query parallelism, intra_i is the maximum value of the intra-query parallelism corresponding to inter_i, and W total It represents the total number of thread bundles, Protection_Value is the protection factor, and floor() rounds down.

[0070] In an optional embodiment, the parallelism determination module is further specifically used for: The candidate parallelism scheme with the highest measured throughput and that satisfies the memory access constraints was selected as the final inter-query parallelism and intra-query parallelism. The memory access constraint is that the local storage space required by the candidate parallelism scheme is less than or equal to the upper limit of the local storage space capacity of a parallel computing unit.

[0071] It should be noted that the large model inference acceleration processing device provided in this application embodiment can execute the large model inference acceleration processing method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0072] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a large model inference acceleration processing method, which includes: The query sequence is divided into multiple query groups, where each query group contains multiple queries that share the same key-value cache; A parallel computing unit is allocated to each query group, and the key-value cache is preloaded in the local storage space of the parallel computing unit; In each of the parallel computing units, parallel computing resources are used to calculate the dot product of the current token query and all historical keys in parallel based on preset inter-query parallelism and intra-query parallelism to generate a similarity vector; wherein, the parallel computing is performed simultaneously within the same parallel computing unit: the calculation of different queries is processed in parallel at the same token position based on the inter-query parallelism, and the words at different positions are processed in parallel within a query based on the intra-query parallelism. The similarity vectors are normalized to output multiple attention weights that sum to 1; The multiple attention weights are summed with the value Value to obtain a weighted sum, which is then temporarily stored in the local storage space. Once all parallel computing units have completed their calculations, the weighted sums of the parallel computing units are combined to obtain the complete output result, which is then written back to the global storage space.

[0073] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the large model inference acceleration processing method provided by the above methods, the method including: The query sequence is divided into multiple query groups, where each query group contains multiple queries that share the same key-value cache; A parallel computing unit is allocated to each query group, and the key-value cache is preloaded in the local storage space of the parallel computing unit; In each of the parallel computing units, parallel computing resources are used to calculate the dot product of the current token query and all historical keys in parallel based on preset inter-query parallelism and intra-query parallelism to generate a similarity vector; wherein, the parallel computing is performed simultaneously within the same parallel computing unit: the calculation of different queries is processed in parallel at the same token position based on the inter-query parallelism, and the words at different positions are processed in parallel within a query based on the intra-query parallelism. The similarity vectors are normalized to output multiple attention weights that sum to 1; The multiple attention weights are summed with the value Value to obtain a weighted sum, which is then temporarily stored in the local storage space. Once all parallel computing units have completed their calculations, the weighted sums of the parallel computing units are combined to obtain the complete output result, which is then written back to the global storage space.

[0075] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the large model inference acceleration processing method provided by the methods described above, the method comprising: The query sequence is divided into multiple query groups, where each query group contains multiple queries that share the same key-value cache; A parallel computing unit is allocated to each query group, and the key-value cache is preloaded in the local storage space of the parallel computing unit; In each of the parallel computing units, parallel computing resources are used to calculate the dot product of the current token query and all historical keys in parallel based on preset inter-query parallelism and intra-query parallelism to generate a similarity vector; wherein, the parallel computing is performed simultaneously within the same parallel computing unit: the calculation of different queries is processed in parallel at the same token position based on the inter-query parallelism, and the words at different positions are processed in parallel within a query based on the intra-query parallelism. The similarity vectors are normalized to output multiple attention weights that sum to 1; The multiple attention weights are summed with the value Value to obtain a weighted sum, which is then temporarily stored in the local storage space. Once all parallel computing units have completed their calculations, the weighted sums of the parallel computing units are combined to obtain the complete output result, which is then written back to the global storage space.

[0076] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for accelerating large-scale model inference, characterized in that, include: The query sequence is divided into multiple query groups, where each query group contains multiple queries that share the same key-value cache; A parallel computing unit is allocated to each query group, and the key-value cache is preloaded in the local storage space of the parallel computing unit; In each of the parallel computing units, parallel computing resources are used to calculate the dot product of the current token query and all historical keys in parallel based on preset inter-query parallelism and intra-query parallelism to generate a similarity vector; wherein, the parallel computing is performed simultaneously within the same parallel computing unit: the calculation of different queries is processed in parallel at the same token position based on the inter-query parallelism, and the words at different positions are processed in parallel within a query based on the intra-query parallelism. The similarity vectors are normalized to output multiple attention weights that sum to 1; The multiple attention weights are summed with the value Value to obtain a weighted sum, which is then temporarily stored in the local storage space. Once all parallel computing units have completed their calculations, the weighted sums of the parallel computing units are combined to obtain the complete output result, which is then written back to the global storage space.

2. The method for accelerating large model inference according to claim 1, characterized in that, The inter-query parallelism and the intra-query parallelism are determined based on the following steps: Based on the range of values ​​for the inter-query parallelism and the intra-query parallelism corresponding to the inter-query parallelism, multiple parallelism candidate schemes are generated. Based on the measured throughput of each of the proposed parallelism candidate schemes, the final inter-query parallelism and intra-query parallelism are determined.

3. The method for accelerating large model inference according to claim 2, characterized in that, The intra-query parallelism corresponding to the inter-query parallelism is determined based on the following steps: The total number of thread bundles is determined based on the total number of threads on the GPU. The intra-query parallelism corresponding to the inter-query parallelism is determined based on the ratio of the total number of thread bundles to the inter-query parallelism.

4. The method for accelerating large model inference according to claim 3, characterized in that, The step of determining the intra-query parallelism corresponding to the inter-query parallelism based on the ratio of the total number of thread bundles to the inter-query parallelism includes: Based on the ratio of the total number of thread bundles to the equivalent resource units, the intra-query parallelism corresponding to the inter-query parallelism is determined. The equivalent resource unit is the product of the inter-query parallelism and the protection coefficient, whereby the protection coefficient is used to conservatively reduce the theoretical hardware resources.

5. The method for accelerating large model inference according to claim 4, characterized in that, The parallelism within the query is calculated based on the following formula: intra_i=floor(W total / (inter_i×Protection_Value)); Where inter_i is the inter-query parallelism, intra_i is the maximum value of the intra-query parallelism corresponding to inter_i, and W total It represents the total number of thread bundles, Protection_Value is the protection factor, and floor() rounds down.

6. The method for accelerating large model inference according to any one of claims 2-5, characterized in that, The determination of the final inter-query parallelism and intra-query parallelism based on the measured throughput of each of the candidate parallelism schemes includes: The candidate parallelism scheme with the highest measured throughput and that satisfies the memory access constraints was selected as the final inter-query parallelism and intra-query parallelism. The memory access constraint is that the local storage space required by the candidate parallelism scheme is less than or equal to the upper limit of the local storage space capacity of a parallel computing unit.

7. A large-scale model inference acceleration processing device, characterized in that, include: The sequence grouping module is used to divide a query sequence into multiple query groups, wherein each query group contains multiple queries that share the same key-value cache. The resource allocation module is used to: allocate a parallel computing unit for each query group and preload the key-value cache in the local storage space of the parallel computing unit; The parallel computing module is used to: in each of the parallel computing units, utilize parallel computing resources to calculate the dot product of the current token query and all historical keys in parallel based on preset inter-query parallelism and intra-query parallelism, and generate a similarity vector; wherein, the parallel computing is performed simultaneously within the same parallel computing unit: based on the inter-query parallelism, it processes the calculation of different queries in parallel at the same token position, and based on the intra-query parallelism, it processes words at different positions in parallel within a query; The normalization module is used to: normalize each of the similarity vectors and output multiple attention weights that sum to 1; The weighted summation module is used to: sum the multiple attention weights and the value Value in a weighted manner to obtain a weighted sum and temporarily store it in the local storage space; The result integration module is used to: merge the weighted sums of all parallel computing units after all parallel computing units have completed their calculations, obtain a complete output result, and write the complete output result back to the global storage space.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the large model inference acceleration processing method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the large model inference acceleration processing method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the large model inference acceleration processing method as described in any one of claims 1 to 6.

Images