Hybrid precision matrix multiplication acceleration method and system
By constructing a policy dispatch graph and implementing online adaptive scheduling, the optimal computation strategy is dynamically selected, which solves the problems of low hardware resource utilization and low inference efficiency caused by computational heterogeneity in LLM inference, and achieves more efficient inference performance.
Patent Information
- Application Number
- CN202511720914.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-02-24
AI Technical Summary
Existing technologies employ static, workload-agnostic computational strategies during Large Language Model (LLM) inference, which cannot adapt to computational heterogeneity, resulting in low hardware resource utilization and suboptimal inference efficiency.
The optimal computation strategy is dynamically selected by constructing an offline strategy graph generation phase and an online adaptive scheduling phase. The offline phase constructs a strategy dispatch graph to record the mapping relationship between workload and the best-performing strategy; the online phase uses a lightweight dispatcher to quickly find and execute the optimal strategy during inference.
It significantly reduces the latency of the pre-filling stage, increases the throughput of the decoding stage, improves the inference efficiency of arbitrary precision quantized large language models, and improves hardware utilization.
Smart Images
Figure CN121560486A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computing and artificial intelligence acceleration technology, specifically to a mixed-precision matrix multiplication acceleration method and system, and more particularly to a mixed-precision matrix multiplication acceleration method and system for arbitrary-precision quantization models. Background Technology
[0002] In recent years, Large Language Models (LLMs) have been widely adopted. Driven by considerations of privacy and usability, LLMs have seen a trend towards deployment on edge devices. However, their enormous computational and storage requirements pose significant challenges to edge deployment. To address this issue, Arbitrary-Precision Quantization (APQ) technology has emerged. APQ quantizes the model's weights and activations using different bit widths (e.g., 4-bit weights and 8-bit activations, i.e., W4A8), achieving an excellent balance between model compression efficiency and inference accuracy. Models quantized using APQ technology are called APQ LLMs.
[0003] However, APQ technology introduces a core computational challenge: mixed-precision general matrix multiplication (mpGEMM), where the two operand matrices involved in the matrix multiplication operation have different data bit widths. Current mainstream hardware acceleration units, such as Tensor Cores in GPUs, are typically designed to perform symmetric precision operations (e.g., INT8×INT8), and cannot directly support asymmetric precision mpGEMM operations.
[0004] Therefore, software adaptation is necessary for mpGEMM to utilize the accelerated computing units on mainstream hardware. Existing solutions in academia and industry mainly include padding and bit-disaggregation. These solutions primarily employ static strategies for software adaptation, such as padding (boosting low-width data to a higher bit width to match the hardware) or bit-disaggregation (decomposing high-width data into multiple 1-bit matrices for computation). The core problem with these existing technologies is that they are all based on a static, workload-unaware computation paradigm. Regardless of whether padding or bit-disaggregation is chosen, the same strategy is applied uniformly throughout the entire LLM inference process.
[0005] However, in-depth analysis reveals that the inference process of LLM exhibits significant computational heterogeneity, making it impossible for any single static strategy to achieve optimal performance in all scenarios. This computational heterogeneity manifests in two ways: (1) LLMs with different quantization precipitates have different mpGEMM operand bit widths; and (2) the problem size of matrix multiplication in LLM inference varies drastically, especially in the first dimension M. Specifically, (1) is caused by the different quantization precipitates, while (2) is mainly due to the different computational requirements of the prefill and decoding stages, the different prompt lengths and batch sizes for different requests, and the different computational requirements of the attention layer and feed-forward layer.
[0006] Therefore, the existing technology has a fundamental flaw: it adopts a static, workload-agnostic mpGEMM algorithm, which cannot adapt to this computational heterogeneity. No single static strategy can achieve optimal performance under all these changing scenarios, resulting in low hardware resource utilization and suboptimal inference efficiency.
[0007] In summary, the current market needs a method that can fundamentally address the limitations of the static computing paradigm. This method should employ an adaptive mechanism to dynamically select and execute the optimal computing strategy based on the characteristics of each specific mpGEMM task encountered during LLM inference with arbitrary precision quantization, thereby achieving better inference performance. Summary of the Invention
[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a mixed-precision matrix multiplication acceleration method and system that solves the suboptimal performance problem caused by using a static, workload-agnostic computational strategy to perform mixed-precision matrix multiplication (mpGEMM) in arbitrary-precision quantization (APQ) LLM.
[0009] A mixed-precision matrix multiplication acceleration method provided by the present invention includes: Offline policy graph generation phase: a. Constructing a diverse set of computational policies: Systematically construct a combination of various highly optimized computational policies. b. Exhaustive evaluation: For a specific target LLM and target hardware platform, scan all possible mpGEMM task types encountered during its inference process (defined by the problem size M, N, K of matrix multiplication and the bit width of the operands), forming a complete workload space. At each point in this space, perform precise performance benchmarking on each policy in the policy set. c. Generating a policy dispatch graph: Save the performance evaluation results as an efficient lookup table or decision graph, i.e., a "policy dispatch graph". This graph stores the mapping relationship from any workload point (defined by task characteristics) to its empirically proven optimal computational policy (or corresponding computational kernel).
[0010] Online Adaptive Scheduling Phase: a. Policy Dispatcher: A pre-generated policy dispatch graph is embedded into a lightweight dispatcher, which is integrated into the LLM inference engine. b. Runtime Optimal Policy Selection and Execution: During LLM inference, whenever an mpGEMM task is encountered, the dispatcher extracts the task's features (problem size, bit width) and uses these as keys to perform a fast lookup in the policy graph. c. Execution: The lookup operation directly returns a function pointer or identifier pointing to the optimal computation kernel, and the dispatcher then calls that kernel to perform the computation. Since the lookup process is only a simple table lookup operation, its runtime overhead is negligible.
[0011] Compared with the prior art, the present invention has the following beneficial effects: 1. Compared with existing techniques that employ a single static strategy, this invention effectively overcomes computational heterogeneity in the LLM inference process by dynamically matching and executing the currently optimal computational strategy for each mixed-precision matrix multiplication task. This significantly reduces latency in the pre-filling stage and greatly increases throughput in the decoding stage, thereby improving the overall inference efficiency of arbitrary-precision quantized large language models.
[0012] 2. The online adaptive scheduling process of this invention is mainly a single efficient search operation, which reduces the time cost compared to the entire computing task, achieving high-performance adaptive scheduling without introducing additional overhead.
[0013] 3. By dynamically selecting the most suitable computing strategy for the current task, this invention can fully and flexibly utilize computing units of different types and granularities in the hardware (such as NVIDIA GPUs), avoiding the situation where some hardware resources are idle or used inefficiently under a single strategy, and improving the overall utilization rate of the hardware. Attached Figure Description
[0014] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a system block diagram of the present invention; Figure 2 This is a schematic diagram of the calculation strategy set in an embodiment of the present invention. Detailed Implementation
[0015] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0016] This invention matches the optimal computational strategy for each specific mixed-precision matrix multiplication task through an offline policy graph generation stage and an online adaptive scheduling stage. This solves the problem of low inference efficiency caused by the static and computationally insensitive approach in existing systems, and better meets the application's demand for efficient model inference.
[0017] An adaptive acceleration method for mixed-precision matrix multiplication provided by the present invention is applied to a target neural network model on a target hardware architecture. The method includes: utilizing a pre-constructed policy dispatch graph, which stores the correlation between workload characteristics representing computational tasks and optimal computational strategies evaluated based on preset performance indicators. The method further includes: upon receiving a computational task, a policy dispatcher determines the workload characteristics of the computational task; using the workload characteristics, the policy dispatch graph searches for and invokes the corresponding optimal computational strategy to execute the computational task.
[0018] Offline policy graph generation phase: For mixed-precision matrix multiplication tasks executed on the target hardware architecture, a predefined set of computational strategies containing multiple dissimilar computational strategies is established. For the target hardware platform and target neural network model, a workload space is defined, comprising the problem size (M, N, K) of matrix multiplication and the operand bit width. Within this workload space, the performance of all computational strategies is exhaustively evaluated for each workload point, generating empirical performance data. The search space for this exhaustive evaluation is finite. Considering typical edge-side inference scenarios with batch combinations of 1–8 and cue word lengths of 1–2048, the mixed-precision matrix multiplication problem size occurring in model inference does not exceed 60,000, making the computational load of offline evaluation controllable. Based on the empirical performance data, a policy dispatch graph is constructed. This graph stores the association between each workload point and the optimal computational strategy at that point and is embedded in a policy dispatcher.
[0019] The strategy dispatch graph is a key-value storage structure, where the keys are tuples representing workload characteristics, and the values are function pointers or identifiers pointing to the optimal computational kernel. Assuming there are 60,000 possible mixed-precision matrix multiplication problems, each problem size stores a 64-bit function pointer, occupying approximately 458KB of space, and the lookup overhead is only a single lookup table operation. Each computational strategy in the set invokes different accelerated computing units on the target hardware platform, possessing distinct computational characteristics such as latency-oriented, throughput-oriented, or balanced. These diverse computational strategies correspond to mapping the task to different types or granularities of computational instructions supported by the hardware. The target neural network model is an arbitrary-precision quantized large language model, where its weights and activations are quantized to different integer bit widths.
[0020] The exhaustive evaluation involves scanning all possible mpGEMM task types (defined by the problem size M, N, K of matrix multiplication and the bit width of operands) encountered during the inference process of a specific target LLM and target hardware platform, forming a complete workload space. At each point in this space, precise performance benchmarking is performed on each policy in the policy set.
[0021] In the online adaptive scheduling phase: the policy dispatcher adaptively selects and executes the optimal computational strategy based on the characteristics of the received mixed-precision matrix multiplication task, achieving adaptive hardware acceleration for the inference process of arbitrary-precision quantized large language models. Specifically, the policy dispatcher receives a request to execute a specific mixed-precision matrix multiplication task and determines the characteristics of the specific mixed-precision matrix multiplication task, which defines a workload point; the policy dispatcher uses the determined task characteristics as a key to search in the pre-constructed policy graph to retrieve the corresponding optimal computational strategy; the policy dispatcher calls and uses the optimal computational strategy to execute the specific mixed-precision matrix multiplication task.
[0022] The various computational strategies include at least: padding strategies, bit decomposition strategies, and partitioning strategies. A throughput-oriented padding strategy involves increasing the data bit width of lower-width operands to match that of higher-width operands to utilize symmetric precision computation units for computation. A latency-oriented bit decomposition strategy involves decomposing operand matrices of different bit widths into multiple unit-width matrices, performing multiple unit-width matrix multiplications, and then reconstructing the result. A balanced partitioning strategy involves decomposing higher-width operand matrices into multiple lower-width matrices, performing multiple symmetric lower-width matrix multiplications with the original lower-width operand matrices, and then reconstructing the result.
[0023] The two-stage adaptive computation mapping framework of this invention abandons the traditional approach of using static strategies. Instead, it combines offline exhaustive analysis and online lightweight scheduling to match the optimal execution strategy for each specific computation task.
[0024] The core problem addressed by this invention is how to effectively address the various computational heterogeneities in the Large Language Model (LLM) inference process (heterogeneous mpGEMM load caused by different computational characteristics between pre-filling and decoding stages, within stages, and with different quantization bit widths), thereby maximizing the execution efficiency of mpGEMM and improving the inference efficiency of APQ-LLM.
[0025] Furthermore, in conjunction with the appendix Figures 1 to 2 The implementation process of this invention is described in detail below: Offline policy graph generation phase. This phase is completed once before model deployment, and its core is to generate a policy dispatch graph for use by the policy dispatcher in the online phase.
[0026] First, the system needs to predefine a set of computational strategies. This set contains various dissimilar computational strategies with different computational characteristics for the mpGEMM task. According to claim 2, the essential difference between these strategies lies in how they map the same logical computational task to different types or granularities of computational instructions supported by the hardware. In a preferred embodiment, the strategy set includes at least the following three strategies, such as... Figure 2 As shown (using a W4A8, i.e., 4-bit weights and 8-bit activation, mpGEMM task as an example): a. Filling strategy: such as Figure 2 As shown above, this is a throughput-oriented strategy. The input 4-bit weight matrix is decomposed (in this case, bit-width boosting) into an 8-bit weight tensor to match the 8-bit activation tensor. These two 8-bit tensors are then fed into the INT8 Tensor Core for a single, high-throughput matrix multiplication operation, directly yielding a 32-bit integer partial sum, which forms the final computation result. This strategy offers high concurrency due to less shared memory usage, making it suitable for computationally intensive tasks with saturated computing resources.
[0027] b. Bit decomposition strategy: such as Figure 2As shown in the middle, this is a latency-oriented strategy. The original 4-bit weight matrix and 8-bit activation matrix are decomposed into multiple 1-bit weight tensors and 1-bit activation tensors, respectively. These 1-bit tensors are fed into the INT1 tensor core for multiple low-latency matrix multiplications. The resulting multiple 32-bit integer parts require a weighted summation reconstruction process to form the final computation result. This strategy can achieve lower latency when the matrix size is small by leveraging the low-latency characteristics of the INT1 tensor core.
[0028] c. Segmentation strategy: such as Figure 2 The lower section illustrates a balanced strategy. It employs an intermediate-granularity decomposition method. In this example, the 4-bit weight matrix remains unchanged, while the 8-bit activation matrix is decomposed into two 4-bit activation tensors. Subsequently, the 4-bit weight tensors and these two 4-bit activation tensors are fed into the INT4 tensor core for computation in two separate steps. Similarly, the resulting multiple 32-bit integer partial sums also undergo a weighted summation reconstruction process to form the final computation result. This strategy strikes a balance between throughput and latency, achieving optimal performance under certain specific workloads.
[0029] To ensure the high performance of these strategies, various optimization techniques can be employed at the kernel level. For example, when implementing strategies involving decomposing high-bit matrices into low-bit matrices, a hardware-aware boosting principle can be followed. This means that after logical decomposition, the bit width of all logical partition matrices is uniformly boosted to a target bit width, which is the minimum bit width natively supported by the hardware architecture and not less than the maximum bit width of all logical partition matrices. This ensures that subsequent computations can utilize homogeneous hardware acceleration units. Furthermore, to reduce costly global memory reads and writes, the final step of the computation process (such as weighted summation) can be merged with subsequent data processing steps (such as dequantization) and executed in the same computation kernel, avoiding the writing of high-precision intermediate accumulation results back to global memory.
[0030] After acquiring the computational strategy set, the system performs an offline, exhaustive performance evaluation for a target hardware platform and a target APQ LLM. This process scans all workload points that the target LLM might encounter during inference (defined by the problem size M, N, K of matrix multiplication and the operand bit width), and performs precise performance benchmarking on each strategy in the computational strategy set, recording its execution latency. Finally, based on the empirical data from the performance evaluation, the system constructs a strategy dispatch graph. This graph is essentially an efficient key-value store structure that accurately records the mapping relationship between each workload point and its optimal computational strategy at that point. This strategy dispatch graph is ultimately embedded into the system's strategy dispatcher for use in the online phase.
[0031] 2. Online Adaptive Scheduling Phase. This phase executes during actual model inference and its core is lightweight, zero-overhead decision-making. When the LLM inference engine needs to execute an mpGEMM computation task, the request is received by the policy dispatcher. The policy dispatcher first determines the characteristics of the task (i.e., problem size M, N, K and bit width), and then uses these characteristics as keys to perform a fast policy selection (lookup operation) in the embedded policy dispatch graph. This lookup operation directly returns an identifier or function pointer to the optimal computation policy kernel. The policy dispatcher then calls the optimal policy kernel to perform the computation and returns the result to the inference engine.
[0032] In summary, this embodiment generates a policy dispatch graph containing optimal decisions offline and performs fast searching and scheduling online using a lightweight policy dispatcher. This successfully matches the optimal computational policy for each mpGEMM task in APQ LLM inference, thereby solving the suboptimal performance problem caused by the use of static policies in the prior art and achieving a significant inference acceleration effect.
[0033] To verify the practical effectiveness of the method and system described in this invention, an end-to-end performance comparison test was conducted on an experimental platform equipped with NVIDIA Jetson AGXOrin (a typical edge computing GPU platform) to compare the invention (hereinafter referred to as ADAngel) with two benchmark systems representing existing technologies. The two benchmark systems were: 1) a highly optimized system based on bit-disaggregation (hereinafter referred to as ABQ-LLM); and 2) a highly optimized system based on padding (hereinafter referred to as llama.cpp). The test model used was the mainstream large language model Llama3-8B, quantized to W4A8 precision.
[0034] The performance of the pre-filling stage is measured by the "Time-To-First-Token" (TTFT), with shorter times being better. Table 1 shows a comparison of the TTFT of the method and system described in this invention with existing highly optimized systems under different prompt word lengths on a Llama3-8B with W4A8 accuracy and a batch size of 2. Experimental results show that the system of this invention significantly outperforms the two benchmark systems under all test models and workloads.
[0035] Table 1. Test results of the first token generation time of the present invention (unit: ms)
[0036] The performance of the decoding stage is measured by "Tokens Per Second" (TPS), with higher values being better. Table 2 shows a comparison of the TPS of the method and system described in this invention with existing highly optimized systems at different batch sizes on a Llama3-8B with W4A8 precision and a prompt word length of 128.
[0037] Table 2 Test results of the number of tokens generated per second in this invention (unit: tokens / second)
[0038] In addition, the latency of strategy dispatch in the entire inference process was measured in the scenarios described in Tables 1 and 2. The results show that the latency of strategy dispatch accounts for less than 0.1% of the total inference time.
[0039] Experimental results show that the hybrid precision matrix multiplication acceleration method and system disclosed in this invention for arbitrary precision quantized models achieves significant performance improvements in end-to-end large language model inference performance during both the pre-filling and decoding stages when handling different workloads. Specifically, for arbitrary precision quantized large language models, it achieves a latency reduction of 2.62 to 28.44 times compared to existing static policy systems in the pre-filling stage, and a throughput improvement of up to 3.38 times in the decoding stage. Furthermore, when implementing online adaptive scheduling, the overhead of the policy dispatcher in this invention accounts for only 0.01% of the total inference time, which is negligible. Therefore, the hybrid precision matrix multiplication acceleration method and system disclosed in this invention successfully solves the problem of high inference latency and low hardware utilization caused by existing systems' static, workload-agnostic computational strategies, which cannot adapt to the widespread computational heterogeneity in large language model inference. This invention can provide low-latency, high-throughput inference acceleration services for large-scale language models with arbitrary precision quantization, possessing significant innovation and practical application value.
[0040] The present invention also provides an adaptive acceleration system for mixed-precision matrix multiplication. The adaptive acceleration system for mixed-precision matrix multiplication can be implemented by executing the process steps of the adaptive acceleration method for mixed-precision matrix multiplication. That is, those skilled in the art can understand the adaptive acceleration method for mixed-precision matrix multiplication as a preferred embodiment of the adaptive acceleration system for mixed-precision matrix multiplication.
[0041] An adaptive acceleration system for mixed-precision matrix multiplication according to the present invention includes a processor and a memory. The memory stores a policy dispatch graph and a set of executable computational kernels. Each computational kernel corresponds to a predefined, disparate computational strategy. The policy dispatch graph stores the association between workload characteristics and optimal computational kernels. The processor is configured to perform the following operations: upon receiving a mixed-precision matrix multiplication task, determine the workload characteristics of the task; and use the workload characteristics to find the corresponding optimal computational strategy in the policy dispatch graph, and invoke the computational kernel corresponding to the optimal computational strategy to execute the mixed-precision matrix multiplication task. The policy dispatch graph is a key-value storage structure, where the key is a tuple of workload characteristics and the value is a function pointer or identifier pointing to the optimal computational kernel.
[0042] The various disparate computation strategies correspond to mapping the task to different types or granularities of computation instructions supported by the hardware, and the target hardware platform can accelerate matrix multiplication with at least two bit widths. The target neural network model is a large language model with arbitrary precision quantization, where weights and activations are quantized to different integer bit widths. The various disparate computation strategies include at least: padding strategies, bit decomposition strategies, and segmentation strategies. A throughput-oriented padding strategy involves increasing the data bit width of low-bit-width operands to match that of high-bit-width operands to utilize symmetric precision computation units for computation. A latency-oriented bit decomposition strategy involves decomposing operand matrices of different bit widths into multiple unit-width matrices, performing multiple unit-width matrix multiplications, and then reconstructing the result. A balanced segmentation strategy involves decomposing higher-bit-width operand matrices into multiple lower-bit-width matrices, performing multiple symmetric lower-bit-width matrix multiplications with the original lower-bit-width operand matrices, and then reconstructing the result.
[0043] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0044] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. An adaptive acceleration method for mixed-precision matrix multiplication, characterized in that, The method utilizes a pre-constructed policy dispatch graph, which stores the correlation between workload characteristics representing computational tasks and optimal computational strategies evaluated based on preset performance metrics. Upon receiving a computing task, the policy dispatcher determines the workload characteristics of the computing task. The corresponding optimal computing strategy is found and invoked in the strategy dispatch graph using the workload characteristics to execute the computing task.
2. The adaptive acceleration method for mixed-precision matrix multiplication according to claim 1, characterized in that, The computational task is a mixed-precision matrix multiplication task in a large language model with arbitrary precision quantization.
3. The adaptive acceleration method for mixed-precision matrix multiplication according to claim 1, characterized in that, The workload characteristics include: problem size, characterized by matrix dimensions M, N, and K, and operand bit width.
4. The adaptive acceleration method for mixed-precision matrix multiplication according to any one of claims 1 to 3, characterized in that, The strategy dispatch graph is a key-value storage structure; The key of the key-value storage structure is the workload characteristic, and the value is a function pointer or identifier pointing to the computing kernel used to implement the optimal computing strategy.
5. The adaptive acceleration method for mixed-precision matrix multiplication according to claim 1, characterized in that, When constructing the strategy dispatch graph, a set of computational strategies is first constructed, which includes a variety of different computational strategies; The various different computational strategies include: filling strategy, bit decomposition strategy, and partitioning strategy; The padding strategy includes increasing the data bit width of the low-width operand to match that of the high-width operand, so as to perform calculations using a symmetric precision computing unit; The bit decomposition strategy includes decomposing operand matrices of different bit widths into multiple unit-width matrices, performing multiple unit-width matrix multiplications, and then reconstructing the result. The segmentation strategy includes decomposing the higher bit-width operand matrix into multiple lower bit-width matrices, performing multiple symmetric lower bit-width matrix multiplications with the original lower bit-width operand matrix, and then reconstructing the result.
6. An adaptive acceleration system for mixed-precision matrix multiplication, comprising a processor and a memory, characterized in that, The memory stores the computational kernel and the policy dispatch graph; Each computing kernel corresponds to a predefined, dissimilar computing strategy; The strategy dispatch graph stores the correlation between the workload characteristics used to characterize computing tasks and the optimal computing strategy evaluated based on preset performance indicators; The processor is configured to execute a policy dispatcher to: Upon receiving a computing task, determine the workload characteristics of the computing task; The optimal computing strategy is found in the mapping data structure using the workload characteristics, and the computing kernel corresponding to the optimal computing strategy is called to execute the computing task.
7. The adaptive acceleration system for mixed-precision matrix multiplication according to claim 6, characterized in that, The computational task is a mixed-precision matrix multiplication task in a large language model with arbitrary precision quantization.
8. The adaptive acceleration system for mixed-precision matrix multiplication according to claim 6, characterized in that, The workload characteristics include: problem size, characterized by matrix dimensions M, N, and K, and operand bit width.
9. The adaptive acceleration system for mixed-precision matrix multiplication according to any one of claims 6 to 8, characterized in that, The strategy dispatch graph is a key-value storage structure; The key of the key-value storage structure is the workload characteristic, and the value is a function pointer or identifier pointing to the computing kernel used to implement the optimal computing strategy.
10. The adaptive acceleration system for mixed-precision matrix multiplication according to any one of claims 6, characterized in that, When constructing the strategy dispatch graph, a set of computational strategies is first constructed, which includes a variety of different computational strategies; The various different computational strategies include: filling strategy, bit decomposition strategy, and partitioning strategy; The padding strategy includes increasing the data bit width of the low-width operand to match that of the high-width operand, so as to perform calculations using a symmetric precision computing unit; The bit decomposition strategy includes decomposing operand matrices of different bit widths into multiple unit-width matrices, performing multiple unit-width matrix multiplications, and then reconstructing the result. The segmentation strategy includes decomposing the higher bit-width operand matrix into multiple lower bit-width matrices, performing multiple symmetric lower bit-width matrix multiplications with the original lower bit-width operand matrix, and then reconstructing the result.