LLM reasoning acceleration method and device based on RISC-V vector extension

Abstract

The invention discloses an LLM reasoning acceleration method and device based on RISC-V vector extension, and belongs to the field of large language model design. The objective of the invention is to solve the problem of performance bottleneck in a large-scale language model (LLM) reasoning process, especially deployment on edge computing equipment. The method comprises the following steps: analyzing an inference process through a performance analysis tool, identifying bottleneck operators, and reconstructing the bottleneck operators by using an RISC-V vector extension RVV instruction set to improve the calculation efficiency. And the reconstructed operator can realize parallel calculation of SIMD, and the utilization of hardware resources is maximized by dynamically adjusting the length of a vector register. The method is further based on a dynamic vector length adaptation mechanism of RVV, it is ensured that the optimized operator can be automatically adapted according to different hardware platforms, and therefore efficient migration and execution of the reasoning task among multiple platforms are achieved, and the real-time performance of LLM reasoning and the system efficiency are effectively improved.

Description

Technical Field

[0001] This application relates to the field of large language model design, and in particular to an LLM inference acceleration method and apparatus based on RISC-V vector extension. Background Technology

[0002] With the continuous advancement of natural language processing technology, especially the rapid development of large language models (LLMs), they have demonstrated strong application potential in various fields such as natural language understanding, automatic generation, machine translation, and intelligent question answering. However, as the model size continues to grow, the computational complexity, memory bandwidth requirements, and power consumption issues in the LLM inference process have gradually become the main bottlenecks restricting its application.

[0003] Currently, large-scale language model inference tasks typically rely on GPU clusters or dedicated AI accelerators (such as TPUs and NPUs) for computation. While these hardware solutions offer powerful computing capabilities, they face numerous challenges when deployed on edge devices and embedded systems. Edge devices have relatively limited computing resources, and their memory bandwidth cannot meet the massive data transfer demands of LLM inference. Furthermore, high-performance computing platforms like GPUs consume significant power, making it difficult to meet the stringent low-power requirements of edge computing devices. Therefore, how to efficiently execute LLM inference on resource-constrained edge devices has become a pressing technical challenge.

[0004] As an open-source instruction set architecture (ISA), the RISC-V architecture has gradually become a research hotspot in the field of edge computing due to its flexible modular design, scalability, and open ecosystem advantages. In particular, the RISC-V Vector Extension (RVV), by supporting Single Instruction Multiple Data (SIMD) parallelism, can provide hardware-level acceleration for computationally intensive operations such as matrix operations. However, existing LLM inference frameworks (such as llama.cpp) have not fully utilized the advantages of the RISC-V Vector Extension, still mainly relying on scalar computation patterns, resulting in performance bottlenecks in LLM inference and failing to realize the potential of the RISC-V architecture on edge computing platforms.

[0005] Therefore, developing an LLM inference acceleration method based on RISC-V vector extension can fully leverage its computational efficiency, memory bandwidth utilization, and power management capabilities on edge devices, which has significant technical and application value. Summary of the Invention

[0006] This application provides an LLM inference acceleration method and apparatus based on RISC-V vector extension. The technical solution is as follows:

[0007] On the one hand, a method for accelerating LLM inference based on RISC-V vector extension is provided. This method is used to accelerate LLM inference of large language models on the RISC-V platform. The method includes:

[0008] The inference process of the llama.cpp framework was analyzed using performance analysis tools to identify the bottleneck operators in the inference process.

[0009] The identified bottleneck operators are reconstructed using the RISC-V Vector Extension RVV instruction set, and the reconstructed bottleneck operators have the ability to perform parallel SIMD computation.

[0010] Adjust the length of the vector register used by each reconstructed bottleneck operator according to the maximum vector register length supported by the hardware platform.

[0011] The optimized bottleneck operators are executed through a pre-designed dynamic vector length initialization module, which provides an environment for each bottleneck operator to be executed on different RISC-V platforms.

[0012] Optionally, the reconstructing of the identified bottleneck operator using the RISC-V Vector Extension RVV instruction set includes:

[0013] The computational logic of the bottleneck operator is rewritten in a parallel form;

[0014] The identified bottleneck operators are processed in parallel using the RISC-V Vector Extension (RVV) instruction set. The parallel processing includes executing the vfmacc instruction for vector multiplication and addition, and using the vfredmax instruction for vector maximization.

[0015] Optionally, the bottleneck operator includes vector dot product ggml_vec_dot_f16, vector scaling ggml_vec_scale_f32, vector maximum value ggml_vec_max_f32, and root mean square normalization ggml_compute_forward_rms_norm_f32.

[0016] Optionally, the step of using the RISC-V Vector Extension RVV instruction set to perform SIMD parallel processing on the identified bottleneck operators includes:

[0017] For the ggml_vec_dot_f16 operator, the zvfh extension is used to support FP16 computation, and the vfmacc instruction is used to perform vector multiplication and addition operations. The vector multiplication and addition operations are used to realize the vectorized parallel processing of the operator.

[0018] For the ggml_vec_scale_f32 and ggml_vec_max_f32 operators, parallel processing is performed using the vfmul and vfredmax instructions. The vfmul and vfredmax instructions are used to optimize the execution logic of the bottleneck operator, reduce latency in a single calculation, and improve data processing speed.

[0019] Optionally, adjusting the length of the vector register used by each reconstructed bottleneck operator according to the maximum vector register length supported by the hardware platform includes:

[0020] The CSRR instruction retrieves the maximum vector register length (VLEN) and vector register group (LMUL) information supported by the platform.

[0021] Based on the maximum vector register length supported by the hardware platform, the vsetvl instruction is used to dynamically set the vector register length used by each bottleneck operator.

[0022] Optionally, the method further includes:

[0023] Deploying the refactored bottleneck operator across platforms;

[0024] include:

[0025] Design an RVV-based dynamic vector length adaptation mechanism to ensure that the optimized operator is adjusted according to the hardware capabilities of different RISC-V platforms;

[0026] The optimized bottleneck operator is implemented using C inline assembly or the built-in vector function form supported by RISC-VGCC.

[0027] Optionally, the method further includes an evaluation process, which includes:

[0028] The benchmark tool included in llama.cpp was used to test the processing speed of the prefill and decode stages during the inference process, and to compare the performance before and after optimization.

[0029] On the Gemma-2B model, the LLM inference acceleration method based on RISC-V vector extension was evaluated to achieve a 1.7x speedup in the prefill stage and a 53% speedup in the decoding stage. On the Llama-3.1 8B model, the speedup was 36% in the prefill stage and 19% in the decoding stage.

[0030] Optionally, the analysis of the inference process of the llama.cpp framework using performance analysis tools to identify performance bottleneck operators during inference includes:

[0031] By statistically analyzing the number of times each operator is called during the inference process in the llama.cpp framework, and combining this with computational complexity evaluation, the bottleneck operator with the highest computation time percentage was identified.

[0032] Based on the analysis results, the parallelization level of each bottleneck operator is further analyzed, and a decision is made on whether to reconstruct the current bottleneck operator into a parallel computing form.

[0033] On the other hand, a RISC-V vector extension-based LLM inference acceleration device is also provided, which is used for accelerating LLM inference of large language models on the RISC-V platform. The device includes:

[0034] The performance analysis module is used to analyze the inference process of the llama.cpp framework using performance analysis tools and identify the bottleneck operators in the inference process.

[0035] The operator reconstruction module is used to reconstruct the identified bottleneck operators using the RISC-V Vector Extension (RVV) instruction set. The reconstructed bottleneck operators have parallel computing (SIMD) capabilities.

[0036] The length adjustment module is used to adjust the length of the vector register used by each reconstructed bottleneck operator according to the maximum vector register length supported by the hardware platform.

[0037] An initialization module is used to execute the optimized bottleneck operator through a pre-designed dynamic vector length initialization module, which provides an environment for each bottleneck operator to be executed on different RISC-V platforms.

[0038] On the other hand, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which is loaded and executed by a processor to implement the LLM inference acceleration method based on RISC-V vector extension as described above.

[0039] This application provides a method and apparatus for accelerating LLM inference based on RISC-V Vector Extensions (RVV), belonging to the field of large language model design. It aims to solve the performance bottleneck problem in the LLM inference process of large language models, especially in deployment on edge computing devices. The method includes analyzing the inference process using performance analysis tools to identify bottleneck operators, and reconstructing these bottleneck operators using the RISC-V Vector Extensions (RVV) instruction set to improve computational efficiency. The reconstructed operators can achieve parallel computation of SIMD, and maximize hardware resource utilization by dynamically adjusting the vector register length. The method also leverages RVV's dynamic vector length adaptation mechanism to ensure that the optimized operators can automatically adapt to different hardware platforms, thereby achieving efficient migration and execution of inference tasks across multiple platforms, effectively improving the real-time performance and system efficiency of LLM inference. Attached Figure Description

[0040] Figure 1 A flowchart of an exemplary embodiment of this application is shown. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0042] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0043] Example 1

[0044] Please refer to Figure 1 This document illustrates a flowchart of an exemplary embodiment of an LLM inference acceleration method based on RISC-V vector extension provided in this application. The method is used to accelerate LLM inference for large language models on the RISC-V platform, and includes:

[0045] Step 101: Analyze the inference process of the llama.cpp framework using performance analysis tools to identify the bottleneck operators in the inference process.

[0046] Bottleneck operators include vector dot product (ggml_vec_dot_f16), vector scaling (ggml_vec_scale_f32), vector maximization (ggml_vec_max_f32), and root mean square normalization (ggml_compute_forward_rms_norm_f32).

[0047] Root mean square normalization (ggml_compute_forward_rms_norm_f32) is an operator used to perform root mean square normalization (RMSNorm), which is widely used in deep learning models, especially in models based on the Transformer architecture. The mathematical expression of RMSNorm is as follows (4):

[0048]

[0049] Where, x i is the i-th element of the input vector x, d is the number of elements in x, and ∈ is a very small positive number used to prevent division by zero errors.

[0050] As can be seen from the formula, the multiplication-addition operation dominates the entire calculation process. Therefore, we optimized this operator at the instruction level, utilizing the vfmacc vector instruction to achieve parallel acceleration of this calculation process. Furthermore, the scaling operation performed on the accumulated result is handled by the ggml_vec_scale_f32 operator, which we also optimized. Since this implementation relies on the implementations of vector dot product and vector scaling, which are already provided elsewhere, their specific implementations will not be repeated here.

[0051] By using performance analysis tools (such as those included in the llama.cpp framework), the inference process can be statistically analyzed in real time to identify bottleneck operators affecting inference efficiency. These operators mainly include computationally intensive operations such as vector dot product, vector scaling, and normalization operations. Identifying bottleneck operators is the foundation for subsequent optimization.

[0052] Step 102: The identified bottleneck operators are reconstructed using the RISC-V Vector Extension RVV instruction set. The reconstructed bottleneck operators have the ability to perform parallel SIMD computation.

[0053] The bottleneck operator is converted into a parallel form using the RISC-V Vector Extension (RVV) instruction set. Parallel computation is achieved through vector instructions such as vfmacc and vfredmax, significantly improving the computational power of the bottleneck operator, increasing computational throughput during inference, and reducing computational latency.

[0054] Step 103: Adjust the length of the vector register used by each reconstructed bottleneck operator according to the maximum vector register length supported by the hardware platform.

[0055] Based on the maximum vector register length (e.g., 128-bit, 256-bit, or 512-bit) obtained from the hardware platform via the CSRR instruction, the vsetvl instruction is used to dynamically adjust the vector register length used by each refactored bottleneck operator. This adjustment ensures that each operator can maximize the use of hardware resources for parallel computation, thereby further improving computational throughput.

[0056] Step 104: Execute the optimized bottleneck operator through a pre-designed dynamic vector length initialization module. The pre-designed dynamic vector length initialization module is used to provide an environment for each bottleneck operator to be executed on different RISC-V platforms.

[0057] The dynamic vector length initialization module ensures that the optimized bottleneck operator can automatically adapt to the hardware resources of different RISC-V platforms. The automated deployment of the adaptation environment ensures that the inference task can be migrated and executed efficiently across multiple platforms.

[0058] Example 2

[0059] Step 102 involves reconstructing the identified bottleneck operators using the RISC-V Vector Extension RVV instruction set, including the following:

[0060] The computational logic of the bottleneck operator is rewritten in a parallel form; the identified bottleneck operator is processed in parallel using the RISC-V Vector Extension (RVV) instruction set. The parallel processing includes executing the vfmacc instruction for vector multiplication and addition, and using the vfredmax instruction for vector maximization.

[0061] In one possible implementation, the computational efficiency of the bottleneck operator can be further improved by rewriting its computational logic into a parallel computational form and using the RISC-V Vector Extension (RVV) instruction set for SIMD parallel processing.

[0062] By executing the vfmacc instruction for vector multiplication and addition, and using the vfredmax instruction for vector maximization, the execution efficiency of the operator is optimized, enabling it to process multiple data elements in parallel.

[0063] Example 3

[0064] Adjust the length of the vector register used by each reconstructed bottleneck operator according to the maximum vector register length supported by the hardware platform, including the following:

[0065] The maximum vector register length (VLEN) and vector register grouping (LMUL) information supported by the platform are obtained through the CSRR instruction. Based on the maximum vector register length supported by the hardware platform, the vector register length used by each bottleneck operator is dynamically set using the vsetvl instruction.

[0066] In one possible implementation, the CSRR instruction is used to obtain the maximum vector register length (VLEN) and vector register grouping (LMUL) information supported by the platform, ensuring that the operator can be adjusted according to the maximum capabilities supported by the hardware.

[0067] Depending on the hardware platform, the vsetvl instruction is used to dynamically set the length of the vector register used by each operator, thereby maximizing the parallel computing capability of each operator.

[0068] Example 4

[0069] The method also includes content on cross-platform deployment of refactored bottleneck operators.

[0070] The design incorporates a dynamic vector length adaptation mechanism based on RVV to ensure that the optimized operator is adjusted according to the hardware capabilities of different RISC-V platforms. The optimized bottleneck operator is implemented using C inline assembly or built-in vector functions supported by RISC-V GCC.

[0071] In one possible implementation, a dynamic vector length adaptation mechanism based on RVV is designed to ensure that the optimized operator can be adjusted according to the hardware capabilities of different hardware platforms, enabling it to migrate and execute seamlessly across different platforms.

[0072] The implementation is carried out using C inline assembly or built-in vector functions supported by RISC-V GCC, replacing the original operator implementation to ensure consistent execution efficiency across multiple platforms.

[0073] Example 5

[0074] The reasoning acceleration effect of the method is evaluated in the following ways.

[0075] Using the benchmark tool included in llama.cpp, we tested the processing speed of the prefill and decode stages during inference and compared the results before and after optimization. On the Gemma-2B model, the LLM inference acceleration method based on RISC-V vector extension was evaluated to achieve a 1.7x speedup in the prefill stage and a 53% speedup in the decode stage. On the Llama-3.18B model, the speedup was 36% in the prefill stage and 19% in the decode stage.

[0076] In one possible implementation, the benchmark tool included in llama.cpp is used to test the processing speed of the prefill and decode stages during the inference process, and to compare the results before and after optimization.

[0077] On the Gemma-2B model, the pre-filling stage is accelerated by 1.7 times and the decoding stage by 53%; on the Llama-3.1 8B model, the pre-filling stage is accelerated by 36% and the decoding stage by 19%, demonstrating the significant inference acceleration effect of the optimization method.

[0078] Example 6

[0079] The inference process of the llama.cpp framework is analyzed using performance analysis tools to identify performance bottlenecks in the inference process. The performance analysis tools are used in the following ways.

[0080] By statistically analyzing the number of calls to each operator in the llama.cpp framework during inference and combining this with computational complexity assessment, the bottleneck operators with the highest computation time percentage were identified. Based on the analysis results, the parallelization level of each bottleneck operator was further analyzed, and a decision was made on whether to refactor the current bottleneck operator into a parallel computing form.

[0081] In one possible implementation, by counting the number of calls to each operator in the llama.cpp framework and combining this with computational complexity analysis, the bottleneck operator with the highest computation time percentage can be identified.

[0082] Further analysis of the parallelization level of each operator determines whether it needs to be reconstructed into a parallel computing form, thereby reducing computational latency and optimizing the inference process.

[0083] Example 7

[0084] The instruction-level optimization in the above embodiments includes the following: using the RISC-V vector extension RVV instruction set to perform SIMD parallel processing on the identified bottleneck operators, and also includes the following.

[0085] Content 1: For the ggml_vec_dot_f16 operator, the zvfh extension is used to support FP16 computation, and the vfmacc instruction is used to perform vector multiplication and addition operations. The vector multiplication and addition operations are used to realize the vectorized parallel processing of the operator, reduce memory access latency and improve computational throughput.

[0086] FP16 vector dot product (ggml_vec_dot_f16) suffers from high computational latency on the RISC-V platform when performed using traditional scalar computation methods, failing to fully utilize hardware parallelism. Since the base RVV does not support FP16 computation, the "zvfh" extension is used to support F16 operations. The RVV refactoring algorithm is as follows. This optimization significantly reduces memory accesses and improves instruction-level parallelism.

[0087]

[0088]

[0089] Content 2: For the ggml_vec_scale_f32 and ggml_vec_max_f32 operators, the vfmul and vfredmax instructions are used for parallel processing. The vfmul and vfredmax instructions are used to optimize the execution logic of the bottleneck operator, reduce the latency in a single calculation, and improve the data processing speed.

[0090] Vector scaling (ggml_vec_scale_f32) is an operator used to multiply each element of an FP32 vector by the same scaling factor, commonly used in RMSNorm and Softmax modules. The RVV reconstruction algorithm is as follows. Significant speedup is achieved through instruction-level parallelization.

[0091]

[0092] The vector maximum value (ggml_vec_max_f32) is used to calculate the maximum value before Softmax, improving numerical stability. RVV optimization uses vfredmax_vs_f32m2_f32m1 to implement piecewise reduction. The pseudocode is as follows:

[0093]

[0094] In one possible implementation, for the ggml_vec_dot_f16 operator, the zvfh extension is used to support FP16 computation, and the vfmacc instruction is used to perform vector multiplication and addition operations, thereby realizing vectorized parallel processing of the operator, significantly reducing memory access latency and improving computational throughput.

[0095] For the ggml_vec_scale_f32 and ggml_vec_max_f32 operators, parallel processing is performed using instructions such as vfmul and vfredmax to reduce computational latency and improve data processing speed.

[0096] On the other hand, a RISC-V vector extension-based LLM inference acceleration device is also provided for accelerating large-scale language model LLM inference on the RISC-V platform. The device includes:

[0097] The performance analysis module is used to analyze the inference process of the llama.cpp framework using performance analysis tools and identify the bottleneck operators in the inference process.

[0098] The operator reconstruction module is used to reconstruct the identified bottleneck operators using the RISC-V Vector Extension (RVV) instruction set. The reconstructed bottleneck operators have parallel computing (SIMD) capabilities.

[0099] The length adjustment module is used to adjust the length of the vector register used by each reconstructed bottleneck operator according to the maximum vector register length supported by the hardware platform.

[0100] An initialization module is used to execute the optimized bottleneck operator through a pre-designed dynamic vector length initialization module, which provides an environment for each bottleneck operator to be executed on different RISC-V platforms.

[0101] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the LLM inference acceleration method based on RISC-V vector extension as provided in the above embodiments.

[0102] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0103] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0104] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0105] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for accelerating LLM inference based on RISC-V vector extension, characterized in that, The method is used to accelerate large-scale language model LLM inference on the RISC-V platform. The method includes: The inference process of the llama.cpp framework was analyzed using performance analysis tools to identify the bottleneck operators in the inference process. The identified bottleneck operators are reconstructed using the RISC-V Vector Extension RVV instruction set, and the reconstructed bottleneck operators have the ability to perform parallel SIMD computation. Adjust the length of the vector register used by each reconstructed bottleneck operator according to the maximum vector register length supported by the hardware platform. The optimized bottleneck operators are executed through a pre-designed dynamic vector length initialization module, which provides an environment for each bottleneck operator to be executed on different RISC-V platforms.

2. The method according to claim 1, characterized in that, The process of reconstructing the identified bottleneck operators using the RISC-V Vector Extension RVV instruction set includes: The computational logic of the bottleneck operator is rewritten in a parallel form; The identified bottleneck operators are processed in parallel using the RISC-V Vector Extension (RVV) instruction set. The parallel processing includes executing the vfmacc instruction for vector multiplication and addition, and using the vfredmax instruction for vector maximization.

3. The method according to claim 2, characterized in that, The bottleneck operators include vector dot product ggml_vec_dot_f16, vector scaling ggml_vec_scale_f32, vector maximum value ggml_vec_max_f32, and root mean square normalization ggml_compute_forward_rms_norm_f32.

4. The method according to claim 3, characterized in that, The step of using the RISC-V Vector Extension RVV instruction set to perform SIMD parallel processing on the identified bottleneck operators includes: For the ggml_vec_dot_f16 operator, the zvfh extension is used to support FP16 computation, and the vfmacc instruction is used to perform vector multiplication and addition operations. The vector multiplication and addition operations are used to realize the vectorized parallel processing of the operator. For the ggml_vec_scale_f32 and ggml_vec_max_f32 operators, parallel processing is performed using the vfmul and vfredmax instructions. The vfmul and vfredmax instructions are used to optimize the execution logic of the bottleneck operator, reduce latency in a single calculation, and improve data processing speed.

5. The method according to claim 1, characterized in that, The step of adjusting the vector register length used by each reconstructed bottleneck operator according to the maximum vector register length supported by the hardware platform includes: The CSRR instruction retrieves the maximum vector register length (VLEN) and vector register group (LMUL) information supported by the platform. Based on the maximum vector register length supported by the hardware platform, the vsetvl instruction is used to dynamically set the vector register length used by each bottleneck operator.

6. The method according to claim 1, characterized in that, The method also includes cross-platform deployment of the reconstructed bottleneck operator; The restructured bottleneck operator deployed across platforms includes: Design an RVV-based dynamic vector length adaptation mechanism to ensure that the optimized operator is adjusted according to the hardware capabilities of different RISC-V platforms; The optimized bottleneck operator is implemented using C inline assembly or the built-in vector function form supported by RISC-VGCC.

7. The method according to claim 1, characterized in that, The method further includes an evaluation process, which includes: The benchmark tool included in llama.cpp was used to test the processing speed of the prefill and decode stages during the inference process, and to compare the performance before and after optimization. On the Gemma-2B model, the LLM inference acceleration method based on RISC-V vector extension was evaluated to achieve a 1.7x speedup in the prefill stage and a 53% speedup in the decoding stage. On the Llama-3.18B model, the speedup was 36% in the prefill stage and 19% in the decoding stage.

8. The method according to claim 1, characterized in that, The analysis of the inference process of the llama.cpp framework using performance analysis tools to identify performance bottleneck operators during inference includes: By statistically analyzing the number of times each operator is called during the inference process in the llama.cpp framework, and combining this with computational complexity evaluation, the bottleneck operator with the highest computation time percentage was identified. Based on the analysis results, the parallelization level of each bottleneck operator is further analyzed, and a decision is made on whether to reconstruct the current bottleneck operator into a parallel computing form.

9. An LLM inference acceleration device based on RISC-V vector extension, characterized in that, The device for accelerating large-scale language model LLM inference on the RISC-V platform includes: The performance analysis module is used to analyze the inference process of the llama.cpp framework using performance analysis tools and identify the bottleneck operators in the inference process. The operator reconstruction module is used to reconstruct the identified bottleneck operators using the RISC-V Vector Extension (RVV) instruction set. The reconstructed bottleneck operators have parallel computing (SIMD) capabilities. The length adjustment module is used to adjust the length of the vector register used by each reconstructed bottleneck operator according to the maximum vector register length supported by the hardware platform. An initialization module is used to execute the optimized bottleneck operator through a pre-designed dynamic vector length initialization module, which provides an environment for each bottleneck operator to be executed on different RISC-V platforms.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the LLM inference acceleration method based on RISC-V vector extension as described in any one of claims 1 to 8.

Images