An inference operator optimization method based on RVV and IME extended instruction set

Abstract

The application discloses a reasoning operator optimization method based on RVV and IME extended instruction set, comprising: performing operator-level performance profiling on the reasoning process of a target large language model on a RISC-V processor, and identifying a bottleneck operator; performing RVV vectorization reconstruction on the bottleneck operator; in the pre-filling stage, for a matrix multiplication operator with arithmetic intensity higher than a preset threshold, a special microkernel is constructed by using an IME matrix extension instruction set, input data is rearranged into a two-dimensional block according to the IME specification, and then bound to a vector register, and a dot product matrix multiplication and addition instruction is executed; for the bottleneck operator executed by using the RVV instruction set, an optimal configuration is selected from candidate logical vector register grouping multiples according to hardware architecture parameters and operator code characteristics; and the optimized bottleneck operator is integrated into a target reasoning engine to replace the original operator to perform a reasoning task. The application realizes the improvement of large model reasoning efficiency on a heterogeneous RISC-V processor.

Description

Technical Field

[0001] This invention relates to the optimization of the operation of large language models in artificial intelligence on the edge RISC-V platform, specifically to an optimization method for inference operators based on RVV and IME extended instruction sets. Background Technology

[0002] With the increasing demand for deploying large language models in edge computing and edge devices, building an independent and controllable AI computing power foundation based on the open-source instruction set architecture RISC-V has become an important technological path. RISC-V processors, with their modular design and high customizability, combined with Vector Extensions (RVV) and the emerging Matrix Extensions (IME), offer hardware-level acceleration potential for dense operators such as matrix multiplication and attention mechanisms in deep learning. However, due to the rapidly evolving RISC-V hardware and software ecosystem, efficiently migrating large model inference to this platform still faces significant system-level optimization challenges.

[0003] In existing technologies, large model inference engines typically rely on the automatic vectorization capabilities of general-purpose compilers (such as GCC and Clang / LLVM) to utilize the RVV instruction set. Evaluation studies show that automatic vectorization falls far short of expectations when handling computational patterns in large model inference, such as mixed-precision conversions, lookup table indirect addressing, and complex circular dependencies. For example, for INT8 quantization operators and floating-point precision conversion operators, compilers often fail to recognize their inherent data parallelism, even degenerating into scalar execution, resulting in significant additional migration latency. Simultaneously, overly aggressive loop unrolling (such as at the O3 optimization level) often leads to performance degradation by violating instruction cache locality, preventing theoretical parallelism gains from translating into practical speedups. This performance boundary of automatic vectorization severely limits the end-to-end throughput of large model inference on the RISC-V platform.

[0004] On the other hand, the RISC-V hardware ecosystem exhibits significant fragmentation, with different vendors showing substantial differences in Vector Register Length (VLEN), pipeline depth, cache hierarchy, and support for matrix extensions. While RVV's unique Logical Vector Register Grouping Multiple (LMUL) mechanism can improve single-instruction data throughput, excessively large LMULs proportionally reduce the number of available logical registers, easily leading to overflows in complex operators with high register pressure, resulting in frequent memory access penalties. Furthermore, IME matrix extensions have strict requirements on data arrangement and calling conventions, and existing compilers cannot automatically generate adapted microkernel code. This contradiction between hardware differences and lagging software adaptation makes the performance of the same inference code highly unstable on different RISC-V devices, hindering the full utilization of the hardware's theoretical computing power.

[0005] Furthermore, the pre-filling and decoding stages in large model inference exhibit distinctly different computational and memory access characteristics—the pre-filling stage primarily utilizes computationally intensive operators such as matrix multiplication, while the decoding stage is dominated by matrix-vector multiplication and frequent KVCache access. Existing optimization solutions often employ a single strategy to address the needs of both stages, lacking differentiated instruction set adaptations for bottleneck operators. This results in the inability to fully leverage the high-throughput matrix multiplication and addition capabilities of the IME during the pre-filling stage, while the decoding stage is constrained by memory bandwidth due to the lack of memory access optimization for the RVV kernel.

[0006] Therefore, there is an urgent need for a method that can systematically identify inference bottlenecks and flexibly apply RVV and IME instruction sets for deep optimization based on different operator characteristics, so as to break through the performance limit of the current compiler's automatic vectorization and achieve efficient large model inference on the RISC-V heterogeneous hardware platform. Summary of the Invention

[0007] This invention aims to provide an inference operator optimization method based on RVV and IME extended instruction set. By reconstructing the bottleneck operators in the large language model inference process using RVV vectorization, constructing an IME-specific microkernel for high arithmetic strength matrix multiplication operators in the pre-filling stage, and combining hardware-aware LMUL optimal configuration selection, this method overcomes the performance limitations of automatic vectorization by general compilers, thereby improving the inference efficiency of large models on heterogeneous RISC-V processors.

[0008] Based on the first main aspect of the present invention, an inference operator optimization method based on RVV and IME extended instruction sets is provided, the method comprising:

[0009] Operator-level performance analysis was performed on the inference process of the target large language model on the RISC-V processor, and operators whose execution time ratio exceeded a preset threshold were identified as bottleneck operators.

[0010] The bottleneck operators are reconstructed using RVV vectorization: for matrix multiplication and linear projection operators, a general matrix multiplication kernel with loop expansion is constructed using vector floating-point multiply-add instructions; for dot product operators, data is loaded in blocks using vector loading instructions and reduced using vector ordered reduction instructions; for quantization operators, bit width expansion is automatically completed during multiplication operations using RVV variable-width arithmetic instructions.

[0011] For the pre-filling stage in large language model inference, after the user inputs a prompt word, a forward propagation is performed on the complete prompt word to generate the first output token. For matrix multiplication operators with arithmetic strength higher than a preset threshold in this stage, a dedicated microkernel is constructed using the IME matrix extension instruction set. The input data is rearranged into two-dimensional blocks according to the IME specification and then bound to the vector register. The dot product matrix multiplication and addition instruction is executed to complete the multiplication and addition accumulation.

[0012] For bottleneck operators executed using the RVV instruction set, the optimal configuration is selected from the candidate logical vector register grouping multiples based on hardware architecture parameters and operator code characteristics.

[0013] The optimized bottleneck operator is integrated into the target inference engine to replace the original operator in performing inference tasks.

[0014] In the overall scheme of this invention, the performance of the large language model inference process is first analyzed to identify operators with high execution time. Then, different types of bottleneck operators are reconstructed using differentiated RVV vectorization. Next, for matrix multiplication operators with high arithmetic strength in the pre-filling stage, a dedicated microkernel is constructed using IME matrix expansion. At the same time, for operators using RVV, the optimal register grouping configuration is adaptively selected based on hardware characteristics. Finally, the optimized operators are integrated into the inference engine. The synergy of the above steps solves the problems of low efficiency of automatic vectorization by general compilers when facing computational modes such as mixed precision conversion and complex circular dependencies in large model inference, and the inability to adapt instruction sets for the differentiated computational characteristics of the pre-filling and decoding stages.

[0015] Preferably, the general matrix multiplication kernel constructed using vector floating-point multiply-add instructions with loop expansion includes:

[0016] A general matrix multiplication kernel is constructed using an N-fold cyclic expansion strategy, where N is an integer greater than or equal to 2. The elements of matrix A are broadcast to N independent vector accumulator registers via vector broadcast instructions, and the row vectors of matrix B are loaded into the vector registers all at once via vector load instructions.

[0017] The vector floating-point multiply-add instruction is used to multiply and add the elements of each group of broadcast matrix A with the row vectors of the loaded matrix B, and then accumulate them into the corresponding vector accumulation registers.

[0018] Furthermore, there is no data dependency between the cumulative results of each group.

[0019] The above scheme solves the technical problems of compilers being unable to effectively utilize vector register resources and the frequent pauses in the multiplication-accumulation instruction pipeline caused by write-after-read data dependency chains when processing high-dimensional matrix multiplication by adopting a combination of loop expansion, vector broadcasting and vector loading in the vectorized reconstruction of matrix multiplication operators RVV, and parallel execution of multiplication and addition by multiple independent accumulator registers with no data dependency between the results of each group. This improves the throughput of the floating-point multiplication-accumulation unit.

[0020] Preferably, the automatic bit-width expansion during multiplication using the variable-width arithmetic instruction of RVV includes:

[0021] For matrix multiplication with INT8 precision, configure the first logical vector register grouping multiple, and load the INT8 precision input vector and weight vector with this grouping multiple;

[0022] The variable-width multiplication instruction of RVV is invoked to perform multiplication on two INT8 vector operands and automatically expand them into a product vector with INT16 precision. The logical vector register grouping multiple of this product vector is twice that of the first logical vector register grouping multiple.

[0023] The variable-width reduction instruction of RVV is invoked to reduce and accumulate the product result vector with INT16 precision along the vector dimension to the scalar accumulation register with INT32 precision, thus completing the accumulation calculation of a single output element.

[0024] In the above scheme, the cross-precision operation process of the INT8 quantization operator is used. First, the input vector and weight vector are loaded with a certain grouping multiple. Then, the RVV variable-width multiplication instruction is called to automatically expand the INT8 product to INT16 precision during multiplication. Finally, the variable-width reduction instruction is called to accumulate along the vector dimension to the INT32 scalar register. This solves the problem of extra cycle overhead and memory bandwidth waste caused by the need to insert a large number of explicit type conversion instructions when processing quantization inference in traditional compilers. This allows the type conversion process to be completed in a pure vectorized manner, and the throughput is close to the upper limit of memory bandwidth.

[0025] Preferably, the construction of a dedicated microkernel using the IME matrix to extend the instruction set includes:

[0026] At the microkernel entry point, vectorized memory access instructions rearrange the input data in contiguous memory into two-dimensional blocks according to the IME specification data layout format.

[0027] The rearranged first input block is bound to the first vector register, the second input block is bound to the second vector register, and the accumulator is bound to the destination vector register through inline assembly.

[0028] Configure the vector type register, set the standard element width to the first precision, and the logical vector register grouping multiple to the first logical vector register grouping multiple. Execute the IME dot product matrix multiplication and addition instruction, interpret the data in the first vector register as a first shape matrix, interpret the data in the second vector register as a second shape matrix, perform dot product operation along K dimensions for each row and each column of the matrix, and accumulate it to the corresponding output position in the destination vector register.

[0029] Switch the configuration of the vector type register, set the standard element width to the second precision, the logical vector register grouping multiple to the second grouping multiple, and write the accumulated result in the destination vector register back to memory, wherein the second precision is higher than the first precision, and the second grouping multiple is greater than or equal to the first logical vector register grouping multiple.

[0030] In the above scheme, the complete construction of the IME microkernel includes rearranging data into two-dimensional blocks according to the IME specification, binding the blocks to fixed vector registers through inline assembly to avoid compiler interference, executing dot product matrix multiplication and addition instructions to complete block multiplication and addition accumulation after configuring the vector type register, and writing back the accumulation result after switching configurations. This solves the problem that the computing power of the matrix expansion unit cannot be released during the pre-filling stage because the compiler cannot automatically generate code adapted to the IME specification, and enables the core multiplication and addition operation to be executed efficiently by the hardware dot product instruction.

[0031] Preferably, the step of selecting the optimal configuration from the candidate logical vector register grouping multiples based on hardware architecture parameters and operator code characteristics includes:

[0032] Obtain hardware parameters and operator code characteristics. The hardware parameters include at least the vector register bit width, the maximum logical vector register grouping multiple supported by the hardware, the total number of physical vector registers, and the memory bandwidth. The operator code characteristics include at least the number of logical registers required for a single iteration, the number of memory access instructions, the number of arithmetic operation instructions, the memory access type, and whether it is in the innermost loop.

[0033] The inherent arithmetic strength of the operator is determined by the ratio of the number of arithmetic operation instructions to the number of memory access instructions. The hardware computing power level is determined by the vector register bit width and memory bandwidth. The register pressure is determined by the ratio of the number of logic registers required for a single iteration to the total number of physical vector registers. The evaluation index is calculated by combining the inherent arithmetic strength of the operator, the hardware computing power level, and the register pressure. The evaluation index is then corrected based on the memory access type and the attributes of the innermost loop.

[0034] Based on the numerical range of the evaluation index, a performance-oriented value is determined from the candidate logic vector register grouping multiples, wherein the higher the evaluation index, the larger the corresponding value.

[0035] Determine the maximum feasible value under the register constraint based on the physical register resource constraint: Based on the ratio of the total number of physical vector registers to the number of logic registers required for a single iteration, combined with the preset safety factor and the maximum logical vector register grouping multiple supported by the hardware, calculate the maximum grouping multiple that satisfies the register resource constraint.

[0036] The smaller of the performance-oriented value and the maximum feasible value under the register constraint is selected as the optimal grouping multiple configuration.

[0037] The above scheme actually proposes a hardware-aware LMUL configuration decision method, which calculates an evaluation index by comprehensively considering factors such as the inherent arithmetic strength of the operator, the hardware computing power level determined by the vector register bit width and memory bandwidth, and register pressure. Based on this index, a performance-oriented value is determined, and the maximum feasible value is calculated in combination with register resource constraints. Finally, the smaller value of the two is taken as the optimal configuration. This solves the problem that a single LMUL configuration in the RISC-V hardware fragmentation environment leads to insufficient parallelism of some operators or register overflow, and achieves a reasonable trade-off between parallelism gains and register resource consumption on different hardware platforms.

[0038] One possible specific computational step for this hardware-aware LMUL configuration decision-making method is as follows:

[0039] Obtain hardware parameter set ,in, Indicates the bit width of the vector register. Indicates the maximum number of logical vector register groups supported by the hardware. This represents the total number of physical vector registers. Indicates memory bandwidth;

[0040] Obtain the operator code feature set ,in, Indicates the element width. This represents the number of logic registers required for a single iteration when the logic vector register grouping multiple is 1. This represents the number of memory access instructions in a single iteration. This represents the number of arithmetic operation instructions in a single iteration. Indicates the memory access type. Indicates the byte step size for strafing. This represents a Boolean value used to determine whether the innermost loop is in progress;

[0041] Based on the hardware parameter set and the operator code feature set, the corrected evaluation index is calculated according to the following formula. :

[0042]

[0043] in, This is the bandwidth normalization constant. As a penalty factor for step-by-step memory access, when Indicates continuous access ,when Indicates step access ; As the innermost loop penalty factor, when hour ,when hour ;

[0044] According to the revised evaluation indicators The value of is determined according to the following rules to determine the performance-oriented logic vector register grouping multiple. :

[0045]

[0046] Based on physical register resource constraints, the logical vector register grouping multiple under register constraints is calculated using the following formula. :

[0047]

[0048] in, express The function indicates that the value within the parentheses is restricted to a specified lower bound of 1 and an upper bound of 1. Functions between

[0049] The optimal block size configuration is determined by taking the smaller value between the performance-oriented logic vector register block size and the register-constrained logic vector register block size. :

[0050] .

[0051] Preferably, integrating the optimized bottleneck operator into the target inference engine includes:

[0052] The metadata and intermediate data of the operators are organized using a structure array format. The vector loading instruction of RVV is used to perform batch data reading of contiguous memory regions. The vector reduction instruction of RVV is used to perform scoring aggregation operations. The mask predicate control instruction of RVV is used to replace the conditional branch statement to perform vectorized condition judgment and data selection.

[0053] The above solutions provide specific implementation methods for the integration and deployment phase, namely, using structure array format to organize the metadata and intermediate data of operators to improve cache space locality, using RVV vector loading instructions to perform batch reading of contiguous memory regions, using vector reduction instructions to efficiently complete score aggregation, and using mask predicate control instructions to replace conditional branches to avoid pipeline prediction penalties. These solutions solve the technical problems of decreased cache hit rate and instruction pipeline blockage caused by unfriendly data layout and dense conditional branches after the optimized operators are integrated into the inference engine.

[0054] Furthermore, the bottleneck operator includes at least matrix multiplication operator, vector dot product operator and weighted summation operator in the attention mechanism, linear projection operator, and quantization / dequantization operator.

[0055] The above limitations clearly define the optimization targets that need to be identified and located in the performance profiling phase. This solves the technical problem that requires precise focus on the optimization scope in large model inference, since the execution time of each linear layer and sequence computation operator in the feedforward neural network and attention mechanism accounts for the highest proportion. This enables the subsequent instruction set adaptation work to be targeted at the most beneficial core computing kernel.

[0056] Based on a third key aspect of the present invention, an inference operator optimization system based on RVV and IME extended instruction sets for implementing the aforementioned method is provided, comprising:

[0057] The bottleneck identification module is used to perform operator-level performance analysis on the inference process of the target large language model on the RISC-V processor, and identify operators whose execution time exceeds a preset threshold as bottleneck operators.

[0058] The RVV vectorization reconstruction module is used to perform RVV vectorization reconstruction on the bottleneck operators identified by the bottleneck identification module. The vectorization reconstruction includes: for matrix multiplication and linear projection operators, using vector floating-point multiply-add instructions to construct a general matrix multiplication kernel with loop expansion; for dot product operators, using vector loading instructions to load data in blocks and using vector ordered reduction instructions for reduction; for quantization operators, using RVV variable-width arithmetic instructions to automatically complete bit-width expansion when performing multiplication operations.

[0059] The IME microkernel building module is used for matrix multiplication operators with arithmetic strength higher than a preset threshold in the pre-filling stage of generating the first output token by performing a forward propagation of the complete prompt word after the user inputs the prompt word in the inference of large language models. It uses the IME matrix extension instruction set to build a dedicated microkernel, rearranges the input data into two-dimensional blocks according to the IME specification and binds them to the vector register, and executes the dot product matrix multiplication and addition instruction to complete the multiplication and addition accumulation.

[0060] The LMUL configuration selection module is used to select the optimal configuration from the candidate logical vector register grouping multiples for bottleneck operators that are executed using the RVV instruction set, based on hardware architecture parameters and operator code characteristics.

[0061] The integrated deployment module is used to integrate the optimized bottleneck operators into the target inference engine, replacing the original operators to perform inference tasks.

[0062] The system of the present invention, through the collaborative work of the bottleneck identification module, RVV vectorization reconstruction module, IME microkernel construction module, LMUL configuration selection module and integration deployment module, defines the complete system architecture for implementing the aforementioned method from the product construction level, and solves the technical problem of modular deployment and engineering reuse of the technical solution of the present invention on different RISC-V hardware platforms and inference engines.

[0063] Based on a third key aspect of the present invention, an electronic device is provided, comprising one or more processors;

[0064] Storage device for storing one or more programs;

[0065] When one or more programs are executed by one or more processors, the processors implement the aforementioned inference operator optimization method based on the RVV and IME extended instruction sets. Electronic devices include various hardware entities such as servers, computers, and mobile terminals.

[0066] Based on a fourth key aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed, implements the aforementioned inference operator optimization method based on RVV and IME extended instruction sets. The readable storage medium includes optical discs, hard disks, USB flash drives, ROMs, PROMs, etc.

[0067] Compared to existing technologies, this invention addresses the inherent shortcomings of automatic vectorization in general-purpose compilers for large-scale model inference scenarios by proposing a systematic, low-level operator deep optimization scheme. Existing compilers (such as GCC and Clang / LLVM) fail to achieve the expected results with automatic vectorization when handling computational patterns such as mixed-precision conversion, lookup table indirect addressing, and complex circular dependencies in large-scale model inference. In some cases, aggressive loop unrolling strategies even disrupt instruction cache locality, leading to performance degradation. This invention reconstructs the execution kernel of core bottleneck operators such as matrix multiplication, dot product, and quantization using manual inline assembly and RVV built-in functions. It explicitly manages vector register allocation and issue order at the instruction level, effectively compensating for the compiler's shortcomings in extracting complex operator semantics and truly unleashing the hardware potential of RVV vector extensions.

[0068] At the instruction set utilization level, this invention integrates RVV general vector extension and IME-specific matrix extension within the same inference scheme, optimizing the differentiated computational characteristics of the pre-filling and decoding stages in large model inference. Existing optimization schemes often employ a single strategy to handle all computational scenarios, failing to fully utilize the high throughput advantage of IME in matrix multiplication and addition. This invention constructs an IME-specific microkernel for high arithmetic strength matrix multiplication operators in the pre-filling stage, directly mapping core multiplication and addition operations to dot product matrix multiplication and addition instructions. Simultaneously, in the decoding stage, it employs an RVV-optimized kernel adapted to memory access characteristics, achieving a balance between computationally intensive and memory-intensive workloads, overcoming the performance limitations of a single instruction set in different computational stages.

[0069] Furthermore, this invention addresses the cross-platform compatibility challenges caused by the fragmentation of the RISC-V hardware ecosystem by proposing a hardware-aware static LMUL adaptive selection strategy. Existing inference optimization schemes typically employ fixed register grouping configurations, which struggle to achieve consistent acceleration on RISC-V devices with varying vector register widths, cache hierarchies, and memory bandwidths. Excessively large LMUL configurations can easily trigger overflow penalties in complex operators with high register pressure. This invention constructs a cost evaluation model that integrates multi-dimensional features such as operator arithmetic strength, register pressure, memory access patterns, and loop structures. During the compilation phase, it automatically matches the optimal LMUL configuration for different operators, effectively balancing the relationship between instruction-level parallelism gains and physical register resource constraints. This improves the generalization ability and execution stability of the optimization scheme across different microarchitecture hardware, providing a practical optimization path for the engineering implementation of large-model inference within the RISC-V ecosystem. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0071] Figure 1 This illustration shows an execution flow diagram of an inference operator optimization method based on RVV and IME extended instruction set in one embodiment of the present invention;

[0072] Figure 2 The illustration shows a schematic of the RVV FP32 GEMM kernel code with 4x loop expansion in one embodiment of the present invention;

[0073] Figure 3 The illustration shows a schematic of the Int8 matrix multiplication kernel code using variable-width arithmetic instructions in one embodiment of the present invention;

[0074] Figure 4 The illustration shows a schematic of IME microkernel code based on inline assembly in one embodiment of the present invention. Detailed Implementation

[0075] The preferred embodiments of the present invention will be described in detail below to provide a clearer understanding of the purpose, features, and advantages of the invention. It should be understood that the following embodiments are not intended to limit the scope of the invention, but are merely illustrative of the essential spirit of the technical solution of the invention.

[0076] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known techniques associated with the invention may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0077] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0078] The specific meanings of the technical terms that may be used in this invention are explained as follows:

[0079] RVV (RISC-V Vector Extension): The official standard vector extension for the RISC-V instruction set architecture, employing a vector length-independent (VLA) design to decouple the width of physical vector registers from the software instruction set. Its core mechanism involves dynamically setting the vector length using the `vsetvli` instruction, configuring the standard element width through the vector type register, and bundling multiple physical vector registers into a single logical register using logical vector register grouping multiples to improve single-instruction data throughput. In this invention, RVV is primarily used for vectorized reconstruction of operators such as matrix multiplication, dot product, and quantization.

[0080] IME (Integrated Matrix Extension) is a dedicated matrix extension instruction set for the RISC-V architecture. Its key feature is that it does not introduce a separate matrix register file; instead, it directly reuses the vector registers of the RVV architecture and reinterprets these vector registers as two-dimensional matrix blocks during instruction execution. IME provides dot product matrix multiplication and addition instructions such as vmadotu, which can complete a two-dimensional block of multiplication and addition accumulation operations within a single instruction. In this invention, IME is only used for matrix multiplication operators with arithmetic strength exceeding a preset threshold during the pre-filling stage. Block-level multiplication and addition throughput is improved by directly calling IME instructions in the microkernel.

[0081] LMUL (Logic Vector Register Grouping Multiplier): A parameter in RVV used to configure the grouping multiplier of logical vector registers, with selectable values ​​of 1, 2, 4, 8 (and some partial values). By bundling multiple physical registers into a single logical register, LMUL increases the data size processed by a single vector instruction, but proportionally reduces the number of independent logical registers available in the system. For example, when LMUL=8, 32 physical registers form only 4 logical registers, which can easily lead to register overflow and additional memory access penalties in complex operators containing a large number of intermediate variables. This invention uses a hardware-aware cost evaluation model to statically select the optimal LMUL configuration for different operators to balance parallelism gains and register pressure.

[0082] Variable-width arithmetic instructions: These are a special class of vector instructions provided by RVV that can automatically expand bit width while performing arithmetic operations. For example, the variable-width multiplication instruction `vwmulu` directly expands the product to INT16 precision when performing multiplication of two INT8 vectors, and the variable-width reduction instruction `vwredsumu` adds the INT16 portion to the INT32 scalar register. This invention utilizes these instructions to avoid explicit type conversion instructions in cross-precision operations of quantized operators, reducing instruction cycle overhead.

[0083] Microkernel: In this invention, a microkernel refers to a low-level computational core code unit written in manual inline assembly and optimized for specific computational modes (such as matrix multiplication). Its key feature is that it bypasses the compiler's automatic instruction generation and register allocation logic, directly managing vector register binding, instruction issuance order, and data arrangement to achieve precise control over hardware resources. This invention constructs a general-purpose matrix multiplication microkernel for RVV and a dedicated matrix multiply-add microkernel for IME.

[0084] Inline assembly: A programming method that embeds assembly code directly into C / C++ source code. In this invention, inline assembly is used to bind critical data to fixed-vector registers during the IME microkernel construction process. This avoids uncertainties in register allocation caused by the compiler during optimization, ensuring that the instruction sequence executes as expected.

[0085] AoS (Array of Structures): A data organization format that encapsulates all attributes of each data element in a structure, and then stores multiple structures sequentially and consecutively. This invention uses the AoS format to organize the metadata and intermediate data of operators during the integration and deployment phase, thereby improving the spatial locality and hit rate of the CPU cache.

[0086] Ordered reduction instructions: RVV provides instructions that reduce data in a vector register to a scalar along the vector dimension, such as vfredusum (unordered vector floating-point reduction summation). In this invention, the dot product operator utilizes these instructions to efficiently perform local accumulation calculations after loading data in blocks.

[0087] In the following descriptions of the embodiments, the use of standard instruction mnemonics and register operands defined in the RVV and IME instruction set specifications may be involved. For ease of understanding, the following explanation is provided: Mnemonics starting with 'v' in the code (such as vle32.v, vfmv.vf, vfmacc.vv, vfredusum.v, etc.) are standard RVV vector instructions, where vle32.v indicates loading 32-bit elements into a vector register, vfmv.vf indicates broadcasting a scalar floating-point number to each element of the vector register, vfmacc.vv ​​indicates performing vector-to-vector floating-point multiplication and addition, and vfredusum.v indicates performing unordered reduction summation on the floating-point numbers in the vector register. Instructions starting with 'vw' or 'vf' (such as vwmulu.vv, vfwcvt_f_f_v, etc.) are RVV variable-width arithmetic instructions or precision conversion instructions, used to perform bit-width expansion multiplication from INT8 to INT16, precision widening conversion from FP16 to FP32, and other operations, respectively. `vmadotu` is an IME matrix extension instruction used to perform two-dimensional block dot product matrix multiplication and accumulation. The above instructions are all standard mnemonics defined in the RISC-V vector extension and matrix extension specifications, and those skilled in the art can learn their precise semantics from the RVV / IME specification documents.

[0088] In the code, v0 to v31 represent the 32 physical vector registers of the RVV. Specific numbers (such as v0, v1, and v28) are selected for specific purposes in the embodiments and are merely exemplary choices, not limitations on the scope of the invention. The notations m1, m2, etc., indicate the configuration values ​​of the logical vector register grouping multiples. For example, e8, m1 indicates a standard element width of 8 bits and a logical vector register grouping multiple of 1; e32, m2 indicates a standard element width of 32 bits and a logical vector register grouping multiple of 2. acc0-acc3 are variable names used for accumulators in the code, corresponding to each group of independent vector accumulator registers. Their names themselves indicate their function as accumulators. exp(x - x_max) is the standard calculation form used in the Softmax function to ensure numerical stability, i.e., subtracting the maximum value from each element of the input vector before taking the exponent. This technique is known to those skilled in the art.

[0089] The instruction mnemonics, register numbers, and configuration abbreviations mentioned above are standard terms in the RISC-V Vector / Matrix Extensions specification and the field of deep learning. In the specific implementations described in this specification, the purpose and function of each code symbol are explained in context. Those skilled in the art, combining the RVV / IME specification document and the description in this specification, can clearly understand the meaning of each code example and implement the technical solution of this invention without misunderstanding or ambiguity.

[0090] In one feasible embodiment, combined with Figure 1 As shown, this invention provides an inference operator optimization method based on RVV and IME extended instruction sets, comprising the following steps executed by a computer system:

[0091] Step S100: Perform operator-level performance analysis on the inference process of the target large language model on the RISC-V processor, and identify the operators whose execution time exceeds a preset threshold as bottleneck operators.

[0092] Step S200: RVV vectorization reconstruction is performed on the bottleneck operator: For matrix multiplication and linear projection operators, a general matrix multiplication kernel with loop expansion is constructed using vector floating-point multiply-add instructions; for dot product operators, data is loaded in blocks using vector loading instructions and reduced using vector ordered reduction instructions; for quantization operators, bit width expansion is automatically completed during multiplication operations using RVV variable-width arithmetic instructions.

[0093] Step S300: For the pre-filling stage in the large language model inference where the user inputs a prompt word and then performs a forward propagation of the complete prompt word to generate the first output token, for the matrix multiplication operator with arithmetic strength higher than a preset threshold in this stage, a dedicated microkernel is constructed using the IME matrix extension instruction set. The input data is rearranged into two-dimensional blocks according to the IME specification and then bound to the vector register. The dot product matrix multiplication and addition instruction is executed to complete the multiplication and addition accumulation.

[0094] Step S400: For bottleneck operators executed using the RVV instruction set, select the optimal configuration from the candidate logical vector register grouping multiples based on hardware architecture parameters and operator code characteristics.

[0095] Step S500: Integrate the optimized bottleneck operator into the target inference engine and replace the original operator to perform the inference task.

[0096] Those skilled in the art should understand that Figure 1 The sequence of steps shown is for illustrative purposes only. There is no strict temporal dependency between the steps. Some steps may be adjusted in order or executed in parallel depending on the actual hardware conditions and operator characteristics.

[0097] The specific implementation process of the bottleneck operator identification step (step S100) will be further described in detail in one of the following possible embodiments.

[0098] In this invention, step S100 aims to perform fine-grained operator-level performance analysis on the entire inference process of the target large language model on the RISC-V processor, accurately locate the computational kernel with the highest execution time, and thus focus limited optimization resources on the most profitable operators.

[0099] In an optional embodiment, the streamlined llama2.c inference framework is used as the profiling vehicle, and Llama2-7B-Chat is used as the target model. This framework removes complex system-level encapsulation and multi-threaded scheduling logic, effectively eliminating performance interference introduced by the software framework, thereby accurately quantifying the execution time distribution of basic operators. The experiment sets a single-turn dialogue load, inputting 256 tokens and generating 64 tokens. Compilation is performed using a GCC 14.2 compiler supporting RVV extensions, with O3 optimization level and corresponding RVV target architecture parameters enabled.

[0100] During the performance data acquisition phase, the Linux perf performance analysis tool was used to perform instruction-level sampling of the entire inference process. The perf tool can record the CPU cycle usage of each function and instruction address during program execution with very low intrusion. During the experiment, the execution time percentage of each operator was statistically analyzed, and operators whose time consumption percentage exceeded a preset threshold were selected as bottleneck operators.

[0101] Analysis results show that scalar-dominated, non-computationally intensive operators such as token embedding, RMSnorm normalization, and output sampling account for less than 1% of the time consumption, having a negligible impact on end-to-end inference performance. In contrast, the attention mechanism and feedforward neural network constitute the main computational components of the Transformer architecture, contributing approximately 99.83% of the execution time cumulatively. Further decomposition reveals that scaling dot products, weighted summation, and Q / K / V / O linear projection in the attention mechanism are the main sources of attention overhead. Linear up projection and linear down projection in the feedforward neural network constitute the main bottlenecks in the FFN stage. All linear layer operators (including various projections in the attention mechanism, the FFN layer, and the final language model head) collectively account for approximately 52.54% of the total inference time, and their underlying execution can be mapped to general matrix multiplication or general matrix-vector multiplication. Based on the above analysis results, matrix multiplication operators, vector dot products and weighted summation operators in the attention mechanism, linear projection operators, and quantization and dequantization operators are identified as the optimization targets of this invention.

[0102] Those skilled in the art should understand that the selection of the llama2.c framework and the Llama2-7B-Chat model described above is merely illustrative. In other embodiments, different inference engines (such as llama.cpp or fastllm) and target models with different parameter sizes (such as Qwen2.5-0.5B, Qwen2.5-3B, etc.) can be selected according to actual deployment needs. The corresponding bottleneck operator types and their time consumption proportions may vary due to differences in model architecture and computational load, but this does not affect the applicability of the core optimization method of this invention. The specific types and number of bottleneck operators should not be construed as limiting the invention.

[0103] The specific implementation process of the RVV vectorization reconstruction step (step S200) will be further described in detail in one of the following possible embodiments.

[0104] In this invention, step S200 performs vectorization reconstruction on the bottleneck operators identified in step S100 based on the RISC-V Vector Extension (RVV) instruction set to overcome the performance limitations of automatic vectorization in general-purpose compilers. During the RVV vectorization reconstruction of bottleneck operators, three main scenarios exist: for matrix multiplication and linear projection operators, a general matrix multiplication kernel with loop expansion is constructed using vector floating-point multiply-add instructions; for dot product operators, data is loaded in blocks using vector loading instructions and reduced using vector ordered reduction instructions; for quantization operators, bit-width expansion is automatically performed during multiplication operations using RVV variable-width arithmetic instructions.

[0105] The following possible embodiments illustrate the RVV vectorization reconstruction of matrix multiplication and linear projection operators.

[0106] For matrix multiplication and linear projection operators, this invention employs vector floating-point multiply-add instructions to construct a general matrix multiplication kernel with cyclic expansion. Taking single-precision floating-point matrix multiplication as an example, let the shape of matrix A involved in the operation be (M, K), and the shape of matrix B be (K, N), where M is the number of rows in matrix A, K is the number of columns in matrix A and equal to the number of rows in matrix B, and N is the number of columns in matrix B.

[0107] In an optional embodiment, an N-fold cyclic expansion strategy is employed (N is an integer greater than or equal to 2, typically 4). Specifically, the elements of matrix A are broadcast to N independent vector accumulation registers (e.g., v0 to v3) using RVV vector broadcast instructions (e.g., vfmv.vf), while the row vectors of matrix B are loaded into the vector registers all at once using vector loading instructions (e.g., vle32.v). Subsequently, vector floating-point multiply-accumulate instructions (e.g., vfmacc.vv) are used to multiply and accumulate the broadcast elements of matrix A with the loaded row vectors of matrix B into the corresponding vector accumulation registers.

[0108] like Figure 2 As shown in one possible implementation of the code, this general-purpose matrix multiplication kernel explicitly expands the computation loop and utilizes multiple independent sets of accumulator registers (acc0-acc3) for concurrent computation. This design effectively breaks the read-after-write (RAW) data dependency chain commonly found in compiler-generated code, allowing the processor's out-of-order execution units to be continuously filled with the issue queue, thereby masking the deep execution latency of floating-point multiply-accumulate instructions and significantly improving instruction throughput.

[0109] The following possible embodiments illustrate the RVV vectorization reconstruction of the dot product operator.

[0110] For the dot product operator in the attention mechanism, the core challenge in optimization lies in completely mapping the scalar-dominated control flow to an RVV vector data flow. During the dot product stage of calculating the attention score, historical key vectors and the current query vector are loaded in blocks using vle32.v. Local sums are obtained by calling vector ordered reduction instructions (such as vfredusum.v), and tail data is dynamically processed using the vsetvl instruction, eliminating the high boundary check overhead in the compiler's automatic scalarization of the code.

[0111] In the Softmax normalization stage, to ensure the stability of floating-point values, the calculation is performed in the form of exp(x - x_max). The maximum value of the vector segment is extracted using vfredmax.v, and a combination of vector memory access and scalar computation is used to compensate for the limitation that the standard RVV instruction set does not support native vector exponentiation operations. In the final weighted summation stage, vfmacc.vv ​​is used to broadcast and accumulate the weights, thus reducing the number of instructions.

[0112] The RVV vectorization reconstruction of the quantization operator is illustrated in the following possible embodiments.

[0113] For quantization and dequantization operators, this invention utilizes the widening arithmetic properties of RVV to achieve cross-precision operations. For example... Figure 3 The code for one possible implementation, taking INT8 precision matrix multiplication as an example, firstly, the grouping factor of the first logical vector register is configured (e.g., LMUL = 4). The input vector and weight vector with INT8 precision are loaded using the vle8.v instruction with this grouping factor to improve data throughput. The core multiplication step calls the RVV variable-width multiplication instruction (e.g., vwmulu.vv), which automatically expands the product result to INT16 precision while performing the multiplication of two INT8 vector operands. The grouping factor of the logical vector register of the resulting product vector is automatically doubled (becoming LMUL = 8). The subsequent reduction stage calls the RVV variable-width reduction instruction (e.g., vwredsumu.vs) to accumulate the INT16 precision portion into an INT32 precision scalar accumulation register, ensuring the numerical precision of the accumulation process.

[0114] In the precision conversion path between FP16 and FP32, floating-point type conversion instructions natively supported by RVV hardware are directly adopted (such as vfncvt_f_f_w for compression and vfwcvt_f_f_v for expansion). The exponent truncation and mantissa rounding are strictly followed in accordance with the IEEE754 standard, eliminating the penalty for branch prediction failure in scalar loops and increasing the type conversion throughput to the upper limit that matches the system memory bandwidth.

[0115] Those skilled in the art should understand that the above-described embodiments of the INT8 quantization operator are merely illustrative examples. In other embodiments, the variable-width arithmetic instructions are also applicable to cross-precision operations of other low-precision data types (such as INT4, FP16, etc.). The specific instruction names and data type parameters can be adjusted accordingly based on the actual RVV specification version and the instruction set supported by the hardware.

[0116] The specific implementation process of the ME-specific microkernel construction step (step S300) will be further described in detail in one of the following possible embodiments.

[0117] Step S300 addresses matrix multiplication operators with arithmetic strength exceeding a preset threshold in the pre-filling stage of large language model inference by constructing a dedicated microkernel using the IME matrix extension instruction set. It's important to note that the pre-filling stage is the first stage in the autoregressive inference of the large language model: after the user inputs a prompt, the system enters the pre-filling stage. The task of this stage is to perform a complete forward propagation of the complete prompt to understand the user's intent and generate the first output token. The main operation in this stage is matrix-matrix multiplication, whose computational complexity increases proportionally to the sequence length (O(n)). 2 The growth can fully activate the matrix operation unit of the hardware, exhibiting computing power intensive characteristics, and therefore is suitable for acceleration using IME.

[0118] In an optional embodiment, the hardware is based on the Selective Spacetime M1 platform, which supports IME matrix expansion. The IME is characterized by not introducing a separate matrix register file, but instead directly reusing the 32 vector registers (v0 to v31) of the RVV. When executing dot product matrix multiplication and addition instructions, the data in the vector registers is reinterpreted as two-dimensional matrix blocks. The IME provides the vmadotu family of instructions to accelerate low-level GEMM calculations. Taking the integer matrix multiplication and addition instruction as an example, the data in vs1 is interpreted as a Copyes×(M,K) matrix A, vs2 is interpreted as a Copyes×(K,N) matrix B, and the resulting matrix C is accumulated into the destination register vd.

[0119] The IME microkernel build process is as follows:

[0120] Data rearrangement stage: At the microkernel entry point, vectorized memory access instructions rearrange the input data in contiguous memory into two-dimensional blocks according to the IME specification's data layout format. Because IME has specific alignment and block requirements for data layout, the original contiguous memory layout needs to be converted to a format conforming to the IME execution specification.

[0121] Register binding phase: Inline assembly binds the rearranged first input block to the first vector register (e.g., v0), the second input block to the second vector register (e.g., v1), and the accumulator to the destination vector register (e.g., v28). The reason for using inline assembly instead of relying on the compiler for automatic register allocation is that IME instructions have specific constraints on register numbering (e.g., integer output requires two consecutive vector registers and the destination register index must be even), making it difficult for the compiler to automatically generate register allocation schemes that satisfy these constraints.

[0122] Execution phase: Configure the vector type register, set the standard element width to the first precision (e.g., INT8, i.e., SEW=8), and the logical vector register grouping multiple to the first logical vector register grouping multiple (e.g., LMUL=1), then execute the IME dot product matrix multiplication and addition instruction. At this point, the data in v0 is interpreted as a matrix A of Copies×(M,K), and the data in v1 is interpreted as a matrix B of Copies×(K,N). v28 acts as an accumulator. For any output element C(i,j), the hardware takes the i-th row of v0 and the j-th column of v1, performs the dot product along the K-dimensional axis, accumulates it, and writes it back to the corresponding output position in v28. Specifically, fixing the same row of v0 and switching the columns of v1 sequentially yields C(i,0) to C(i,N-1); fixing the same column of v1 and switching the rows of v0 sequentially yields C(0,j) to C(M-1,j). A row of A is multiplexed across multiple output columns, and a column of B is multiplexed across multiple output rows. The efficiency of IME comes from this combination of register-based block multiplexing and multiply-accumulate. For example... Figure 4 The code for one possible implementation is shown below.

[0123] Result write-back stage: Switch the configuration of the vector type register, set the standard element width to the second precision (e.g., INT32, i.e., SEW=32), and the logical vector register grouping multiple to the second grouping multiple (e.g., LMUL=2), and write the accumulated result in the destination vector register v28 back to memory. The second precision is higher than the first precision (INT32 is higher than INT8) because low-precision integers are prone to overflow during continuous multiplication and addition, and the IME automatically widens the result before storage; the second grouping multiple is greater than or equal to the first logical vector register grouping multiple to meet the storage bit width requirements of the widened data.

[0124] In the main loop, the vmadotu multiply-accumulate stage uses an e8, m1 configuration, while e32, m2 is only used for the initialization of the widened accumulator and the write-back of the result. Here, m1 is not a simple compression of parallelism, but rather determined by the IME's execution method—the IME interprets the vector register as a two-dimensional matrix block, and a single vmadotu can complete the multiply-accumulate along the K-dimensional dimension of an output sub-block. The actual throughput is mainly provided by the matrix expansion unit. The fixed overhead from loop branches, address updates, and vsetvli switching is diluted by the computational density of block-level multiply-accumulate; the smaller register grouping also retains more usable logic registers, which is beneficial for simultaneously resident input blocks, accumulators, and temporary variables, reducing register overflow and duplicate loading.

[0125] Those skilled in the art should understand that the specific register allocations (such as the selection of v0, v1, and v28) and configuration parameters (such as e8, m1, e32, and m2) of the above-described IME microkernel are merely illustrative examples. Register numbers, SEW, and LMUL configurations can be adjusted accordingly on different hardware platforms (such as different VLEN processors) and for different data types, but all remain within the core concept of this invention: constructing a dedicated microkernel using the IME instruction set.

[0126] The following possible embodiment provides a more detailed description of the specific implementation process of the LMUL optimal configuration selection step (step S400).

[0127] Step S400, for bottleneck operators executed using the RVV instruction set, adaptively selects the optimal configuration from the candidate logical vector register grouping multiple set ({1, 2, 4, 8}) based on hardware architecture parameters and operator code characteristics. Since different RISC-V vendors implement vector register lengths, pipeline structures, and cache levels differently, a fixed LMUL configuration cannot achieve optimal performance on all hardware. Therefore, this invention proposes a static LMUL selection strategy based on a heuristic cost model.

[0128] In an optional embodiment, a set of hardware parameters is first obtained. The hardware parameters are defined as a quadruple H = {VL, L...} max , R total , BW}, where VL represents the bit width of the vector register, L max R represents the maximum LMUL value supported by the hardware. total BW represents the total number of physical vector registers.

[0129] Simultaneously, the set of operator code features is obtained. The operator features are defined as K = {E, R}. used A, C, T mem , S, }, where E represents the element width and R used Indicates the number of logic registers required for a single iteration when LMUL=1, A represents the number of memory access instructions in a single iteration, C represents the number of arithmetic operation instructions in a single iteration, and T represents the number of logic registers required for a single iteration. mem Indicates the memory access type (0 for sequential access, 1 for strafing), and S represents the byte step size for strafing. It is a Boolean value (1 indicates the innermost loop, 0 indicates a non-innermost loop).

[0130] Based on the hardware parameters and operator code characteristics, the corrected evaluation index M is calculated according to the following formula:

[0131]

[0132] This index incorporates three trade-off factors: in the numerator, (C / A) represents the inherent arithmetic strength of the operator, (VL / 128) represents the vector bit width gain, and (β / BW) represents the bandwidth compensation factor (β is the bandwidth normalization constant, the specific value of which can be set by those skilled in the art based on the bandwidth calibration results of the actual test platform; typical values ​​can refer to the measured bandwidth data of mainstream RISC-V platforms); in the denominator, (1 + 3 × This is a register constraint term. When the logical register occupancy approaches the upper limit of the physical register, this term will increase non-linearly, thereby reducing the value of M. and These are the step-by-step memory access penalty factor and the innermost loop penalty factor, respectively, when T mem Indicates continuous access =1, when T mem Indicates step access =0.5; when =1 =0.8, when =0 =1.

[0133] Based on the value of the modified evaluation metric M, the performance-oriented LMUL value is determined according to the following rules:

[0134] When M < 0.75, take LMUL. perf = 1 (Operator memory access is limited, not suitable for large LMULs);

[0135] When 0.75 ≤ M < 1.9, take LMUL. perf = 2;

[0136] When 1.9 ≤ M < 4.5, take LMUL. perf = 4;

[0137] When M≥4.5, take LMUL. perf = 8 (Operator computation is intensive, suitable for large LMUL).

[0138] Simultaneously, to avoid register overflow, hard constraints must be applied based on physical register resource constraints. A safety factor of 0.75 is introduced to reserve margin, and the maximum feasible LMUL value under register constraints is calculated using the following formula, which is the logical vector register grouping multiple. :

[0139]

[0140] in, express The function indicates that the value within the parentheses is restricted to a specified lower bound of 1 and an upper bound of 1. Functions between;

[0141] The formula means: calculate the ratio of the total number of physical vector registers to the required number of logic registers, multiply by a safety factor of 0.75, take the logarithm to base 2, round down, and then raise the result to base 2, finally limiting the result to [1, L]. max Within the range. The final optimal LMUL configuration takes the smaller of the two:

[0142]

[0143] This strategy prioritizes performance improvement when resources are plentiful and automatically scales down configuration when resources are limited. Since the decision is made during the compile phase and bound statically, it does not introduce additional runtime overhead.

[0144] Those skilled in the art should understand that the above thresholds (0.75, 1.9, 4.5) and safety factor (0.75) are based on experimental statistical results of a typical RISC-V hardware platform. They can be appropriately adjusted according to actual test data under different microarchitectures, but they do not deviate from the core concept of this invention to select the optimal LMUL configuration based on arithmetic strength, register pressure and memory access mode.

[0145] The specific implementation process of the integration deployment step (step S500) will be further described in detail in one of the following possible embodiments.

[0146] Step S500 integrates the bottleneck operator optimized by the aforementioned steps into the target inference engine, replacing the original operator to perform the inference task.

[0147] In terms of data arrangement, the AoS (Array of Structures) format is used to organize the metadata and intermediate data of the operators. The array of structures encapsulates all attributes of each data element in a single structure, and then multiple structures are stored sequentially and consecutively. This format maximizes the spatial locality and hit rate of CPU caches at all levels, reducing memory access latency caused by cache misses.

[0148] At the computational execution level, RISC-V vector extension instructions are intensively invoked: vector loading instructions (such as vle8.v and vle32.v) are used to perform batch data reading of contiguous memory regions; vector reduction instructions (such as vfredsum.v) are used to efficiently complete scoring aggregation operations; and hardware mask predicate control mechanisms (such as vmerge.vvm) are used to replace traditional conditional branch statements, effectively eliminating the prediction penalty caused by intensive conditional branches to the processor instruction pipeline, and achieving a dual improvement in memory access and computational efficiency.

[0149] Regarding inference engine adaptation, for different inference frameworks such as llama.cpp and fastllm, the optimized kernel functions replace the original implementations according to the operator registration interfaces of each framework. For different RISC-V hardware platforms, the corresponding machine code is compiled and generated according to the optimal LMUL configuration selected in step S400 to ensure that the optimized operators achieve the best execution efficiency on the target hardware.

[0150] The following examples further illustrate the synergistic relationship between the five steps described above.

[0151] Step S100 clarifies the optimization objective through operator-level performance profiling, providing a precise target for subsequent steps. Without this step, subsequent optimizations would lack focus and may invest limited optimization resources in operators that have little impact on end-to-end performance.

[0152] Steps S200 and S300 respectively perform deep optimization on the bottleneck operators on the two instruction set paths RVV and IME. The two are aimed at different computational characteristics—RVV is suitable for comprehensive acceleration of general vector operations, while IME is specifically designed to provide ultra-high throughput multiplication and addition capabilities for matrix multiplication with high arithmetic strength in the pre-filling stage. The two are complementary rather than replacements, and together they cover the differentiated needs of the pre-filling and decoding stages.

[0153] Step S400 provides hardware-aware LMUL adaptive configuration for the operator using RVV in step S200, ensuring that the same set of optimization code can achieve stable acceleration on different RISC-V microarchitectures; if this step is missing, the RVV optimization in step S200 may experience performance degradation on some hardware due to register overflow.

[0154] Step S500 integrates the optimization results of the first three steps into the actual inference engine in an engineering manner, and further amplifies the acceleration benefits of the previous steps through data arrangement and instruction-level optimization. Thus, the five steps are progressive and interconnected; the absence of any step will significantly weaken the overall optimization effect, together forming a complete and inseparable technical solution.

[0155] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0156] Those skilled in the art should understand that the system module division and workflow step division in the above embodiments are only logical functional divisions. In actual implementation, the functions of multiple modules can be merged into one module as needed, or the functions of one module can be split into multiple modules. Similarly, the numbering of the above process steps is only for ease of description and reference, and does not limit the execution order of each step. The actual execution order should be dynamically determined according to the data dependencies in the workflow.

[0157] This invention can be implemented on various computing devices, including but not limited to: general-purpose servers, dedicated inference servers, virtual machines or container instances on cloud computing platforms, edge computing devices, etc. The controller's large language model can be deployed locally or can call remote services via API. Inference services for candidate models can come from various model service platforms such as Hugging FaceInference API, Replicate, and AWS SageMaker.

[0158] It should be noted that the various technical features described in the above specific embodiments can be freely combined and used in different embodiments, as long as they do not contradict each other. For those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

[0159] It should be understood that the program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0160] The acquisition, storage, and application of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0161] The technical terms, principles, or means related to the technical solutions of the present invention mentioned in the above embodiments, which are not described in detail above, are all well-known technologies or common practices that are known to those skilled in the art.

[0162] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. An inference operator optimization method based on RVV and IME extended instruction sets, characterized in that, The method includes: Operator-level performance analysis was performed on the inference process of the target large language model on the RISC-V processor, and operators whose execution time ratio exceeded a preset threshold were identified as bottleneck operators. The bottleneck operators are reconstructed using RVV vectorization: for matrix multiplication and linear projection operators, a general matrix multiplication kernel with loop expansion is constructed using vector floating-point multiply-add instructions; for dot product operators, data is loaded in blocks using vector loading instructions and reduced using vector ordered reduction instructions; for quantization operators, bit width expansion is automatically completed during multiplication operations using RVV variable-width arithmetic instructions. For the pre-filling stage in large language model inference, after the user inputs a prompt word, a forward propagation is performed on the complete prompt word to generate the first output token. For matrix multiplication operators with arithmetic strength higher than a preset threshold in this stage, a dedicated microkernel is constructed using the IME matrix extension instruction set. The input data is rearranged into two-dimensional blocks according to the IME specification and then bound to the vector register. The dot product matrix multiplication and addition instruction is executed to complete the multiplication and addition accumulation. For bottleneck operators executed using the RVV instruction set, the optimal configuration is selected from the candidate logical vector register grouping multiples based on hardware architecture parameters and operator code characteristics. The optimized bottleneck operator is integrated into the target inference engine to replace the original operator in performing inference tasks.

2. The inference operator optimization method based on RVV and IME extended instruction set according to claim 1, characterized in that, The general matrix multiplication kernel constructed using vector floating-point multiply-add instructions with loop expansion includes: A general matrix multiplication kernel is constructed using an N-fold cyclic expansion strategy, where N is an integer greater than or equal to 2. The elements of matrix A are broadcast to N independent vector accumulator registers via vector broadcast instructions, and the row vectors of matrix B are loaded into the vector registers all at once via vector load instructions. The vector floating-point multiply-add instruction is used to multiply and add the elements of each group of broadcast matrix A with the row vectors of the loaded matrix B, and then accumulate them into the corresponding vector accumulation registers. Furthermore, there is no data dependency between the cumulative results of each group.

3. The inference operator optimization method based on RVV and IME extended instruction set according to claim 1, characterized in that, The automatic bit-width expansion during multiplication using the variable-width arithmetic instructions of RVV includes: For matrix multiplication with INT8 precision, configure the first logical vector register grouping multiple, and load the INT8 precision input vector and weight vector with this grouping multiple; The variable-width multiplication instruction of RVV is invoked to perform multiplication on two INT8 vector operands and automatically expand them into a product vector with INT16 precision. The logical vector register grouping multiple of this product vector is twice that of the first logical vector register grouping multiple. The variable-width reduction instruction of RVV is invoked to reduce and accumulate the product result vector with INT16 precision along the vector dimension to the scalar accumulation register with INT32 precision, thus completing the accumulation calculation of a single output element.

4. The inference operator optimization method based on RVV and IME extended instruction set according to claim 1, characterized in that, The construction of a dedicated microkernel using IME matrix-extended instruction sets includes: At the microkernel entry point, vectorized memory access instructions rearrange the input data in contiguous memory into two-dimensional blocks according to the IME specification data layout format. The rearranged first input block is bound to the first vector register, the second input block is bound to the second vector register, and the accumulator is bound to the destination vector register through inline assembly. Configure the vector type register, set the standard element width to the first precision, and the logical vector register grouping multiple to the first logical vector register grouping multiple. Execute the IME dot product matrix multiplication and addition instruction, interpret the data in the first vector register as a first shape matrix, interpret the data in the second vector register as a second shape matrix, perform dot product operation along K dimensions for each row and each column of the matrix, and accumulate it to the corresponding output position in the destination vector register. Switch the configuration of the vector type register, set the standard element width to the second precision, the logical vector register grouping multiple to the second grouping multiple, and write the accumulated result in the destination vector register back to memory, wherein the second precision is higher than the first precision, and the second grouping multiple is greater than or equal to the first logical vector register grouping multiple.

5. The inference operator optimization method based on RVV and IME extended instruction set according to claim 1, characterized in that, The step of selecting the optimal configuration from candidate logical vector register grouping multiples based on hardware architecture parameters and operator code characteristics includes: Obtain hardware parameters and operator code characteristics. The hardware parameters include at least the vector register bit width, the maximum logical vector register grouping multiple supported by the hardware, the total number of physical vector registers, and the memory bandwidth. The operator code characteristics include at least the number of logical registers required for a single iteration, the number of memory access instructions, the number of arithmetic operation instructions, the memory access type, and whether it is in the innermost loop. The inherent arithmetic strength of the operator is determined by the ratio of the number of arithmetic operation instructions to the number of memory access instructions. The hardware computing power level is determined by the vector register bit width and memory bandwidth. The register pressure is determined by the ratio of the number of logic registers required for a single iteration to the total number of physical vector registers. The evaluation index is calculated by combining the inherent arithmetic strength of the operator, the hardware computing power level, and the register pressure. The evaluation index is then corrected based on the memory access type and the attributes of the innermost loop. Based on the numerical range of the evaluation index, a performance-oriented value is determined from the candidate logic vector register grouping multiples, wherein the higher the evaluation index, the larger the corresponding value. Determine the maximum feasible value under the register constraint based on the physical register resource constraint: Based on the ratio of the total number of physical vector registers to the number of logic registers required for a single iteration, combined with the preset safety factor and the maximum logical vector register grouping multiple supported by the hardware, calculate the maximum grouping multiple that satisfies the register resource constraint. The smaller of the performance-oriented value and the maximum feasible value under the register constraint is selected as the optimal grouping multiple configuration.

6. The inference operator optimization method based on RVV and IME extended instruction set according to claim 1, characterized in that, The integration of the optimized bottleneck operator into the target inference engine includes: The metadata and intermediate data of the operators are organized using a structure array format. The vector loading instruction of RVV is used to perform batch data reading of contiguous memory regions. The vector reduction instruction of RVV is used to perform scoring aggregation operations. The mask predicate control instruction of RVV is used to replace the conditional branch statement to perform vectorized condition judgment and data selection.

7. The inference operator optimization method based on RVV and IME extended instruction set according to claim 1, characterized in that, The bottleneck operators include at least matrix multiplication operators, vector dot product operators and weighted summation operators in attention mechanisms, linear projection operators, and quantization / dequantization operators.

8. A reasoning operator optimization system based on RVV and IME extended instruction sets for implementing the method of any one of claims 1-7, characterized in that, include: The bottleneck identification module is used to perform operator-level performance analysis on the inference process of the target large language model on the RISC-V processor, and identify operators whose execution time exceeds a preset threshold as bottleneck operators. The RVV vectorization reconstruction module is used to perform RVV vectorization reconstruction on the bottleneck operators identified by the bottleneck identification module. The vectorization reconstruction includes: for matrix multiplication and linear projection operators, using vector floating-point multiply-add instructions to construct a general matrix multiplication kernel with loop expansion; For the dot product operator, data is loaded in blocks using vector loading instructions and reduced using vector ordered reduction instructions; for the quantization operator, the variable-width arithmetic instructions of RVV are used to automatically expand the bit width when performing multiplication operations. The IME microkernel building module is used for matrix multiplication operators with arithmetic strength higher than a preset threshold in the pre-filling stage of generating the first output token by performing a forward propagation of the complete prompt word after the user inputs the prompt word in the inference of large language models. It uses the IME matrix extension instruction set to build a dedicated microkernel, rearranges the input data into two-dimensional blocks according to the IME specification and binds them to the vector register, and executes the dot product matrix multiplication and addition instruction to complete the multiplication and addition accumulation. The LMUL configuration selection module is used to select the optimal configuration from the candidate logical vector register grouping multiples for bottleneck operators that are executed using the RVV instruction set, based on hardware architecture parameters and operator code characteristics. The integrated deployment module is used to integrate the optimized bottleneck operators into the target inference engine, replacing the original operators to perform inference tasks.

9. An electronic device, characterized in that, Includes one or more processors; Storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement the inference operator optimization method based on the RVV and IME extended instruction set as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the inference operator optimization method based on the RVV and IME extended instruction set as described in any one of claims 1-7.

Images