Vector extension based register dynamic grouping method and system

Abstract

The present application belongs to the technical field of artificial intelligence, and particularly relates to a register dynamic grouping method and system based on vector expansion. The register dynamic grouping method based on vector expansion comprises the following steps: constructing an LMUL characteristic database of a neural network operator; receiving a neural network calculation graph, identifying a fusible operator cluster therein, and querying the native LMUL preference of the operators in the corresponding operator cluster in the LMUL characteristic database; constructing a cost evaluation model to determine an optimal LMUL configuration scheme; using the LMUL of the optimal LMUL configuration scheme to annotate the calculation graph; and parsing the optimized graph with LMUL annotation to adjust the memory layout and the blocking strategy. Through the operator LMUL characteristic database, the present application solves the problem of lack of operator-level semantic perception of the compiler; through the LMUL negotiation algorithm, the present application solves the problem of non-uniformity of cross-operator LMUL, eliminates the data rearrangement overhead, and realizes the maximization of end-to-end AI inference performance.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a method and system for dynamic register grouping based on vector extension. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence (AI) and deep learning, the demand for deploying neural network models on various computing devices to perform inference tasks has been increasing, especially on power- and cost-constrained edge computing devices. The RISC-V (Reduced Instruction Set Computer Version 5) architecture, with its open, streamlined, and scalable characteristics, has become a popular choice for edge AI processors.

[0003] To enhance processing power for computationally intensive AI tasks, RISC-V introduced Vector Extensions (RVV). A core feature of RVV is the Configurable Vector Length Multiplier (LMUL). LMUL allows programmers or compilers to group multiple vector registers (e.g., LMUL=1, 2, 4, 8, which can be represented as m1, m2, m4, m8) to process longer data vectors in a single instruction. The choice of LMUL is crucial to AI inference performance, among which:

[0004] Large LMUL (e.g., LMUL=8): Suitable for element-wise operations with low register pressure and simple computation, such as ReLU (Rectified Linear Unit) and Add, which can maximize data throughput.

[0005] Small LMULs (e.g., LMUL=1, LMUL=2): Suitable for computationally intensive operations with high register pressure and complex data dependencies, such as Conv2D (two-dimensional convolution) or MatMul (matrix multiplication), because these operations need to hold a large amount of input data (e.g., sliding window), weights, and accumulated values ​​in registers at the same time.

[0006] However, existing AI model compilation technologies suffer from the following serious technical challenges when optimizing LMULs for the RISC-V vector platform:

[0007] 1. The compiler lacks operator-level semantic awareness.

[0008] Existing general-purpose compilers mostly perform loop-level optimizations on lower-level intermediate representations, making it difficult to identify the characteristics of higher-level operators such as Conv2D and Softmax (activation function). They are unable to select appropriate LMUL and register allocation strategies for different operators based on attributes such as register pressure and data reuse, resulting in LMUL selection deviating from operator requirements.

[0009] 2. Optimization fragmentation leads to inconsistent LMULs across operators.

[0010] The compilation process often involves each operator performing its own local optimum, without unifying the LMUL at the computation graph level. This can easily introduce additional data rearrangement or format conversion overhead between adjacent operators, or force downstream operators to execute with suboptimal LMULs.

[0011] 3. Poor support for AI features such as quantization.

[0012] Existing general-purpose compilers have limited awareness of the complex dependencies between high-level quantization strategies, low-level LMUL selections, and specialized instruction sets, making it difficult to generate optimal quantization inference code. Summary of the Invention

[0013] To overcome the shortcomings of the prior art, this invention provides a register dynamic grouping method and system based on vector extension. It can overcome the defects of existing compilers in operator-level semantic awareness and local optimization. From a global perspective, it combines the high-level semantic characteristics of AI operators to intelligently select and negotiate the optimal LMUL configuration for the RISC-V vector platform, realize register dynamic grouping, eliminate data format conversion overhead, and maximize end-to-end AI inference performance.

[0014] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0015] The first aspect of this invention provides a register dynamic grouping method based on vector extension.

[0016] The register dynamic grouping method based on vector extension includes the following steps:

[0017] Based on the native LMUL preferences of neural network operators, a database of LMUL characteristics of neural network operators is constructed.

[0018] Receive the neural network computation graph, identify the fusionable operator clusters within it, and query the native LMUL preferences of the operators in the corresponding operator cluster in the LMUL feature database;

[0019] A cost assessment model is constructed, and the cost of various LMUL configuration schemes of operators in the operator cluster is calculated by combining the native LMUL preferences of operators in the operator cluster. Then, the optimal LMUL configuration scheme is determined and the register dynamic grouping is completed.

[0020] The computation graph is annotated with the LMUL using the optimal LMUL configuration scheme to form an optimized graph with LMUL annotations;

[0021] The system parses the optimization graph annotated with LMUL and adjusts the memory layout and block strategy in coordination with the LMUL annotations to finally generate executable code.

[0022] A second aspect of the present invention provides a register dynamic grouping system based on vector extension.

[0023] A register dynamic grouping system based on vector extension includes:

[0024] The database construction module is configured to: build a database of LMUL features of neural network operators based on their native LMUL preferences;

[0025] The preprocessing module is configured to: receive the neural network computation graph, identify the fusionable operator clusters within it, and query the native LMUL preferences of the operators in the corresponding operator clusters in the LMUL feature database;

[0026] The cost negotiation module is configured to: construct a cost assessment model, combine the native LMUL preferences of operators in the operator cluster, calculate the cost of multiple LMUL configuration schemes of operators in the operator cluster, and then determine the optimal LMUL configuration scheme to complete the dynamic grouping of registers;

[0027] The annotation optimization module is configured to annotate the computation graph with the LMUL using the optimal LMUL configuration scheme, forming an optimized graph with LMUL annotations;

[0028] The collaborative adjustment module is configured to parse the optimization graph with LMUL annotations, and collaboratively adjust the memory layout and block strategy according to the LMUL annotations to finally generate executable code.

[0029] The above one or more technical solutions have the following beneficial effects:

[0030] This invention provides a register dynamic grouping method and system based on vector extension. By introducing an operator LMUL feature database, the compiler is equipped with high-level semantic awareness of AI operators for the first time. This makes the LMUL selection no longer based on mechanical guessing of the underlying loop, but on the inherent data flow and register pressure characteristics of the operator, fundamentally improving the accuracy of optimization and overcoming the compiler's lack of operator-level semantic awareness.

[0031] The LMUL negotiation algorithm proposed in this invention can achieve optimal register grouping, elevating the optimization perspective from a single operator to the computational graph level of cross-operator fusion. It effectively resolves LMUL conflicts at operator boundaries by balancing different schemes using a quantized cost model, avoiding local optimum traps and achieving end-to-end global performance optimization.

[0032] This invention, through negotiation and propagation of LMULs, tends to generate fusion operator kernels with consistent LMULs, thereby eliminating the expensive data rearrangement or memory overflow (Spill / Reload) overhead introduced between operators due to LMUL mismatch, significantly reducing inference latency and eliminating data transformation overhead.

[0033] This invention uses the selection of LMULs as a high-level compilation instruction, establishing a connection between it and downstream optimizations such as memory layout and partitioning strategies. This collaborative optimization ensures that the arrangement of data in memory and the partitioning strategy of computational units are matched with the configuration of vector instructions (LMULs) from the outset, further squeezing out hardware performance and enhancing compilation coherence.

[0034] The database and negotiation algorithm of this invention can be easily extended to support specific AI scenarios such as INT8 quantization, thus enhancing support for AI features. For example, the database can specify the special requirements of the INT8 vdot instruction for LMUL, and the negotiation algorithm uses this as a strong constraint to generate optimal quantization inference code.

[0035] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 This is a flowchart of the method in Example 1.

[0038] Figure 2 The Conv2D operator and ReLU operator are shown in Example 1 before optimization.

[0039] Figure 3 The optimized Conv2D and ReLU operators are shown in Example 1.

[0040] Figure 4 This is a system structure diagram of Example 2. Detailed Implementation

[0041] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0042] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0043] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0044] Terminology and Glossary:

[0045] (1) RISC-V: refers to "Reduced Instruction Set Computer Version 5", an open standard instruction set architecture (ISA).

[0046] (2) RVV: Refers to RISC-V Vector Extension, which is a standard extension module of the RISC-V instruction set architecture. In this invention, RVV provides instruction and hardware support for performing Single Instruction Multiple Data (SIMD) type operations, including but not limited to vector arithmetic logic operations, vector memory access, and vector register configuration management.

[0047] (3) LMUL: LMUL is a key concept in the RVV extension. It is a vector length multiplier used to combine multiple physical vector registers into a larger logical vector register, thus allowing programmers to handle longer vectors.

[0048] (4) Conv2D: Refers to a mathematical operation applied to two-dimensional or higher-dimensional input data (such as image tensors). This operation involves a convolution kernel sliding across the input data and calculating the dot product between the convolution kernel and the corresponding region of the input data at each position.

[0049] (5) MatMul: refers to a mathematical operation in which two matrices are multiplied to produce a third matrix.

[0050] (6) Softmax: refers to an activation function or mathematical function that takes a real vector as input and normalizes it to a probability distribution. Specifically, it is achieved by exponentializing each element of the input vector and dividing by the sum of all exponentialized elements.

[0051] (7) ReLU: Refers to the Rectified Linear Unit, a non-linear activation function. For any input value x, the output of the ReLU function is max(0,x).

[0052] (8) LayerNorm: Refers to a normalization technique that normalizes the inputs of all neurons within a single training sample layer. Specifically, it calculates the mean and variance of all inputs within that layer and uses these statistics to scale and translate the inputs.

[0053] (9) vfloat32m2_t: Refers to a specific vector data type used in RVV vector extension C inline functions. This type represents a vector whose basic elements are 32-bit single-precision floating-point numbers (float), and its configured vector length multiplier LMUL is 2, that is, using two physical vector registers as a logical vector group.

[0054] (10) vfloat32m8_t: Refers to a vector data type of RVV, whose basic element is a 32-bit single-precision floating-point number and LMUL is configured to 8. This configuration groups 8 physical vector registers to support single-instruction operations on extremely long vectors.

[0055] (11) vfloat32m4_t: refers to a vector data type of RVV, whose basic elements are 32-bit single-precision floating-point numbers and LMUL is configured to 4.

[0056] (12) vint8m1_t vs vint8m8_t: These refer to two vector data types of RVV, both of which are based on 8-bit signed integers (INT8). The difference lies in the configuration of the vector length multiplier LMUL: LMUL=1 for vint8m1_t, indicating a standard, ungrouped vector register; while LMUL=8 for vint8m8_t, indicating a long vector register group composed of 8 physical vector registers.

[0057] (13) INT8: refers to 8-bit integer data.

[0058] (14) INT8 vdot: Refers to an instruction or operation that calculates the dot product between two INT8 data vectors. This operation typically involves multiplying corresponding elements of the two vectors element by element and then accumulating all the multiplications into a higher-precision accumulator.

[0059] (15) VLEN: The bit width of a single vector register, typically 128 / 256 / 512.

[0060] (16) SEW: Standard Element Width, such as 8 / 16 / 32 / 64.

[0061] (17)LMUL∈{1,2,4,8}: Vector register grouping multiple, LMUL=k means that k groups of registers are used as a "vector register group" to participate in a single vector operation.

[0062] (18) : When LMUL = 1, it is the maximum number of elements that can be processed by a single vector instruction at one time; Denotes rounding down.

[0063] (19) VL: The current effective vector length set by the vsetvl instruction, 0 < VL ≤ VLMAX / LMUL, where VLMAX is the maximum current effective vector length.

[0064] (20) R tot (LMUL): The total number of general vector registers visible to the operator under a given LMUL (after RVV reserves dedicated registers). For example, if RVV has a total of 32 vector registers, then R tot (8) ≈ R tot (1) / 8, R tot (1) represents the total number of general vector registers actually available to the operator on the current hardware (after deducting the registers reserved for system / special purposes) when LMUL = 1, that is, when register grouping amplification is not performed.

[0065] (21) C_comp, C_mem, P_spill: Respectively represent the cost coefficients of computation, memory access, and unit spill (counted in cycles or number of instructions).

[0066] Example 1

[0067] This example discloses a method for dynamic register grouping based on vector extension.

[0068] As Figure 1 shown, the method for dynamic register grouping based on vector extension includes the following steps:

[0069] Based on the native LMUL preference of the neural network operator, construct a LMUL characteristic database for the neural network operator;

[0070] Receive the neural network computation graph, identify the fusible operator clusters therein, and query the native LMUL preferences of the operators in the corresponding operator clusters in the LMUL characteristic database;

[0071] Construct a cost evaluation model, combine the native LMUL preferences of the operators in the operator cluster, calculate the costs of various LMUL configuration schemes for the operators in the operator cluster, and then determine the optimal LMUL configuration scheme to complete dynamic register grouping;

[0072] Annotate the computation graph with the LMUL of the optimal LMUL configuration scheme to form an optimized graph with LMUL annotations;

[0073] Parse the optimized graph with LMUL annotations, and jointly adjust the memory layout and block strategy according to the annotated LMUL, and finally generate executable code.

[0074] The vector-extended register dynamic grouping method provided in this embodiment can be applied to the compilation process of AI neural network models to achieve global optimization of RISC-V vector LMUL (vector length multiplier) and thus achieve optimal register dynamic grouping. A detailed technical solution is described below:

[0075] Step 1: Construct the operator LMUL property database.

[0076] First, an operator LMUL characteristic database is pre-built. This database analyzes and stores the inherent computational characteristics of key operators in AI neural networks (such as Conv2D, ReLU, LayerNorm, etc.). These characteristics include, but are not limited to, the typical register pressure and data reuse patterns of the operator during execution, and from these, the "native LMUL preference" of the operator under RISC-V vector extension is derived. For example, element-wise operators such as ReLU have extremely low register pressure and therefore prefer large LMUL values ​​(e.g., LMUL=8); while Conv2D, due to its complex sliding window and weight loading, has high register pressure and therefore prefers small LMUL values ​​(e.g., LMUL=1 or LMUL=2).

[0077] The native LMUL preference of a neural network operator refers to the LMUL configuration that allows the operator to achieve maximum computational throughput when considering only the operator's own computational characteristics (such as register pressure and data reuse) without regard to context.

[0078] The specific method for obtaining native LMUL preferences is as follows:

[0079] Based on the mathematical structure of the operator, memory access and register requirements, the register pressure and data reuse of the operator are derived.

[0080] Based on operator register pressure and data reuse, the native LMUL preference of the operator under RISC-V vector extension is derived.

[0081] It can be understood that the LMUL feature database stores the mapping relationship between neural network operators and their corresponding native LMUL preferences.

[0082] Step 2: Execute the computation graph-based LMUL propagation and negotiation algorithm.

[0083] In this step, we first receive an AI neural network computation graph to be optimized, consisting of multiple operator nodes. We traverse this computation graph to identify clusters of adjacent operator nodes that can be fused, i.e., fused operator clusters. For example, Conv2D and ReLU are identified as a fused operator cluster.

[0084] The specific recognition algorithm can adopt mature technologies that are already in use, which will not be elaborated here.

[0085] For the identified operator clusters, the LMUL propagation and negotiation algorithm is executed. The execution steps of this algorithm include:

[0086] (1) First, query the native LMUL preferences of each operator in the cluster from the LMUL feature database (e.g., Conv2D preference m2, ReLU preference m8).

[0087] (2) Subsequently, the algorithm quantitatively evaluates multiple LMUL configuration schemes based on a preset cost model, wherein the configuration schemes include at least:

[0088] Option A (Propagating LMUL): Merge all operators within an operator cluster into a single kernel, determine the dominant operator among multiple operators within the operator cluster, and force the use of the dominant operator's (e.g., Conv2D) LMUL (e.g., LMUL=2).

[0089] Option B (Insertion Transformation): Maintain the independence of each operator within the operator cluster, use the optimal LMUL for each operator (e.g., 2 and 8), but insert data format transformation operations between them;

[0090] Option C (Compromise LMUL): All operators within an operator cluster use a single compromise LMUL (e.g., 4).

[0091] (3) The algorithm estimates the number of instruction cycles or memory overhead of different schemes and selects the LMUL configuration scheme with the lowest global cost as the optimal LMUL configuration scheme;

[0092] (4) Use the LMUL of the optimal LMUL configuration scheme (for example, select scheme A and use LMUL=2 uniformly) to annotate the optimized computation graph to obtain the optimized graph with LMUL annotation.

[0093] Step 3: Perform collaborative code generation.

[0094] In this step, a collaborative code generator receives the optimized graph annotated with LMUL. Based on the LMUL annotations on the nodes, the code generator collaboratively adjusts the downstream compilation strategy. Specifically, the choice of LMUL directly guides:

[0095] Memory layout strategy: For example, if the negotiated LMUL is m4, then when allocating memory for a tensor, the inner dimension length to be vectorized will be aligned first. Integer multiples (here) , representing the determined optimal LMUL), to match vector loading instructions and reduce mask overhead.

[0096] Tiling Strategy: For operators such as MatMul, the optimal inner loop block size is derived in reverse based on the negotiated LMUL result (e.g., m2) to maximize the data utilization of vfloat32m2_t type.

[0097] Here, existing mature technologies can be used to determine the size of the inner loop block based on LMUL.

[0098] Finally, the code generator instantiates intrinsic code that matches the LMUL from the operator template library, generating the final high-performance executable.

[0099] Intrinsic code refers to a functional interface provided at the level of a high-level programming language (such as C) that can be directly mapped to specific processor hardware instructions (in this case, RISC-V vector instructions). This code is not a regular function call. When the compiler processes intrinsic code, it expands it directly and replaces it with one or more efficient low-level assembly instructions, thus avoiding the additional overhead of traditional function calls.

[0100] More specifically, in this embodiment, the LMUL characteristic database of the operator can be generated and maintained in two ways:

[0101] (1) Analytical modeling: Based on the mathematical structure of the operator, memory access and register requirements, the register pressure and data reuse are derived.

[0102] (2) Empirical analysis: Benchmark tests are conducted on the target RVV with different LMUL and tensor sizes to measure metrics such as cycles / byte and bytes / output. Cycles / byte and bytes / output are two key performance indicators used to measure computational efficiency and memory access efficiency, and are often used in hardware performance benchmark tests.

[0103] cycles / byte: Measures the computational resources required to process a unit of data.

[0104] bytes / output: Measures the memory access bandwidth required to generate a unit of output.

[0105] The database in this embodiment of the invention not only records the operator's native LMUL preferences, but also records constraint and analysis data in a structured form for subsequent use in the cost model.

[0106] Furthermore, in the mathematical modeling of the LMUL propagation and negotiation algorithm:

[0107] The neural network computation graph is modeled as a DAG G=(V,E), where DAG represents a directed acyclic graph, used to represent the inference operator dependency structure processed in the compilation phase of this invention; G represents the entire computation graph, denoted as G=(V,E), which includes the operators participating in LMUL negotiation and necessary auxiliary and transformation nodes; V represents the set of nodes, each node representing an operator; E represents the set of edges, each directed edge (u→v) indicates that the output of u is used by v, forming a data dependency.

[0108] For each node n∈V, define an LMUL variable. Where L(n) is the candidate set, This represents the LMUL variable for the nth node, where n represents the nth node;

[0109] For each node n∈V, define an LMUL variable. L(n) is the candidate set (obtained by the intersection of the feature database native_lmul and the hardware upper limit; the feature database native_lmul refers to the values ​​of the "native_lmul" field in the feature database (a set, which is the native LMUL preference set of the operator). This represents the LMUL variable for the nth node, where n represents the nth node;

[0110] Considering computational and memory access overhead, and taking into account register overflow, define node n in... The execution cost here for The abbreviation, or more precisely It corresponds to a specific value n. Specific value ;

[0111] Calculate the transition cost when adjacent nodes have inconsistent LMUL values;

[0112] By combining execution costs and transition costs, a cost assessment model is obtained.

[0113] Furthermore, define the objective function (cost assessment model):

[0114]

[0115] in:

[0116] For node n in The execution cost, including computation and memory access overhead and considering register overflow:

[0117]

[0118] here, Indicates the calculation cost; Indicates the cost of accessing memory; To estimate the number of vector registers required, Cost coefficient per unit of overflow; This represents the total number of general-purpose vector registers visible to the operator under a given LMUL.

[0119] The transition cost (format conversion / shuffling / save back - reload) when adjacent nodes have inconsistent LMUL values. Represents nodes in the computation graph Pointing to node A directed edge is formed where the output of operator u is used as the input of operator v, thus creating a data dependency. The cost term only includes the conversion cost required if the LMUL values ​​of the two endpoints are inconsistent for such directed dependent edges. Represents a node LMUL variables; Represents a node LMUL variable.

[0120]

[0121] in, for abbreviation; and These represent the derived number of shuffle instructions and the number of extra memory bytes, respectively. These are hardware-related weights.

[0122] It is understandable that the aforementioned transition costs arise from format conversion, shuffling, or save-and-reload.

[0123] Furthermore, the cost calculation method is as follows:

[0124]

[0125] in, For floating-point / integer arithmetic quantities, For kernel instruction parallelism, The utilization rate is due to the structure / mask / tail processing. This indicates the current valid vector length set by the vsetvl command.

[0126] The method for calculating memory access costs is as follows:

[0127]

[0128] in:

[0129] Indicates: Node n in a given context Under the implementation, the total number of bytes of vector loading and storage data required to complete one full computation of the operator can be derived from the operator shape and block strategy.

[0130] This indicates the effective bandwidth (bytes / cycle or bytes / second) that the target hardware can continuously provide at the corresponding memory level (such as on-chip bus / external memory).

[0131] This indicates that the additional memory access overhead correction term is introduced due to factors such as data alignment, padding, and boundary tail block processing. It is used to cover the costs of additional access, masking operations, or invalid loading caused by non-ideal alignment.

[0132] Overflow and Shuffling: When At that time, the intermediate results are written back / reloaded and estimated according to the overflow share; the shuffle cost is estimated based on the RVV vector rearrangement instruction latency model.

[0133] Furthermore, when adjusting the memory layout and block strategy in coordination with the LMUL annotations to finally generate executable code:

[0134] Memory layout: operator annotations If the data type is SEW, then padding is performed on the channel or the innermost dimension to align the dimension length to the specified value. Integer multiples of; generate aligned access sequences and reduce mask branches.

[0135] Block / microkernel: For matrix multiplication, prioritize aligning the block size in the column direction with the vector length, and keep the number of parallel accumulations within the capacity of the register; for 2D convolution, prioritize selecting blocks in the output channel and row direction, so that the input sliding window, weights and accumulation results required for one calculation can be kept in the register at the same time, thereby avoiding overflow and repeated loading.

[0136] Template instantiation: It adopts the approach of "microkernel template + RVV built-in functions" to automatically switch vector data types and instruction combinations according to the negotiated LMUL, covering steps such as loading, multiplication and addition, reduction and tail processing, so as to ensure that the generated code under different LMULs is consistent, maintainable and efficient.

[0137] To better understand the method of this embodiment, a specific application is provided below. This specific application includes the following steps:

[0138] Step 1: Construct the operator LMUL property database.

[0139] This invention pre-constructs an operator LMUL characteristic database. This database is established through in-depth intrinsic-level analysis and performance profiling of mainstream AI operators, and its core function is to store the mapping relationship between AI operators and their native LMUL preferences.

[0140] As mentioned earlier, native LMUL preference refers to the LMUL configuration that maximizes computational throughput when considering only the operator's own computational characteristics (such as register pressure and data reuse) without regard to context. From the perspective of register pressure, operators are divided into high register pressure operators and low register pressure operators.

[0141] (1) High register pressure operators: such as Conv2D (3x3 convolution kernel).

[0142] For Conv2D (3x3 convolution kernels), when computing an output point, a 3x3 input feature map "sliding window" needs to be loaded, along with the corresponding 3x3 weights. If multiple output channels are accumulated simultaneously, the registers will quickly fill up. Therefore, this type of operator cannot withstand the doubling of register grouping caused by large LMULs (such as m8) (i.e., the actual number of usable registers is reduced to 1 / 8). Its native preference is for small LMULs, such as m1 or m2.

[0143] (2) Low register pressure operators: such as ReLU or Add (element-by-element operation).

[0144] ReLU or Add (element-wise operation) requires loading only 1 or 2 input vectors and writing 1 output vector at any given time to compute a vector slice, resulting in extremely low register pressure. Therefore, these operators inherently prefer to use the largest LMUL (e.g., m8) to process as much data as possible in a single instruction, maximizing throughput. This LMUL feature database can be a table, a JSON file, or an embedded database. An example of a table-based LMUL feature database is shown in Table 1.

[0145] Table 1: LMUL Feature Database in Tabular Format

[0146]

[0147] Step 2: Execute the LMUL propagation and negotiation algorithm.

[0148] When the AI ​​compiler receives a neural network computation graph to be optimized (e.g., converted from an ONNX or TensorFlow model, where ONNX is an open intermediate representation and file format for neural network models, often used as input to the "unified computation graph" exported by the front-end training framework; TensorFlow is a mainstream deep learning framework that provides model building, training, and inference capabilities), it executes the LMUL propagation and negotiation algorithm, which aims to solve the problem of "local optima and global suboptima".

[0149] Reference Figure 2 and Figure 3 The images show the Conv2D operator before and after optimization, respectively. Taking the Conv2D and ReLU operators as examples, the specific workflow is explained below:

[0150] (1) Input and recognition: Receive computation graph.

[0151] Figure 2 The example shown is a typical fragment before optimization: a Conv2D operator followed by a ReLU operator. It is identified that the Conv2D and ReLU operators are computationally continuous and fusionable operator clusters.

[0152] (2) Query preferences: Query the database to obtain:

[0153] Conv2D's preference is LMUL=2, while ReLU's preference is LMUL=8. An LMUL configuration conflict has occurred.

[0154] (3) LMUL Negotiation (Cost Assessment): At this point, the LMUL propagation and negotiation algorithm is initiated to assess the cost of different solutions based on the cost model described above. This cost can be quantified as the estimated number of execution cycles or the number of additional instructions introduced by data rearrangement.

[0155] Option A (Propagating LMUL): Merge Conv and ReLU into a large loop, and force the use of Conv's LMUL, i.e., m2, throughout the loop.

[0156] Cost Analysis: The ReLU portion does not operate on its optimal m8, resulting in a loss in throughput. However, the Conv portion operates at its optimal state. There is no data exchange overhead between the two.

[0157] Option B (insertion transformation): Maintain independence between the two, Conv uses m2 output, and ReLU uses m8 input.

[0158] Cost analysis: such as Figure 2As shown, an expensive data format conversion / reordering operation must be inserted between the Conv2D and ReLU operators. This typically involves writing data of type vfloat32m2_t back to memory (Spill) and then reloading it as type vfloat32m8_t (Reload), or performing a series of complex vector register shuffle instructions. This overhead is a significant performance bottleneck in the current technology.

[0159] Option C (Compromise LMUL): Both use a "middle" LMUL, such as m4.

[0160] Cost analysis: The Conv part may experience a small amount of overflow due to increased register pressure (half the number of available registers in m4), and the performance of the ReLU part is not optimal.

[0161] (4) Decision and annotation: Through quantitative evaluation, the negotiation algorithm found that for the common pattern of Conv-ReLU, which is a computationally intensive operator followed by an element-wise operator, the cost of scheme A (propagating LMUL) (i.e. the performance loss of ReLU) is much lower than that of scheme B (data rearrangement) and scheme C.

[0162] Therefore, the decision is to choose option A. By implementing option A, the computation graph is modified as follows: Figure 3 The optimized form is shown. In Figure 3 middle:

[0163] Conv2D and ReLU are merged into a single, larger operator node Fused_Conv_ReLU.

[0164] Add an "LMUL annotation" to this new node Fused_Conv_ReLU, i.e., LMUL=m2.

[0165] The original data rearrangement overhead has been completely eliminated.

[0166] (5) Output: Output an optimized computation graph with LMUL annotation.

[0167] exist Figure 2 and Figure 3 In this context, Input represents input and Output represents output.

[0168] Step 3: Perform collaborative code generation.

[0169] Finally, based on this annotated optimized computation graph, the final RISC-V executable code is generated.

[0170] Collaboration is key to this step. The LMUL annotation is no longer a suggestion, but a high-level compiler directive used to guide other downstream optimization steps, specifically including:

[0171] Collaboration 1: Guide memory layout (Data Layout).

[0172] When allocating memory for tensors in a graph, the code generator reads the LMUL annotations of their downstream operators.

[0173] For example, if a tensor's consumer operator is annotated as LMUL=m4 and its data type is float32 (SEW=32), this means that the operator expects to be processed as vfloat32m4_t on each vector load. elements (of which) ).

[0174] "Consumer operator" refers to the downstream operator node in the computation graph that directly takes the tensor as input, i.e., the direct user of the tensor. The LMUL annotation on the node defines the alignment, padding, and vector access step size strategies used for the tensor when generating code; if the same tensor is consumed by multiple downstream operators and their LMUL preferences are inconsistent, necessary format conversions will be unified or inserted during the aforementioned LMUL negotiation phase.

[0175] Floating-point number (float): A computer data type used to represent numerical values ​​with decimals. Its name comes from the fact that the position of the decimal point ("floating") is determined by an exponent.

[0176] Single-precision floating-point number (float32): Specifically refers to a floating-point number that uses 32 binary bits for storage.

[0177] To ensure the most efficient vector loading, the code generator tends to pad the tensor along the channel dimension of the data when allocating it, aligning its total size to the specified dimensions. Integer multiples of (where) and ).

[0178] This ensures that the boundary handling of vector loops is kept to a minimum, avoiding expensive mask loading operations caused by data misalignment.

[0179] Collaboration 2: Guiding Tiling Strategy.

[0180] For operators like MatMul (matrix multiplication), the choice of LMUL is flexible (see Table 1).

[0181] If the upstream LMUL negotiation algorithm, based on global considerations, annotates a MatMul node with LMUL=m2, then the code generator, when implementing the loop block for that MatMul, will choose a block size that maximizes the utilization of the vfloat32m2_t data type (e.g., the unroll factor of an inner product loop). This ensures that high-level LMUL decisions match the lowest-level loop implementation.

[0182] Collaboration 3: Instantiating code.

[0183] Finally, based on the annotation LMUL=m2, the code generator retrieves the code version of the Fused_Conv_ReLU operator optimized for m2 from the pre-built operator intrinsic template library, fills in the block and memory layout parameters calculated in the previous step, and generates the final high-performance RISC-V assembly code.

[0184] This invention addresses the problem of compilers lacking operator-level semantic awareness by using an operator LMUL feature database; it resolves the issue of inconsistent LMUL values ​​across operators by using an LMUL negotiation algorithm, eliminating data rearrangement overhead; and it maximizes end-to-end AI inference performance by linking high-level LMUL decisions with underlying memory layout and loop block optimization through collaborative code generation.

[0185] Example 2

[0186] This embodiment discloses a register dynamic grouping system based on vector extension.

[0187] like Figure 4 As shown, the register dynamic grouping system based on vector extension includes:

[0188] The database construction module is configured to: build a database of LMUL features of neural network operators based on their native LMUL preferences;

[0189] The preprocessing module is configured to: receive the neural network computation graph, identify the fusionable operator clusters within it, and query the native LMUL preferences of the operators in the corresponding operator clusters in the LMUL feature database;

[0190] The cost negotiation module is configured to: construct a cost assessment model, combine the native LMUL preferences of operators in the operator cluster, calculate the cost of multiple LMUL configuration schemes of operators in the operator cluster, and then determine the optimal LMUL configuration scheme to complete the dynamic grouping of registers;

[0191] The annotation optimization module is configured to annotate the computation graph with the LMUL using the optimal LMUL configuration scheme, forming an optimized graph with LMUL annotations;

[0192] The collaborative adjustment module is configured to parse the optimization graph with LMUL annotations, and collaboratively adjust the memory layout and block strategy according to the LMUL annotations to finally generate executable code.

[0193] The system provided in this embodiment can be a compiler or compiler plugin. Its database construction module stores characteristic information such as the native LMUL preferences and register pressure of key AI operators under RISC-V vector extensions. The cost negotiation module and annotation optimization module receive the AI ​​computation graph, identify fusionable operator clusters, query the database, and run a cost evaluation model to determine the globally optimal LMUL configuration scheme and output an optimized graph with LMUL annotations. The collaborative adjustment module parses the optimized graph with LMUL annotations and collaboratively adjusts the memory layout and block strategy according to the annotated LMULs, ultimately generating the target executable code.

[0194] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0195] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for dynamic grouping of registers based on vector extension, the method comprising: The method comprises the following steps: constructing an LMUL characteristic database of the neural network operator based on the native LMUL preference of the neural network operator; receiving a neural network computation graph, identifying a fusible operator cluster therein, and querying the native LMUL preference of the operator in the corresponding operator cluster in the LMUL characteristic database; constructing a cost evaluation model, combining the native LMUL preference of the operator in the operator cluster, and calculating the cost of various LMUL configuration schemes of the operator in the operator cluster to determine the optimal LMUL configuration scheme and complete dynamic grouping of registers; annotating the computation graph using the LMUL of the optimal LMUL configuration scheme to form an optimized graph with LMUL annotation; parsing the optimized graph with LMUL annotation and adjusting the memory layout and blocking strategy according to the annotated LMUL to finally generate executable code; the native LMUL preference of the neural network operator refers to the LMUL configuration that can achieve the maximum computation throughput of the operator without considering the context and only considering the computation characteristics of the operator itself; The specific acquisition method of the native LMUL preference is: deduce the register pressure and data multiplexing degree of the operator according to the mathematical structure, memory access and register demand of the operator; deduce the native LMUL preference of the operator under the RISC-V vector extension according to the register pressure and data multiplexing degree of the operator; The LMUL characteristic database stores the mapping relationship between the neural network operator and the corresponding native LMUL preference.

2. The method of claim 1, wherein, The various LMUL configuration schemes of the operator in the operator cluster specifically include: propagation LMUL: fuse all operators in the operator cluster into one kernel, determine the dominant operator among the multiple operators in the operator cluster, and forcibly use the LMUL of the dominant operator; insertion conversion: maintain the independence of each operator in the operator cluster, use the optimal LMUL of each operator, but insert a data format conversion operation between adjacent operators in the operator cluster; compromise LMUL: each operator in the operator cluster uses a compromise LMUL.

3. The method of claim 1, wherein, The specific construction process of the cost evaluation model is: model the neural network computation graph as a DAG, where DAG represents a directed acyclic graph; G represents the entire computation graph, denoted as G=(V,E), including operators participating in LMUL negotiation and necessary auxiliary and conversion nodes; V represents a node set, each node representing an operator; E represents an edge set; For each node n e V, define the LMUL variable where L(n) is the candidate set, denotes the LMUL variable for the n-th node, n denotes the n-th node; Considering the computation and memory access overheads, and considering register spilling, define the execution cost of a node n at level l as Cn= Cn+ Cn+ Cn+ Cn calculate the transition cost when the LMUL of adjacent nodes is inconsistent; combine the execution cost and the transition cost to obtain the cost evaluation model.

4. The method of claim 3, wherein, The cost evaluation model is specifically represented as: ； ； ； wherein, is the execution cost of node n at ; is the transition cost when the LMUL of adjacent nodes is not consistent; represents a directed edge from node to node ; represents the LMUL variable of node ; represents the LMUL variable of node ; represents the computation cost; represents the memory access cost; represents the cost coefficient of a unit spill; is the estimated number of vector registers needed; represents the total number of general-purpose vector registers visible to the operator at a given LMUL; is the number of vector registers needed for the operator; is a short-hand for with are the derived number of shuffle instructions and the additional memory bytes, respectively, and are hardware-dependent weights; is a shorthand notation.

5. The method of claim 4, wherein, The specific calculation method of the calculation cost is: ； wherein, is the floating point / integer arithmetic quantity, is the intra-core instruction parallelism, is the utilization due to structure / mask / tail processing; represents the current active vector length set by the vsetvl instruction; The specific calculation method of the memory access cost is: ； wherein, represents the total number of data bytes required for vector load and store to complete one full computation of the operator under the implementation of the node n for the given represents the total number of data bytes required for vector load and store to complete one full computation of the operator under the implementation of the node n for the given represents the effective bandwidth that the target hardware can sustainably provide on the corresponding memory hierarchy; represents the additional introduced memory access overhead correction term.

6. The method of claim 1, wherein the vector extension based register dynamic grouping is performed by a processor. After calculating the cost of the various LMUL configuration schemes of the operators in the operator cluster, select the configuration scheme with the lowest cost as the optimal LMUL configuration scheme.

7. The method of claim 1, wherein the vector extension based register dynamic grouping is performed by a processor. Parsing the optimized graph with LMUL annotation and adjusting the memory layout and blocking strategy according to the annotated LMUL to finally generate executable code, specifically including: when allocating memory for a tensor, align the inner dimension length to be vectorized to an integer multiple of the annotated LMUL to match the vector load instruction and reduce the mask overhead; Deduce the optimal inner loop blocking size according to the annotated LMUL; The code generator instantiates the code matching the LMUL from the operator template library to generate an executable file.

8. A register dynamic grouping system based on vector extension, characterized in that, Comprise: The database construction module is configured to construct the LMUL characteristic database of the neural network operator based on the native LMUL preference of the neural network operator; The preprocessing module is configured to receive the neural network computation graph, identify the fusible operator cluster therein, and query the native LMUL preference of the operators in the corresponding operator cluster in the LMUL characteristic database; The cost negotiation module is configured to construct a cost evaluation model, combine the native LMUL preference of the operators in the operator cluster, calculate the cost of various LMUL configuration schemes of the operators in the operator cluster, and then determine the optimal LMUL configuration scheme to complete dynamic grouping of registers; The annotation optimization module is configured to annotate the computation graph using the LMUL of the optimal LMUL configuration scheme to form an optimized graph with LMUL annotation; The collaborative adjustment module is configured to parse the optimized graph with LMUL annotation and collaboratively adjust the memory layout and blocking strategy according to the annotated LMUL to finally generate executable code; The native LMUL preference of the neural network operator refers to the LMUL configuration that can achieve the maximum computing throughput of the operator without considering the context and only considering the computing characteristics of the operator itself; The specific acquisition method of the native LMUL preference is: According to the mathematical structure, memory access and register demand of the operator, deduce the register pressure and data multiplexing degree of the operator; According to the register pressure and data multiplexing degree of the operator, deduce the native LMUL preference of the operator under the RISC-V vector extension; The LMUL characteristic database stores the mapping relationship between the neural network operator and the corresponding native LMUL preference.

Images