RISC-V model reasoning acceleration method and device based on irregular convolution operator adaptive distribution, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an RISC-V model reasoning acceleration method and device based on irregular convolution operator adaptive distribution, equipment and a medium. A selection strategy of various convolution algorithm implementation logics in the original operator distribution codes is adjusted, and candidate operator distribution codes are generated; performing performance test on the candidate operator distribution codes, and determining target operator distribution codes according to test results; and after updating the original operator distribution code aiming at the target model to the target operator distribution code, running a reasoning application program of the target model. According to the method, the operator distribution logic is adaptively adjusted by jointly utilizing the operator fine-grained shape information and the hardware characteristic information, and deep customized reasoning optimization for different application scenes and hardware environments is realized, so that the model reasoning efficiency is high, the hardware resource utilization rate is high, and the generalization ability is high.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for accelerating RISC-V model inference based on adaptive distribution of irregular convolution operators. Background Technology

[0002] With the development of artificial intelligence technology, deploying deep learning models on edge devices based on the RISC-V architecture has become increasingly common. Here, RISC-V is an open-source instruction set architecture (ISA) based on Reduced Instruction Set Computing (RISC). To improve inference performance, existing deep learning libraries typically provide multiple algorithm implementations for core convolution operators, such as implementations based on general matrix multiplication, Winograd algorithms, or direct convolution algorithms. They then select between these implementations using a fixed-rule operator distribution method to balance computational and memory access costs.

[0003] However, existing rule-based operator distribution methods have certain drawbacks in practical applications. First, these methods typically ignore the fine-grained irregular shape information (such as spatial dimension fragmentation and channel dimension irregularities) commonly found in edge-side models of convolution operators. This makes it difficult for the selected algorithm to adapt to the actual data shape, resulting in low computational and memory access efficiency. Second, existing methods cannot adaptively perceive the characteristics of different RISC-V hardware platforms, limiting the utilization of hardware performance. This leads to low execution efficiency and insufficient memory bandwidth utilization of irregular convolution operators in deep learning models on specific RISC-V hardware platforms, thus slowing down the inference speed of the entire model. Summary of the Invention

[0004] This invention provides a RISC-V model inference acceleration method, apparatus, device, and medium based on adaptive distribution of irregular convolution operators, which solves the problem that when existing irregular convolution operators are distributed among different algorithm implementations, they are easily distributed to algorithm implementations with suboptimal performance, resulting in low execution efficiency and poor model inference performance on specific hardware architectures.

[0005] This invention provides a method for accelerating RISC-V model inference based on adaptive distribution of irregular convolution operators, comprising: Obtain fine-grained shape information of the convolution operator in the target model to be accelerated, as well as hardware characteristic information of the target hardware platform; The original operator distribution code of the convolution operator is obtained, and the selection strategy of multiple optional convolution algorithm implementation logics in the original operator distribution code is adjusted based on the fine-grained shape information and the hardware characteristic information to generate candidate operator distribution code. The candidate operator distribution code is subjected to performance testing on the target hardware platform, and the target operator distribution code is determined from the candidate operator distribution code based on the test results. The original operator distribution code for the target model in the deep learning base library is updated to the target operator distribution code, and the inference application of the target model is run on the target hardware platform based on the updated deep learning base library.

[0006] According to the present invention, a method for accelerating RISC-V model inference based on adaptive distribution of irregular convolution operators is provided. The step of obtaining fine-grained shape information of the convolution operators in the target model to be accelerated includes: The original operator distribution code is embedded into the deep learning base library, and the inference application of the deep learning base library and the target model is compiled to generate the original executable program. The original executable program is run on the target hardware platform, and performance data is collected during the running process; Based on the performance data, fine-grained shape information of the convolution operator is extracted. The fine-grained shape information includes the input tensor shape, output tensor shape, convolution kernel size, stride size, padding parameters, and dilation rate.

[0007] According to the present invention, a method for accelerating RISC-V model inference based on adaptive distribution of irregular convolution operators is provided. The target hardware platform is a processor platform that supports vector extension instruction set. The hardware characteristic information includes the memory hierarchy information, actual available memory bandwidth information, and vector register length parameters of the target hardware platform.

[0008] According to the present invention, a method for accelerating RISC-V model inference based on adaptive distribution of irregular convolution operators is provided. The method involves adjusting the selection strategy for multiple optional convolution algorithm implementation logics in the original operator distribution code based on the fine-grained shape information and the hardware characteristic information to generate candidate operator distribution code, including: The fine-grained shape information, the hardware characteristic information, and the original operator distribution code are input as prompt information into the large language model; The large language model is guided to analyze the adaptation relationship between the fine-grained shape information and the implementation logic of the various optional convolution algorithms; The large language model is guided to predict the execution efficiency of each convolution algorithm implementation logic on the target hardware platform based on the adaptation relationship and the hardware characteristic information. Based on the prediction results, the candidate operator distribution code is generated. The candidate operator distribution code is configured to distribute the convolution operator to the convolution algorithm implementation logic with the highest predicted execution efficiency.

[0009] According to the present invention, a method for accelerating RISC-V model inference based on adaptive distribution of irregular convolution operators is provided. The step of performing performance testing on the candidate operator distribution code on the target hardware platform and determining the target operator distribution code from the candidate operator distribution code based on the test results includes: The current candidate operator distribution code generated by the large language model is embedded into the deep learning base library, and the inference application of the deep learning base library and the target model is compiled to generate a test program; Run the test program on the target hardware platform and record the actual execution time of the convolution operator under the current candidate operator distribution code; The actual execution time is compared with the original execution time corresponding to the original operator distribution code. If the reduction in the actual execution time compared to the original execution time exceeds a preset threshold, the current candidate operator distribution code and the corresponding actual execution time are added to the preferred set. The candidate operator distribution code with the shortest actual execution time is selected from the preferred set as the target operator distribution code.

[0010] According to the present invention, a method for accelerating RISC-V model inference based on adaptive distribution of irregular convolution operators is provided, wherein selecting the candidate operator distribution code with the shortest actual execution time from the preferred set as the target operator distribution code includes: Determine whether the number of candidate operator distribution codes in the preferred set has reached a preset number, or whether the number of code generation times of the large language model has reached a preset upper limit; If so, then stop generating new candidate operator distribution codes, and traverse the preferred set to select the candidate operator distribution code with the shortest actual execution time as the target operator distribution code; If not, the performance test results of the current candidate operator distribution code are fed back to the large language model, and the large language model is instructed to regenerate new candidate operator distribution code.

[0011] According to the present invention, a method for accelerating RISC-V model inference based on adaptive distribution of irregular convolution operators is provided. The convolution operator is an irregular convolution operator, and the implementation logic of the various optional convolution algorithms includes at least two of the following: a convolution algorithm based on general matrix multiplication, a Winograd convolution algorithm, a direct convolution algorithm, and a customized convolution algorithm for a convolution kernel of a specified size.

[0012] The present invention also provides a RISC-V model inference acceleration device based on irregular convolution operator adaptive distribution, comprising: The information acquisition unit is used to acquire fine-grained shape information of the convolution operators in the target model to be accelerated, as well as hardware characteristic information of the target hardware platform. The code generation unit is used to obtain the original operator distribution code of the convolution operator, and based on the fine-grained shape information and the hardware characteristic information, adjust the selection strategy of multiple optional convolution algorithm implementation logic in the original operator distribution code to generate candidate operator distribution code; The performance testing unit is used to perform performance testing on the candidate operator distribution code on the target hardware platform, and determine the target operator distribution code from the candidate operator distribution code based on the test results. The model execution unit is used to update the original operator distribution code in the deep learning base library for the target model to the target operator distribution code, and run the inference application of the target model on the target hardware platform based on the updated deep learning base library.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the RISC-V model inference acceleration method based on adaptive distribution of irregular convolution operators as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the RISC-V model inference acceleration method based on adaptive distribution of irregular convolution operators as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the RISC-V model inference acceleration method based on adaptive distribution of irregular convolution operators as described above.

[0016] This invention provides a RISC-V model inference acceleration method, apparatus, device, and medium based on adaptive distribution of irregular convolution operators. By collaboratively utilizing the fine-grained shape information of convolution operators and the hardware characteristics of the target hardware platform, the algorithm selection strategy in the original operator distribution code is specifically adjusted and reconstructed. This optimization method, combining software and hardware information, can uncover convolution algorithm implementation paths with better performance under specific irregular shapes and hardware resource constraints, generating candidate operator distribution code with higher execution efficiency. Furthermore, through field performance testing and screening on the target hardware platform, theoretical prediction bias is eliminated, ensuring that the final target operator distribution code has optimal computation and memory access efficiency in actual operation. This effectively solves the problem of performance limitations caused by the mismatch between irregular operators and hardware characteristics, significantly improving end-to-end model inference speed. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution provided by the present invention. Figure 2 This is a schematic diagram of the architecture of the RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution provided by the present invention; Figure 3 This is a schematic diagram of the RISC-V model inference acceleration device based on adaptive distribution of irregular convolution operators provided by the present invention. Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] To facilitate understanding of the solutions provided by this invention, some technical terms involved in this invention will first be explained.

[0021] RISC-V is an open-source instruction set architecture (ISA) based on the principles of Reduced Instruction Set Computing (RISC).

[0022] Operators: In deep learning libraries for edge devices, operators are code modules that implement specific computational logic. They are a set of basic operations performed on tensors, including arithmetic operations, mathematical functions, and even modifications to tensor metadata, such as dimension compression and dimension modification. Common operators include addition, subtraction, multiplication, division, matrix multiplication, convolution, deconvolution, depthwise separable convolution, and upsampling.

[0023] Operator distribution: In the deep learning libraries of edge devices, there are different algorithm implementations for convolution operators in order to obtain better performance for different convolution operator characteristics and different target hardware. Therefore, operator distribution is to select different algorithm implementations based on the characteristics of convolution operators and target hardware to obtain better performance.

[0024] Irregular convolution operators: Irregular convolution operators are characterized by two dimensions of input and output tensors: ① Spatial dimension fragmentation: The spatial dimension (i.e., width × height) of the input and output tensors of the convolution operator is fragmented both within a single model and across multiple models; ② Channel dimension irregularity: The number of input and output channels of the convolution operator cannot be divided by the packing width of SIMD.

[0025] SIMD (Single Instruction Multiple Data) is a technique that uses a single controller to control multiple processors, simultaneously performing the same operation on each of a set of data (also known as a "data vector"), thereby achieving spatial parallelism. For example, scalar multiplication instructions can only perform multiplication on one pair of data at a time, while SIMD multiplication instructions can perform multiplication on multiple pairs of data simultaneously.

[0026] GEMM (General Matrix Multiplication) is the operation of calculating the product C of matrices A and B. In deep learning, convolution operations are often converted to GEMM using im2col (Image to Column) technology, thereby leveraging hardware optimizations for matrix multiplication, such as the SIMD instructions of the CPU (Central Processing Unit).

[0027] In the Winograd algorithm implementation of convolution operators, "tile" refers to the transform block, input block, and output block.

[0028] best-of-N: Under the same input conditions, the model independently generates N candidate outputs and selects the best one as the final result according to a predefined performance index, so as to achieve a balance between generation quality and computational cost.

[0029] With the rapid development of artificial intelligence technology, deploying deep learning models on edge devices to perform inference tasks has become a mainstream trend. RISC-V, as an open-source instruction set architecture based on the principle of reduced instruction set computing, has been widely used in the field of edge computing due to its flexibility and scalability. In deep learning libraries running on edge devices, convolution operators are the most computationally intensive and time-consuming core components. To achieve better performance in different application scenarios, existing deep learning inference libraries typically provide various algorithmic implementations for convolution operators, such as GEMM-based implementations, Winograd algorithm implementations, direct convolution implementations, and dedicated implementation logic for specific convolution kernel sizes (such as 3×3, 7×7, etc.) or specific vectorized data packaging methods.

[0030] To select the most suitable execution path for the current convolution operator among the various algorithm implementations mentioned above, existing technologies typically employ rule-based operator distribution methods. This method primarily relies on the basic parameters of the convolution operator (such as kernel size, stride, and dilation rate) and the vectorized packing method of the input and output data (such as pack1, packn, etc.), combined with preset fixed rules, to select the algorithm, aiming to balance computational and memory access costs.

[0031] However, this traditional rule-based operator distribution method has significant drawbacks and shortcomings when facing complex real-world application scenarios. First, existing methods fail to fully consider the impact of fine-grained shape information of convolution operators on performance. In practical edge deep learning models, the input and output tensors of convolution operators often exhibit significant irregularities. This irregularity manifests in the high fragmentation of spatial dimensions (width × height) within a single model and across different models, as well as the non-alignment of channel dimensions (e.g., the number of channels cannot be divided by the SIMD packing width). Existing fixed rules often ignore these fine-grained shape features, resulting in the distributed algorithm being unable to effectively utilize hardware resources when faced with irregular shapes. This severely impacts the computational and memory access efficiency of the operators, thereby slowing down the inference speed of the entire model.

[0032] Secondly, existing methods lack the ability to adaptively perceive the specific hardware characteristics of RISC-V. Different RISC-V hardware platforms exhibit significant differences in memory hierarchy, actual available memory bandwidth, and vector length (VLEN) of RISC-V Vector Extension (RVV). In particular, RISC-V's vector extension supports variable vector lengths, which places extremely high demands on data packing methods and circular block strategies. Traditional rule-based distribution methods cannot dynamically capture these hardware differences, often leading to problems such as low vector register utilization, cache thrashing, or bandwidth bottlenecks on specific hardware, preventing the algorithm from reaching the hardware's full performance potential.

[0033] Finally, existing technologies struggle to achieve targeted performance tuning at low cost. Because the shapes of irregular convolution operators are highly fragmented both within a single model and across different models, and because RISC-V hardware platforms are highly diverse, relying on human experts to design and distribute rules for specific shapes and hardware requires enormous human and time costs. While deep learning compilers such as TVM (Tensor Virtual Machine) exist, they rely on runtime search and tuning, which is time-consuming and often lacks good support for RISC-V vector-length-agnostic model expressions, limiting systematic optimization of RVV performance.

[0034] To address this, the present invention provides a RISC-V model inference acceleration method based on adaptive distribution of irregular convolution operators. By adaptively sensing the fine-grained shape information of irregular convolution operators and the hardware characteristics of RISC-V edge devices, the method selects the optimal algorithm implementation for irregular convolution operators to improve their performance, thereby further accelerating the end-to-end inference speed of the entire model.

[0035] It should be noted that the execution entity of the method of this invention can be an electronic device with model compilation and optimization capabilities. This can be a host computer development device (such as a server or high-performance computer with a cross-compilation toolchain installed), used to perform offline compilation and tuning of the model before deployment, generating an executable file adapted to the target RISC-V platform. Alternatively, it can be the target RISC-V device itself (if it has sufficient computing power to support online compilation), which performs adaptive distribution and adjustment of operators at runtime.

[0036] Figure 1 This is a flowchart illustrating the RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step S10: Obtain the fine-grained shape information of the convolution operator in the target model to be accelerated, as well as the hardware characteristic information of the target hardware platform.

[0037] Specifically, in this step, the first step is to analyze the target model to be accelerated. Here, the target model is the deep learning model that needs inference acceleration, such as ResNet, YOLO, MobileNet, etc. In deep learning models, the convolution operator is the core computational unit for performing feature extraction and also accounts for a large proportion of the computational cost.

[0038] In this process, the embodiments of the present invention do not merely obtain coarse information such as the convolution kernel size, but rather extract fine-grained shape information. Here, fine-grained shape information refers to a specific set of parameters that can completely describe the data dimensions of a convolution operator during computation. Specifically, for a two-dimensional convolution, its fine-grained shape information can be represented as: (1) in, , , , The above formula (1) follows the input shape ( ), kernel size is ,filling stride Expansion rate and output shape ( The standard relationship between them. Here, Indicates batch size, and Indicates the number of input and output channels. and Indicates the input and output heights. and Indicates the input and output widths. and This represents the height and width of the convolution kernel. and Indicates the stride length in the height direction and the stride length in the width direction. and This indicates fill in the height direction and fill in the width direction. and This represents the expansion rate in the height direction and the expansion rate in the width direction.

[0039] Based on the above formula (1), if the shape of the convolution operator exhibits the following characteristics, it is defined as an irregular convolution operator: ① Spatial dimension fragmentation, that is, the input and output spatial dimensions of the convolution operator ( ① The differences are significant within a single deep learning model or across multiple deep learning models, hindering uniform cyclic tiling; ② Irregular channel dimensions, i.e., the number of input or output channels of the convolution operator is not a multiple of the SIMD pack width, resulting in insufficient tail processing and channel utilization, thereby reducing vectorization efficiency.

[0040] Simultaneously, this step also requires obtaining hardware characteristic information of the target hardware platform. Here, the target hardware platform refers to the RISC-V device on which the model ultimately runs. Because the RISC-V architecture allows for high customization, different hardware implementations show significant differences in their support for RISC-V Vector Extensions (RVV). Therefore, key hardware characteristic information includes memory hierarchy (e.g., L1 / L2 cache size), vector register length (VLEN), and actual available memory bandwidth. It should be understood that the theoretical peak memory bandwidth depends on memory type (e.g., DDR4 / DDR5 / LPDDR4X / LPDDR5) and data rate (MT / s), bus / channel configuration, entry organization, and Rank (process), etc. Large models cannot directly perceive this fine-grained RISC-V hardware information. RVV has a total of 32 registers, each with a length of VLEN bits. Larger VLENs generally improve throughput by reducing loop striping and tail overhead.

[0041] These hardware parameters directly determine the vectorization efficiency and memory access bottleneck of a certain convolution algorithm (such as the GEMM-based convolution algorithm or the Winograd convolution algorithm) during execution. For example, the size of VLEN directly affects the amount of data that a vectorized instruction can process at one time, and thus affects the padding overhead for irregular channel numbers.

[0042] Step S20: Obtain the original operator distribution code of the convolution operator, and based on the fine-grained shape information and the hardware characteristic information, adjust the selection strategy of multiple optional convolution algorithm implementation logics in the original operator distribution code to generate candidate operator distribution code.

[0043] Specifically, existing deep learning inference libraries typically contain a set of primitive operator distribution code. This code is essentially a set of predefined selection logic used to determine which algorithm implementation logic to use based on the basic parameters of the convolution operator. Common convolution algorithm implementation logic includes, but is not limited to, GEMM-based convolution algorithms, Winograd convolution algorithms, and direct convolution algorithms.

[0044] However, the original fixed selection strategy often fails to recognize the complex coupling between the fine-grained shape information and the characteristics of RISC-V hardware. For example, some irregular shapes may be theoretically suitable for Winograd, but due to the small VLEN of the target hardware, the memory access overhead during the Winograd transformation stage is too large, making it slower than direct convolution.

[0045] Therefore, the core of this step lies in breaking the original fixed rules and making targeted strategy adjustments based on the specific information obtained in step S10. This adjustment means no longer blindly following the default path in the original library, but comprehensively considering the fragmented space dimension of operators and hardware limitations such as VLEN and bandwidth, to discover potentially better-performing algorithm implementation combinations.

[0046] Based on this, candidate operator distribution code is generated. This candidate code contains call instructions for one or more convolution algorithm implementations predicted to have higher execution efficiency under specific hardware and shape combinations. For example, for a specific 3×3 irregular convolution, the original strategy might distribute it to a GEMM implementation, while the generated candidate code might distribute it to a Winograd implementation with a specific block size or a direct convolution implementation with a specific packing method.

[0047] Step S30: Perform performance testing on the candidate operator distribution code on the target hardware platform, and determine the target operator distribution code from the candidate operator distribution code based on the test results.

[0048] Specifically, to ensure the true effectiveness of the optimization, this embodiment of the invention employs a real-world feedback mechanism. The generated candidate operator distribution code is compiled and deployed to the target hardware platform for actual operation.

[0049] During testing, the execution time and other performance metrics of the convolution operator when executing candidate code are recorded. This performance testing based on real hardware can accurately capture hardware microarchitectural behaviors that are difficult to cover through theoretical analysis, such as cache conflicts and pipeline stalls.

[0050] Based on the test results, the candidate operator distribution code with the best performance (e.g., shortest execution time) is determined as the target operator distribution code. This process essentially involves selecting the globally optimal solution from multiple possible optimization directions for the current irregular operator and the current hardware.

[0051] Step S40: Update the original operator distribution code in the deep learning base library for the target model to the target operator distribution code, and run the inference application of the target model on the target hardware platform based on the updated deep learning base library.

[0052] Specifically, after determining the target operator distribution code that is optimal for a specific convolution operator and specific hardware performance, it needs to be applied to the actual inference system to take effect. Typically, deep learning model inference relies on underlying deep learning libraries (such as the TVM runtime, OpenBLAS, or custom lightweight operator libraries optimized for RISC-V). These libraries contain pre-built raw operator distribution code for scheduling specific computational implementations based on operator parameters.

[0053] In this step, the model weight file is not directly modified; instead, the software environment supporting the model's operation is updated. Specifically, the default distribution logic for the relevant convolution operators (especially those for irregular operators) in the source code or intermediate compilation representation of the deep learning base library is replaced with the target operator distribution code determined in step S30 and verified through testing. This process involves embedding the generated optimized code snippets into the source files of the base library and recompiling to generate new library files, or replacing relevant symbols in the dynamic link library.

[0054] After updating the base library, launch the inference application (i.e., the final user-facing app or service process) of the target model on the target hardware platform. At this point, when the application loads and calls the updated deep learning base library to perform model inference tasks, the base library will no longer execute the original suboptimal distribution path. Instead, based on the updated logic, it will directly and precisely jump to the highest-performing convolutional algorithm implementation. By running the inference application based on the customized and optimized base library, this ensures that the model maximizes the utilization of RISC-V hardware's VLEN and bandwidth resources during runtime, thereby significantly reducing end-to-end inference latency.

[0055] The method provided in this invention, by collaboratively utilizing the fine-grained shape information of convolution operators and the hardware characteristics information of the target hardware platform, specifically adjusts and reconstructs the algorithm selection strategy in the original operator distribution code. This optimization approach, combining software and hardware information, can uncover convolution algorithm implementation paths with better performance under specific irregular shapes and hardware resource constraints, generating candidate operator distribution code with higher execution efficiency. Furthermore, through on-site performance testing and screening on the target hardware platform, theoretical prediction bias is eliminated, ensuring that the final determined target operator distribution code has optimal computational and memory access efficiency in actual operation. This effectively solves the problem of performance limitations caused by the mismatch between irregular operators and hardware characteristics, significantly improving the end-to-end model inference speed.

[0056] Based on any of the above embodiments, step S10, obtaining the fine-grained shape information of the convolution operator in the target model to be accelerated, includes: Step S11: Embed the original operator distribution code into the deep learning base library, and compile the deep learning base library and the inference application of the target model to generate the original executable program.

[0057] It should be noted that in actual deep learning deployment processes, static code analysis sometimes struggles to accurately capture runtime changes in data dimensions, especially for dynamic graph models or scenarios with dynamic inputs. Therefore, this embodiment of the invention employs a runtime feedback-based acquisition method.

[0058] Specifically, first, a benchmark testing environment needs to be built. The raw operator distribution code typically refers to existing, unoptimized operator scheduling logic. This code contains the implementation of how to distribute all convolutional operators in the model.

[0059] This step embeds the original code into a deep learning base library and integrates it with the inference application of the target model to be accelerated. Then, using a cross-compilation toolchain, a backend for the target architecture (e.g., RISC-V) is selected for compilation. The compilation process converts the model described in the high-level language into a binary file that can be directly run on the target hardware platform—the raw executable program. This program can perform the model inference task completely, although its performance may be suboptimal due to the lack of adaptation to irregular operators.

[0060] Step S12: Run the original executable program on the target hardware platform and collect performance data during the running process.

[0061] Specifically, the generated raw executable program is deployed to a real target hardware platform (such as a RISC-V development board or edge device) for execution. During execution, performance analysis tools (profilers) or instrumented code are used for real-time monitoring.

[0062] The collected performance data mainly includes the total latency of model inference, the independent execution time of each operator, and memory usage. By analyzing this data, the inference latency of the entire deep learning model can be decomposed into the specific execution time of each operator. This step is crucial because it helps the system identify operators that perform poorly under the default distribution strategy and consume a large amount of inference time. These operators are often the irregular convolution operators that need to be optimized.

[0063] Step S13: Based on the performance data, extract the fine-grained shape information of the convolution operator. The fine-grained shape information includes the input tensor shape, output tensor shape, convolution kernel size, stride size, padding parameters, and dilation rate.

[0064] Specifically, by combining log information from performance data, the system can accurately extract the parameters actually passed to the convolution operator at runtime. These parameters constitute fine-grained shape information, including the input tensor shape, output tensor shape, convolution kernel size, stride size, padding parameters, and dilation rate. The input and output tensor shapes involve batch size, number of input channels, number of output channels, and the height and width of the spatial dimensions. It should be understood that, through runtime acquisition, this embodiment of the invention can accurately capture the specific values ​​of spatial dimension fragmentation (e.g., H×W is extremely small or drastically changing within the model) and channel dimension irregularities (e.g., the number of input or output channels cannot be divided evenly by the hardware vector width).

[0065] This invention collects performance data by running the compiled original program on the target hardware platform, and then extracts fine-grained shape information. Compared with simple static analysis, it can more accurately locate the bottleneck operators that have the greatest impact on the end-to-end inference speed of the model. At the same time, it ensures that the extracted shape information is data from actual runtime, avoiding shape misjudgment caused by dynamic input, and providing a reliable data foundation for subsequent precise operator distribution optimization for irregular shapes.

[0066] Based on the above embodiments, the convolution operator is an irregular convolution operator, and the implementation logic of the various optional convolution algorithms includes at least two of the following: a convolution algorithm based on general matrix multiplication, a Winograd convolution algorithm, a direct convolution algorithm, and a customized convolution algorithm for a convolution kernel of a specified size.

[0067] Specifically, in this embodiment of the invention, the convolution operator is an irregular convolution operator. This irregularity is mainly reflected in the two dimensions of the input and output tensors, namely, spatial dimension fragmentation and channel dimension irregularity. These characteristics make it difficult for traditional uniform distribution strategies based on fixed rules to work.

[0068] Understandably, spatial dimension fragmentation refers to the highly differentiated and fragmented spatial dimensions (height H × width W) of the input and output tensors of a convolution operator, whether within a single model or across models. For example, in the backend of some detection models, the feature map size may be extremely small, making it impossible to perform effective cyclic tiling in regular matrix multiplication, or to achieve load balancing through standard 4×4 or 6×6 blocks in the Winograd algorithm.

[0069] Channel dimension irregularity refers to a situation where the number of input or output channels of a convolution operator cannot be divided by the SIMD packing width of the target hardware platform (e.g., elempack in RISC-V vector extensions). This misalignment can lead to significant tail processing overhead during vector instruction execution or require additional padding operations, thereby significantly reducing actual computational efficiency.

[0070] For the aforementioned irregular convolution operators, the various optional convolution algorithm implementations in this embodiment constitute a rich candidate algorithm library. The large language model or distribution strategy adjustment module selects the best implementation for each irregular operator from this library. This library contains at least two or all of the following logics: convolution algorithms based on Generalized Matrix Multiplication (GEMM), Winograd convolution algorithms, direct convolution algorithms, and customized convolution algorithms for convolution kernels of specified sizes, etc.

[0071] This invention, by explicitly defining irregular convolution operators as optimization targets and providing a wide range of algorithm choices covering GEMM, Winograd, direct convolution, and customized implementations for specific sizes, ensures that no matter how fragmented the shape of the operator is, or what the hardware's VLEN and bandwidth limitations are, the system can find an implementation logic that achieves the best balance between computational density and memory access overhead from the aforementioned rich candidate library, thereby maximizing the performance of RISC-V hardware.

[0072] The following section will analyze the impact of fine-grained shape information of irregular convolution operators on operator distribution, based on the implementation logic of each convolution algorithm.

[0073] For standard two-dimensional convolution, when using a GEMM-based algorithm, the effective GEMM dimension is represented as: (2) (3) (4) M, N, and K represent the dimensions (i.e., the number of rows and columns) of the matrices involved in the matrix multiplication operation, where M represents the number of rows in the left matrix, N represents the number of columns in the right matrix, and K represents the number of columns in the left matrix, which is also equal to the number of rows in the right matrix. M, N, and K determine the computational and memory access costs of GEMM, and further affect the performance of the convolution operator by influencing computational efficiency, memory bandwidth utilization, vectorization efficiency, and cache hit rate. Therefore, as can be seen from formulas (2) to (4), the fine-grained shape information of the convolution operator will affect the performance of the convolution operator when using the GEMM-based algorithm implementation method, and further affect whether it is suitable to be distributed to the GEMM-based algorithm implementation path.

[0074] The essence of the Winograd algorithm is an addition-to-multiplication transformation. Its general computational optimization process can be divided into three steps: ① performing matrix transformation on the input feature map and weights; ② performing batch matrix multiplication on the transformed input feature map and weights; ③ transforming the output of the matrix multiplication result to obtain the final result. The Winograd algorithm is mainly applied to convolutional kernels with... A 2D convolution with a stride of 1, the Winograd algorithm is suitable for sizes of The convolution kernel is calculated to have a size of When outputting a block, the commonly used notation is as follows: Output tensor shape Divided into multiple tile: (5) (6) (7) (8) in, This indicates the number of tiles in the height direction of the output tensor; This indicates the number of tiles in the width direction of the output tensor; This represents the total number of tiles to be processed per channel; r represents the spatial size of the convolution kernel, i.e., the height and width of the kernel are both r; t represents the side length of the tile used in the Winograd transform domain; (The remaining text appears to be incomplete and possibly contains errors.) (For example, for Winograd23 / 43 / 63, The values ​​2 / 4 / 6 respectively will change the balance between arithmetic reduction and transformation / memory overhead. Therefore, from formulas (5) to (8), it can be seen that the fine-grained shape information of the convolution operator will affect the number of tiles in the Winograd algorithm, and further directly affect cache locality and memory bandwidth utilization, thus indicating which path among the Winograd23, Winograd43 and Winograd63 algorithm implementations has better performance.

[0075] The computational cost of direct convolution implementation is closely related to the input batch size, the number of output channels, the output feature map size (height × width), the number of input channels, and the size of the convolution kernel (height × width). Memory access is closely related to the batch size, the number of input channels, the input feature map (height × width), the number of output channels, the size of the convolution kernel (height × width), and the size of the output feature map (height × width). Computational intensity (Roofline performance model) is closely related to the number of input channels, the number of output channels, the size of the convolution kernel, the output feature map, and the input feature map. Therefore, the fine-grained shape information of the convolution operator affects the performance of the operator in direct convolution implementation, and further affects whether it is suitable to be distributed to the direct convolution algorithm implementation path.

[0076] For convolutional kernels of specific sizes, such as 7×7 and 5×5 kernels, there are specialized operator implementations. There are also specialized operator implementations for cases where the input vector is packed using `pack1` and the output vector using `packn`, the input vector is packed using `packn` and the output vector is packed using `pack1`, and the input vector is packed using `pack1` and the output vector is packed using `pack1`. The performance of these convolutional kernel operator implementations with specific sizes and specific vectorization packing methods is closely related to the fine-grained shape information of irregularly shaped convolutional operators. Therefore, it will further affect whether the operator is suitable for distribution to a specific operator implementation.

[0077] In summary, for common edge deep learning models, the performance of operators on different convolutional operator distribution paths is closely related to the fine-grained shape information of the convolutional operators. Adaptive operator distribution is needed for different irregular operators to obtain better performance of irregular shape operators.

[0078] Based on any of the above embodiments, the target hardware platform is a processor platform that supports vector extension instruction sets, and the hardware characteristic information includes the memory hierarchy information, actual available memory bandwidth information, and vector register length parameters of the target hardware platform.

[0079] Specifically, the target hardware platform refers to a processor platform that supports the RISC-V Vector Extension (RVV) instruction set. RISC-V is an open-source instruction set architecture based on the Reduced Instruction Set Computing (RISC) principle, and RVV is one of its important extensions, designed to improve data parallel processing capabilities through Single Instruction Multiple Data (SIMD) technology. Unlike traditional fixed-length SIMD, RISC-V's vector extension allows hardware designers to choose different vector register lengths during implementation. While this length-agnostic characteristic provides flexibility, it also presents challenges for software optimization.

[0080] To achieve adaptive operator distribution, it is necessary to obtain specific hardware characteristic information of the target hardware platform. This information may include memory hierarchy information, actual available memory bandwidth information, vector register length parameters, etc. The impact of RISC-V hardware characteristics on convolution operator distribution will be discussed below.

[0081] The performance of convolution operators implemented based on GEMM is influenced by the memory hierarchy, available memory bandwidth, and the VLEN of RVV. Regarding the memory hierarchy, it determines whether the multi-level block size of GEMM is appropriate; a suitable block size can significantly improve operator performance. Regarding memory bandwidth, when K or N is small (i.e., a small matrix), reuse is insufficient; im2col writes huge K×N temporary matrices, and improper block size can lead to bandwidth limitations. For RVV VLEN, GEMM's regular memory access pattern is well-suited for vectorization. A larger VLEN allows for caching more data in registers and the use of larger vector groups to improve throughput, but the overhead of frequently switching vector lengths must be avoided. Therefore, RISC-V hardware characteristics affect the performance of convolution operators when distributed to GEMM-based implementation paths, thus influencing whether it is suitable to distribute convolution operators to GEMM-based implementation paths.

[0082] The performance of convolution operators implemented using the Winograd algorithm is constrained by memory hierarchy, available memory bandwidth, and RVV VLEN. Regarding memory hierarchy, the Winograd convolution operator implementation requires multiple intermediate buffers to store the transformed input, weights, and output, significantly increasing memory requirements compared to direct convolution and GEMM. Regarding memory bandwidth, the Winograd algorithm has a large transformation overhead; although it reduces multiplication, it increases memory access significantly, resulting in a low computation-to-memory ratio and bandwidth becoming a bottleneck, especially during the transformation phase. For RVV VLEN, the transformation operations implemented in Winograd have complex memory access patterns, making vectorization difficult. They also involve many scalar operations and irregular accesses, leading to low SIMD efficiency. Furthermore, Winograd has many transformation coefficients, resulting in high register pressure and underutilization of vector registers. Therefore, RISC-V hardware characteristics affect the performance of convolution operators when distributed to the Winograd algorithm implementation path, thus influencing whether it is suitable to distribute the convolution operator to the Winograd algorithm implementation path, and whether to distribute it to the Winograd23, Winograd43, or Winograd63 implementation path.

[0083] The performance of convolution operators implemented via direct convolution is constrained by the memory hierarchy, available memory bandwidth, and the vector length (VLEN) of the RVV implementation. Regarding the memory hierarchy, the size of the block determines whether it can fit into the L1 cache (or at least the L2 cache). If it cannot, frequent write-backs to main memory are required, increasing latency. Direct convolution's sliding window access, if spanning multiple cache lines and with an unfriendly stride, generates fragmented traffic, reducing actual bandwidth utilization. Regarding memory bandwidth, unlike GEMM-based algorithms that adjust data access to a large matrix sequential flow, direct convolution often falls into bandwidth-constrained regions if data reuse is not handled well. Regarding the RVV VLEN, generally, a larger VLEN allows for more elements to be parallelized per instruction; however, if the packing dimension is not a multiple of packn, it introduces tail processing overhead. Therefore, RISC-V hardware characteristics affect the performance of convolution operators when distributed to direct convolution implementations, thus influencing whether it is suitable to distribute convolution operators to direct convolution implementations.

[0084] For convolutional kernels of specific sizes, such as 7×7 and 5×5 kernels, there are specialized operator implementations. There are also specialized algorithm implementations for cases where the input is `pack1` and the output is `packn`, the input is `packn` and the output is `pack1`, and the input is `pack1` and the output is `pack1`. These specific kernel operator implementations with specific sizes and packing methods are closely related to the memory hierarchy, available memory bandwidth, and the value of VLEN in RVV. Therefore, the performance of these specific operator implementations is also affected by the characteristics of RISC-V hardware, which in turn affects whether it is suitable to distribute convolutional operators to these specific implementation paths.

[0085] In summary, for common edge deep learning models, the optimal implementation path for distributing convolution operators at the operator distribution level is closely related to the characteristics of RISC-V hardware and requires specific tuning for different RISC-V hardware platforms to obtain better operator performance.

[0086] This invention, by explicitly defining the target hardware platform as a RISC-V processor supporting RVV, and by deeply acquiring key hardware characteristic information such as memory hierarchy, bandwidth, and VLEN, ensures that subsequent operator distribution optimization is no longer arbitrary. The system can accurately assess the efficiency of vectorized packing based on the VLEN size, avoid memory bottleneck algorithms based on bandwidth limitations (e.g., not selecting Winograd when bandwidth is insufficient), and optimize the block partitioning strategy based on cache size. This hardware-aware mechanism ensures that irregular convolution operators can fully utilize the hardware characteristics of specific RISC-V chips, maximizing computational efficiency.

[0087] Based on any of the above embodiments, in step S20, adjusting the selection strategy for multiple optional convolution algorithm implementation logics in the original operator distribution code based on the fine-grained shape information and the hardware characteristic information to generate candidate operator distribution code includes: Step S21: Input the fine-grained shape information, the hardware characteristic information, and the original operator distribution code as prompt information into the large language model; Step S22: Guide the large language model to analyze the adaptation relationship between the fine-grained shape information and the implementation logic of the various optional convolution algorithms; Step S23: Guide the large language model to predict the execution efficiency ranking of each convolution algorithm implementation logic on the target hardware platform based on the adaptation relationship and the hardware characteristic information, and generate the candidate operator distribution code according to the prediction result. The candidate operator distribution code is configured to distribute the convolution operator to the convolution algorithm implementation logic with the highest predicted execution efficiency.

[0088] It should be noted that in traditional deep learning compilers, operator distribution mainly relies on heuristic rules written by human experts. However, when faced with complex and varied irregular convolution operators and highly customized RISC-V hardware, these static rules often fail to cover all situations, resulting in suboptimal performance. This invention introduces a Large Language Model (LLM) with code understanding and reasoning capabilities to achieve adaptive policy adjustment.

[0089] Specifically, the first step involves constructing prompts to guide large language models (such as specialized models for code optimization). These prompts act as a bridge between specific technical problems and the reasoning capabilities of the large model. In this step, the prompts primarily consist of three parts: fine-grained shape information, hardware characteristic information, and the original operator distribution code.

[0090] Among them, fine-grained shape information is the feature profile of the operator, including the shape of the input tensor, the shape of the output tensor, the size of the convolution kernel, the stride, the padding parameters, the dilation rate, etc. These data directly reflect the irregularity of the operator (such as the number of channels being 33, which is not a power of 2, making alignment difficult).

[0091] Hardware characteristics information refers to the operator's operating environment. For example, it may be explicitly stated that "the target platform is RISC-V, VLEN=256 bits, L1 Cache=32KB, and memory bandwidth=12.8GB / s".

[0092] The raw operator dispatch code is the baseline object to be optimized. It is usually the scheduling logic written in C++ or Python, which contains a set of judgment statements based on general rules.

[0093] The above information is assembled using a structured text template to form a complete prompt, informing the large language model that its current task is to optimize the algorithm selection strategy for this code, given the specific hardware environment and operator shape.

[0094] After receiving the prompts, the large language model first performs logical reasoning analysis. Adaptation refers to the theoretical compatibility and potential problems of a specific convolutional algorithm (such as GEMM, Winograd, direct convolution, etc.) when processing data of a specific shape. During this process, the large model utilizes the computer science knowledge acquired during its pre-training phase for analysis.

[0095] For example, for the Winograd convolution algorithm, the model analyzes whether the operator shape is suitable for the Winograd transform. If the convolution kernel is 7×7, the model might infer that the number of tiles in the Winograd transform is too large, leading to decreased numerical accuracy and excessive transform overhead, thus resulting in a low fit. For GEMM-based algorithms, the model checks whether the im2col operation generates a large amount of redundant data copying when encountering spatial dimensional fragmentation, or whether the number of channels (e.g., 33) will cause the K-dimensionality of GEMM matrix multiplication to be too small, affecting block efficiency. For direct convolution algorithms, the model analyzes whether, given a VLEN, special vectorized packing can be used to avoid invalid computations caused by padding.

[0096] After analyzing the adaptation relationships, the large model further incorporates hardware characteristic information for performance prediction. For example, although the Winograd algorithm theoretically has the fewest multiplications, the large model, considering the lower memory bandwidth of RISC-V hardware, might predict that Winograd will encounter bottlenecks due to memory access intensity, and its actual execution efficiency may be lower than that of direct convolution. Alternatively, considering the characteristic of VLEN=512, the model predicts that direct convolution using packn (vectorized packing) can greatly utilize SIMD parallelism.

[0097] Based on the comprehensive predictions above, the large model determines the optimal path from multiple alternative logics and generates candidate operator distribution code. The generated code here is no longer a vague suggestion, but a specific, compilable code snippet. For example, the large model might modify the general selection logic in the original code to directly call a specific optimized function interface, or adjust the GEMM's block parameters to accommodate the L1 cache size. Essentially, the generated candidate code forces the specific irregular operator to be distributed to the algorithm implementation path with the best prediction performance.

[0098] This invention, through the introduction of a large language model, transforms the complex operator distribution optimization problem into an artificial intelligence reasoning and generation task. Leveraging the powerful code understanding capabilities and cross-domain knowledge of the large model, it can automatically uncover optimal combinations that are difficult for human experts to discover. Particularly for convolution operators with extremely irregular shapes and specific RISC-V hardware configurations, the large model can perform deep adaptation analysis and efficiency prediction based on the ternary relationship between shape, hardware, and algorithm. This generates operator distribution code that is more accurate and performs better than traditional fixed rules, significantly reducing the high cost of relying on manually written optimization rules.

[0099] Based on any of the above embodiments, step S30 specifically includes: Step S31: Embed the current candidate operator distribution code generated by the large language model into the deep learning base library, and compile the deep learning base library and the inference application of the target model to generate a test program; Step S32: Run the test program on the target hardware platform and record the actual execution time of the convolution operator under the current candidate operator distribution code; Step S33: Compare the actual execution time with the original execution time corresponding to the original operator distribution code. If the reduction in the actual execution time compared to the original execution time exceeds a preset threshold, then add the current candidate operator distribution code and the corresponding actual execution time to the preferred set. Step S34: Select the candidate operator distribution code with the shortest actual execution time from the preferred set as the target operator distribution code.

[0100] It should be noted that although large language models possess powerful code generation capabilities, their output is essentially based on probabilistic predictions, which may lead to misleading results or generate syntactically correct code that is not truly optimized in terms of performance. Therefore, this embodiment of the invention further introduces a closed-loop verification mechanism based on the target hardware platform.

[0101] Specifically, after the large language model outputs a piece of code for distributing the current candidate operator, the system first needs to embed it into the source code tree of the deep learning base library, replacing the original distribution logic. Then, the compilation process begins. Since the target runtime environment is a RISC-V architecture, a cross-compiler supporting RISC-V Vector Extensions (RVV) is typically required. The compilation process converts the model source code containing the new distribution strategy into a binary file that can be directly run on the target hardware platform, i.e., a test program. If compilation fails during this process (e.g., due to syntax errors in the LLM-generated code), the candidate code will be directly marked as invalid.

[0102] The successfully compiled test program is pushed to the target hardware platform (such as a RISC-V development board) for execution. During execution, the system uses high-precision timers or performance counters to specifically monitor and record the execution time of the particular convolution operator under the new distributed code, i.e., the actual execution time. This data reflects the performance of the operator under real memory bandwidth pressure and cache conditions.

[0103] To evaluate the effectiveness of the optimization, a benchmark needs to be established. The original execution time refers to the execution time of the operator before the distribution strategy was modified. The system compares the actual execution time of the current candidate code with the original execution time. To prevent minor fluctuations caused by measurement errors from being mistaken for performance improvements, this embodiment of the invention sets a preset threshold, for example, the improvement must be greater than 5%, that is, the actual execution time is less than the original execution time by more than 5%.

[0104] If the performance improvement does not exceed the threshold (or even decreases), the candidate code is not considered an effective optimization and is discarded. If the performance improvement exceeds the preset threshold, the candidate code is considered a potentially high-quality solution. The system will add the code snippet and its corresponding performance data as a valid entry to a preferred set. Here, the preferred set is a container used to temporarily store all candidate solutions that have been tested and verified to provide performance improvements.

[0105] After completing one or more rounds of testing, the preferred set may contain multiple different effective distribution strategies. At this point, the system iterates through the set, compares the actual execution times recorded therein, and selects the scheme with the shortest time (i.e., the fastest inference speed) as the final target operator distribution code.

[0106] This invention effectively filters out inefficient or invalid code that may be generated by large language models through closed-loop verification involving compilation, execution, and testing on real hardware. By introducing preset thresholds and an optimal set, it ensures that the final selected target operator distribution code is not only theoretically feasible but also delivers significant performance gains in engineering practice, thus guaranteeing the reliability of the model acceleration effect.

[0107] Based on any of the above embodiments, step S34 specifically includes: Step S341: Determine whether the number of candidate operator distribution codes in the preferred set has reached a preset number, or whether the number of code generation times of the large language model has reached a preset upper limit; Step S342: If yes, then stop generating new candidate operator distribution codes and traverse the preferred set to select the candidate operator distribution code with the shortest actual execution time as the target operator distribution code. Step S343: If not, the performance test results of the current candidate operator distribution code are fed back to the large language model, and the large language model is instructed to regenerate new candidate operator distribution code.

[0108] It should be noted that the embodiments of the present invention are a further improvement on the screening process in the above embodiments, especially providing a detailed explanation of how to control the iterative process of code generation. Given the complexity of the search space faced by large models in the convolution operator distribution optimization problem (each irregularly shaped convolution operator has multiple different algorithmic implementations to choose from), and the risk of local optima in a single generation of large models, the embodiments of the present invention design a multi-round optimization method, employing the best-of-N principle to determine the target operator distribution code with the best performance from the N candidate operator distribution codes generated by the large model.

[0109] Specifically, after each large model generates code and completes testing, the system needs to determine whether to continue with the next round of optimization. Two termination conditions are set here: first, whether the number of options in the preferred set has reached a preset limit, for example, whether N (e.g., 5) candidate solutions with significantly improved performance have been found. This indicates that the optimization goal has been largely achieved, and the marginal benefit of continuing the search may diminish. Second, whether the number of code generation attempts has reached a preset limit, for example, whether the large model has been requested to generate code M times (e.g., 20 times). This is to control time costs and computational resources, preventing the program from entering an infinite loop.

[0110] If any of the above conditions are met (if so), the stopping mechanism is triggered, and no new generation requests are sent to the large language model. At this point, the system will perform a final traversal of the preferred set to select the code with the best performance data (shortest execution time) and determine it as the final target operator distribution code used to update the model.

[0111] If none of the above conditions are met (if not), it means that the current optimization results are not rich enough, or the upper limit of the number of attempts has not been reached, and further exploration is needed. At this time, the system performs a key feedback operation, and the system will feed back the performance test results just measured (e.g., "Code version V1 compiled successfully, but took 50ms on hardware, which is 10% slower than the benchmark" or "Code version V2 took 30ms, which is 20% faster") as a new prompt to the large language model.

[0112] This constitutes a contextual learning mechanism. By informing the large model of the successes and failures of past attempts, it is instructed to avoid known erroneous paths in the next generation, or to further fine-tune based on successful paths. Subsequently, the large model regenerates new candidate operator distribution codes, and the process returns to step S31 to enter the next round of loop.

[0113] This invention fully leverages the exploratory capabilities of a large language model by constructing an automatic iterative loop with a feedback mechanism. By setting preset termination conditions for a certain number and upper limit, a balance is achieved between search breadth (finding enough good solutions) and search cost (controlling time). In particular, the design of feeding experimental results back to the large model transforms the code generation process from independent random trials into an evolutionary process of continuous self-correction and gradual approximation of the optimal solution, thereby improving the success rate of finding the best distribution strategy for complex and irregular operators.

[0114] Based on any of the above embodiments, this invention provides a RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution. This method is mainly applied to edge computing or edge AI inference scenarios based on the RISC-V architecture. With the development of deep learning technology, more and more deep learning models are being deployed on edge devices such as smart homes, security monitoring, and industrial IoT. These devices typically adopt the RISC-V instruction set architecture, and their hardware resources (such as memory bandwidth, vector register length, etc.) are highly diverse and varied.

[0115] In practical inference, many convolution operators in a model often exhibit irregularities in shape (e.g., spatial dimension fragmentation, channel dimension irregularities, etc.). Existing deep learning inference frameworks typically employ operator distribution mechanisms based on fixed rules, making it difficult to find the optimal algorithm implementation path for these irregular operators on specific RISC-V hardware, resulting in low hardware utilization and high inference latency. This invention aims to accelerate model inference by adaptively combining operator shape details and hardware characteristics to find the optimal execution code for these irregular operators.

[0116] Figure 2 This is a schematic diagram of the architecture of the RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution provided by the present invention, as shown below. Figure 2 As shown, this method deeply integrates hardware and software information through a large model, achieving a transformation from a slow-inference original model to a fast-inference optimized model. Specifically, the method includes the following steps: Step 1: Distribute the code using the original convolution operator in the deep learning base library and cross-compile it using the RVV backend to obtain the executable program for deep learning model inference.

[0117] Step 2: Run the executable program on the target RISC-V hardware platform to perform performance tests, decompose the inference latency of the deep learning model into the execution time of each operator, and extract the execution time of the irregularly shaped convolution operator and the execution time of the entire model inference.

[0118] Step 3: The large model automatically mines the algorithm implementation path with better performance for irregularly shaped convolution operators and target RISC-V hardware based on the fine-grained shape information of the irregularly shaped convolution operators (including fragmented spatial dimensions, irregular channel dimensions, etc.), RISC-V hardware characteristic information (including memory bandwidth, memory hierarchy, RVVVLEN, etc.) and the original convolution operator distribution code, and generates candidate convolution operator distribution optimization code.

[0119] Step 4: Embed the candidate convolution operator distribution optimization code generated by the large model into the deep learning base library for cross-compilation. If the compilation is successful, proceed to Step 5; otherwise, it indicates that the candidate convolution operator distribution optimization code generated by the large model is incorrect, and return to Step 3 to regenerate.

[0120] Step 5: Copy the executable program required for inference of the compiled deep learning model to the target RISC-V hardware platform for testing, and record the execution time of the irregular-shaped convolution operator and the inference time of the entire model.

[0121] Step Six: If the execution time of the irregular-shaped convolution operator is more than 5% less than the execution time of the irregular-shaped convolution operator under the original convolution operator distribution method, it indicates that the convolution operator distribution optimization has an optimization effect. Record the execution time in set T, save the corresponding convolution operator distribution optimization code file, establish a matching pair between the execution time of the irregular convolution operator and the file name of the convolution operator distribution optimization code, and execute Step Seven; otherwise, it indicates that there is no significant optimization effect, discard the code and return to Step Three.

[0122] Step 7: If there are 5 values ​​in set T, stop the multi-round optimization process and proceed to step 8; otherwise, return to step 3.

[0123] Step 8: Take the convolution operator distribution code file corresponding to the maximum value of the 5 values ​​in set T, and use it as the final convolution operator distribution optimization code.

[0124] Step 9: Embed the final convolution operator distribution optimization code into the deep learning base library and select the RVV backend for cross-compilation to obtain the executable program for deep learning model inference, record the inference time of the entire model, and compare it with the model inference time before convolution operator distribution optimization obtained in Step 2 to evaluate the acceleration effect of the entire model inference.

[0125] It is understandable that, such as Figure 2As shown, after optimization and decision-making by the large model, the irregularly shaped convolution operators in the original model are adaptively distributed to the algorithm implementations best suited to their shapes and the current hardware. This distribution is no longer singular but exhibits a high degree of diversity and specificity. For example, irregularly shaped convolution operators 1 and n are distributed to GEMM-based algorithm implementations, indicating that under current hardware conditions, matrix multiplication is the most efficient way to implement the shapes of these two operators. Irregularly shaped convolution operator 2 is distributed to the Winograd23 algorithm implementation, and irregularly shaped convolution operator 3 is distributed to Winograd43, etc., reflecting a fine selection of different tile sizes. For another example, irregularly shaped convolution operator 5 is distributed to a direct convolution algorithm implementation, indicating that avoiding direct computation of the im2col transform is more efficient for this operator. Irregularly shaped convolution operators p and q are distributed to 7×7 and 5×5 convolution kernel implementations, respectively, utilizing optimizations of dedicated kernels. Furthermore, to address the vectorization challenges caused by channel irregularities, the irregular-shaped convolution operator r is distributed to the packnto1 convolution implementation (i.e., n inputs are packed into one output), while operators s and t are distributed to the pack1ton convolution implementation to maximize the throughput of SIMD instructions.

[0126] This invention overcomes the limitations of traditional deep learning compilers that distribute operators based on fixed rules by acquiring fine-grained shape information of convolution operators (especially descriptions of irregular shapes) and the underlying characteristics of the RISC-V target hardware (such as RVV, VLEN, and bandwidth). By adjusting the distribution strategy and generating candidate code based on this fine-grained information, and then selecting the target distribution code through real-device testing, it ensures that irregular convolution operators always run on the most efficient algorithm implementation path on diverse RISC-V hardware. This not only solves the problems of low vectorization efficiency and memory access bottlenecks caused by irregular operators, but also effectively improves the overall inference performance of the model on edge devices.

[0127] The following describes the RISC-V model inference acceleration device based on adaptive distribution of irregular convolution operators provided by the present invention. The RISC-V model inference acceleration device based on adaptive distribution of irregular convolution operators described below can be referred to in correspondence with the RISC-V model inference acceleration method based on adaptive distribution of irregular convolution operators described above.

[0128] Based on any of the above embodiments Figure 3 This is a schematic diagram of the RISC-V model inference acceleration device based on adaptive distribution of irregular convolution operators provided by the present invention, as shown below. Figure 3 As shown, the device also includes: The information acquisition unit 310 is used to acquire fine-grained shape information of the convolution operator in the target model to be accelerated, as well as hardware characteristic information of the target hardware platform. The code generation unit 320 is used to obtain the original operator distribution code of the convolution operator, and based on the fine-grained shape information and the hardware characteristic information, adjust the selection strategy of multiple optional convolution algorithm implementation logic in the original operator distribution code to generate candidate operator distribution code; The performance testing unit 330 is used to perform performance testing on the candidate operator distribution code on the target hardware platform, and determine the target operator distribution code from the candidate operator distribution code based on the test results. The model execution unit 340 is used to update the original operator distribution code in the deep learning base library for the target model to the target operator distribution code, and run the inference application of the target model on the target hardware platform based on the updated deep learning base library.

[0129] The apparatus provided in this invention utilizes fine-grained shape information of convolution operators and hardware characteristic information of the target hardware platform to specifically adjust and reconstruct the algorithm selection strategy in the original operator distribution code. This optimization method, combining software and hardware information, can uncover convolution algorithm implementation paths with better performance under specific irregular shapes and hardware resource constraints, generating candidate operator distribution code with higher execution efficiency. Furthermore, by conducting on-site performance testing and screening on the target hardware platform, theoretical prediction bias is eliminated, ensuring that the final target operator distribution code has optimal computation and memory access efficiency in actual operation. This effectively solves the problem of performance limitations caused by the mismatch between irregular operators and hardware characteristics, significantly improving the end-to-end model inference speed.

[0130] Based on any of the above embodiments, the information acquisition unit is specifically used for: The original operator distribution code is embedded into the deep learning base library, and the inference application of the deep learning base library and the target model is compiled to generate the original executable program. The original executable program is run on the target hardware platform, and performance data is collected during the running process; Based on the performance data, fine-grained shape information of the convolution operator is extracted. The fine-grained shape information includes the input tensor shape, output tensor shape, convolution kernel size, stride size, padding parameters, and dilation rate.

[0131] Based on any of the above embodiments, the target hardware platform is a processor platform that supports vector extension instruction sets, and the hardware characteristic information includes the memory hierarchy information, actual available memory bandwidth information, and vector register length parameters of the target hardware platform.

[0132] Based on any of the above embodiments, the code generation unit is specifically used for: The fine-grained shape information, the hardware characteristic information, and the original operator distribution code are input as prompt information into the large language model; The large language model is guided to analyze the adaptation relationship between the fine-grained shape information and the implementation logic of the various optional convolution algorithms; The large language model is guided to predict the execution efficiency of each convolution algorithm implementation logic on the target hardware platform based on the adaptation relationship and the hardware characteristic information. Based on the prediction results, the candidate operator distribution code is generated. The candidate operator distribution code is configured to distribute the convolution operator to the convolution algorithm implementation logic with the highest predicted execution efficiency.

[0133] Based on any of the above embodiments, the performance testing unit is specifically used for: The current candidate operator distribution code generated by the large language model is embedded into the deep learning base library, and the inference application of the deep learning base library and the target model is compiled to generate a test program; Run the test program on the target hardware platform and record the actual execution time of the convolution operator under the current candidate operator distribution code; The actual execution time is compared with the original execution time corresponding to the original operator distribution code. If the reduction in the actual execution time compared to the original execution time exceeds a preset threshold, the current candidate operator distribution code and the corresponding actual execution time are added to the preferred set. The candidate operator distribution code with the shortest actual execution time is selected from the preferred set as the target operator distribution code.

[0134] Based on any of the above embodiments, the performance testing unit is further configured to: Determine whether the number of candidate operator distribution codes in the preferred set has reached a preset number, or whether the number of code generation times of the large language model has reached a preset upper limit; If so, then stop generating new candidate operator distribution codes, and traverse the preferred set to select the candidate operator distribution code with the shortest actual execution time as the target operator distribution code; If not, the performance test results of the current candidate operator distribution code are fed back to the large language model, and the large language model is instructed to regenerate new candidate operator distribution code.

[0135] Based on any of the above embodiments, the convolution operator is an irregular convolution operator, and the implementation logic of the various optional convolution algorithms includes at least two of the following: a convolution algorithm based on general matrix multiplication, a Winograd convolution algorithm, a direct convolution algorithm, and a customized convolution algorithm for a convolution kernel of a specified size.

[0136] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logic instructions in the memory 430 to execute a RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution. This method includes: acquiring fine-grained shape information of the convolution operators in the target model to be accelerated, and hardware characteristic information of the target hardware platform; acquiring the original operator distribution code of the convolution operators, and adjusting the selection strategy of multiple optional convolution algorithm implementation logics in the original operator distribution code based on the fine-grained shape information and the hardware characteristic information to generate candidate operator distribution codes; performing performance testing on the candidate operator distribution codes on the target hardware platform, and determining the target operator distribution code from the candidate operator distribution codes based on the test results; updating the original operator distribution code in the deep learning base library for the target model to the target operator distribution code, and running the inference application of the target model on the target hardware platform based on the updated deep learning base library.

[0137] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution provided by the above methods. The method includes: obtaining fine-grained shape information of convolution operators in the target model to be accelerated, and hardware characteristic information of the target hardware platform; obtaining the original operator distribution code of the convolution operators, and adjusting the selection strategy of multiple optional convolution algorithm implementation logics in the original operator distribution code based on the fine-grained shape information and the hardware characteristic information to generate candidate operator distribution code; performing performance testing on the candidate operator distribution code on the target hardware platform, and determining the target operator distribution code from the candidate operator distribution code according to the test results; updating the original operator distribution code in the deep learning base library for the target model to the target operator distribution code, and running the inference application of the target model on the target hardware platform based on the updated deep learning base library.

[0139] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution provided by the above methods. The method includes: acquiring fine-grained shape information of convolution operators in the target model to be accelerated, and hardware characteristic information of the target hardware platform; acquiring the original operator distribution code of the convolution operators, and adjusting the selection strategy of multiple optional convolution algorithm implementation logics in the original operator distribution code based on the fine-grained shape information and the hardware characteristic information to generate candidate operator distribution codes; performing performance testing on the candidate operator distribution codes on the target hardware platform, and determining the target operator distribution code from the candidate operator distribution codes according to the test results; updating the original operator distribution code in the deep learning base library for the target model to the target operator distribution code, and running the inference application of the target model on the target hardware platform based on the updated deep learning base library.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for accelerating RISC-V model inference based on adaptive distribution of irregular convolution operators, characterized in that, include: Obtain fine-grained shape information of the convolution operator in the target model to be accelerated, as well as hardware characteristic information of the target hardware platform; The original operator distribution code of the convolution operator is obtained, and the selection strategy of multiple optional convolution algorithm implementation logics in the original operator distribution code is adjusted based on the fine-grained shape information and the hardware characteristic information to generate candidate operator distribution code. The candidate operator distribution code is subjected to performance testing on the target hardware platform, and the target operator distribution code is determined from the candidate operator distribution code based on the test results. The original operator distribution code for the target model in the deep learning base library is updated to the target operator distribution code, and the inference application of the target model is run on the target hardware platform based on the updated deep learning base library.

2. The RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution according to claim 1, characterized in that, The step of obtaining fine-grained shape information of the convolution operator in the target model to be accelerated includes: The original operator distribution code is embedded into the deep learning base library, and the inference application of the deep learning base library and the target model is compiled to generate the original executable program. The original executable program is run on the target hardware platform, and performance data is collected during the running process; Based on the performance data, fine-grained shape information of the convolution operator is extracted. The fine-grained shape information includes the input tensor shape, output tensor shape, convolution kernel size, stride size, padding parameters, and dilation rate.

3. The RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution according to claim 1, characterized in that, The target hardware platform is a processor platform that supports the Vector Extensions instruction set. The hardware characteristic information includes the memory hierarchy information, actual available memory bandwidth information, and vector register length parameters of the target hardware platform.

4. The RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution according to claim 1, characterized in that, The step of adjusting the selection strategy for multiple optional convolution algorithm implementation logics in the original operator distribution code based on the fine-grained shape information and the hardware characteristic information to generate candidate operator distribution code includes: The fine-grained shape information, the hardware characteristic information, and the original operator distribution code are input as prompt information into the large language model; The large language model is guided to analyze the adaptation relationship between the fine-grained shape information and the implementation logic of the various optional convolution algorithms; The large language model is guided to predict the execution efficiency of each convolution algorithm implementation logic on the target hardware platform based on the adaptation relationship and the hardware characteristic information. Based on the prediction results, the candidate operator distribution code is generated. The candidate operator distribution code is configured to distribute the convolution operator to the convolution algorithm implementation logic with the highest predicted execution efficiency.

5. The RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution according to claim 4, characterized in that, The step of performing performance testing on the candidate operator distribution code on the target hardware platform, and determining the target operator distribution code from the candidate operator distribution code based on the test results, includes: The current candidate operator distribution code generated by the large language model is embedded into the deep learning base library, and the inference application of the deep learning base library and the target model is compiled to generate a test program; Run the test program on the target hardware platform and record the actual execution time of the convolution operator under the current candidate operator distribution code; The actual execution time is compared with the original execution time corresponding to the original operator distribution code. If the reduction in the actual execution time compared to the original execution time exceeds a preset threshold, the current candidate operator distribution code and the corresponding actual execution time are added to the preferred set. The candidate operator distribution code with the shortest actual execution time is selected from the preferred set as the target operator distribution code.

6. The RISC-V model inference acceleration method based on irregular convolution operator adaptive distribution according to claim 5, characterized in that, The step of selecting the candidate operator distribution code with the shortest actual execution time from the preferred set as the target operator distribution code includes: Determine whether the number of candidate operator distribution codes in the preferred set has reached a preset number, or whether the number of code generation times of the large language model has reached a preset upper limit; If so, then stop generating new candidate operator distribution codes, and traverse the preferred set to select the candidate operator distribution code with the shortest actual execution time as the target operator distribution code; If not, the performance test results of the current candidate operator distribution code are fed back to the large language model, and the large language model is instructed to regenerate new candidate operator distribution code.

7. The RISC-V model inference acceleration method based on adaptive distribution of irregular convolution operators according to any one of claims 1 to 6, characterized in that, The convolution operator is an irregular convolution operator, and the implementation logic of the various optional convolution algorithms includes at least two of the following: a convolution algorithm based on general matrix multiplication, a Winograd convolution algorithm, a direct convolution algorithm, and a customized convolution algorithm for a convolution kernel of a specified size.

8. A RISC-V model inference acceleration device based on adaptive distribution of irregular convolution operators, characterized in that, include: The information acquisition unit is used to acquire fine-grained shape information of the convolution operators in the target model to be accelerated, as well as hardware characteristic information of the target hardware platform. The code generation unit is used to obtain the original operator distribution code of the convolution operator, and based on the fine-grained shape information and the hardware characteristic information, adjust the selection strategy of multiple optional convolution algorithm implementation logic in the original operator distribution code to generate candidate operator distribution code; The performance testing unit is used to perform performance testing on the candidate operator distribution code on the target hardware platform, and determine the target operator distribution code from the candidate operator distribution code based on the test results. The model execution unit is used to update the original operator distribution code for the target model in the deep learning base library to the target operator distribution code, and run the inference application of the target model on the target hardware platform based on the updated deep learning base library.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the RISC-V model inference acceleration method based on adaptive distribution of irregular convolution operators as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the RISC-V model inference acceleration method based on adaptive distribution of irregular convolution operators as described in any one of claims 1 to 7.

Images