Convolutional neural network compiling method based on cpu / npu cooperative calculation

Abstract

A convolutional neural network compilation method based on CPU / NPU collaborative computing, various CNN models are quantized and compressed under different network frameworks, and exported in ONNX file format; through the fusion of ONNX-MLIR tool chain and NPU compiler, the ONNX file compilation process is completed and the task list distributed to CPU and NPU is generated respectively, then ONNX-MLIR generates effective RISC-V instructions for CPU tasks on the CPU side, supports efficient scheduling, and the customized NPU compiler completes the matching of NPU tasks on the algorithm and hardware on the NPU side and efficiently maps the operators to the NPU platform. The present application solves the problem that CNN cannot run effectively on the heterogeneous architecture (RISC-V CPU / NPU) through the function function and the optimization core stage, and realizes the significant improvement of the computing performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and particularly relates to a convolutional neural network compiling method based on CPU / NPU collaborative calculation. BACKGROUND

[0002] In recent years, with the rapid development of convolutional neural network (CNN), neural network processing unit (NPU) is widely used due to its high energy efficiency and low delay. However, in the face of increasingly complex algorithm models, NPU has a bottleneck in processing performance, and although the traditional central processing unit (CPU) has strong versatility, it has low calculation efficiency. Combining the flexibility of CPU and the high efficiency of NPU provides an effective way to solve these problems. The existing research on CPU / NPU collaborative calculation is limited, and lacks mature architecture and tool chain support. SUMMARY

[0003] The present application proposes a convolutional neural network compiling method based on CPU / NPU collaborative calculation to solve the problem that CNN cannot run effectively on a heterogeneous architecture (RISC-V CPU / NPU) by optimizing the core stage through a function function, thereby achieving significant improvement in computing performance.

[0004] The present application is implemented by the following technical solutions:

[0005] The present application relates to a convolutional neural network compiling method based on CPU / NPU collaborative calculation, comprising:

[0006] Step 1, quantize and compress various CNN models under different network frameworks, and export them in ONNX file format;

[0007] Step 2, complete the ONNX file compiling process through the fusion of ONNX-MLIR tool chain and NPU compiler and generate task lists allocated to CPU and NPU, specifically including:

[0008] 2.1 In the ONNX dialect stage, through a task scheduling function and three different modes, allocate operations between CPU and NPU according to the specific needs of users;

[0009] 2.2 In the MLIR dialect stage, implement encapsulation of CNN operators belonging to CPU tasks divided by the task scheduling function through operator packaging technology, combine multiple operators into more efficient execution units to significantly reduce the switching overhead between CPU and NPU;

[0010] 2.3 In the LLVM dialect stage, the memory management of heterogeneous platforms is unified through memory management functions, and the memory space is compressed through memory optimization, ensuring the correctness of data interaction,

[0011] Step 3, realize the deep integration of CPU and NPU through RISC-V instruction extension, specifically: on the CPU side, ONNX-MLIR generates effective RISC-V instructions for CPU tasks, supports efficient scheduling, and on the NPU side, the customized NPU compiler completes the matching of NPU tasks on algorithms and hardware and efficiently maps operators to the NPU platform. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a flowchart of the present application;

[0013] Figure 2 is an operator packaging flowchart;

[0014] In the figure: (a) input model; (b) downgraded to ONNX dialect; (c) search and operator packaging; (d) mapped to hardware

[0015] Figure 3 is a common NPU memory optimization method;

[0016] In the figure: (a) Resnet50 (first stage BTNK2, original input feature map size is 224x224x3); (b) read-write conflict in double-end memory management mechanism; (c) memory management mechanism using post-release. DETAILED DESCRIPTION

[0017] As Figure 1 shown, a convolutional neural network compilation method based on CPU / NPU collaborative computing is provided, comprising:

[0018] Step 1, quantize and compress various CNN models under different network frameworks, and export in ONNX file format;

[0019] Step 2, complete the ONNX file compilation process through the fusion of ONNX-MLIR tool chain and NPU compiler and generate task lists allocated to CPU and NPU respectively, specifically including:

[0020] 2.1 In the ONNX dialect stage, through task scheduling functions and three different modes, operations are allocated between CPU and NPU according to specific user needs.

[0021] The three different modes include: full CPU mode, user-defined mode and default mode

[0022] The full CPU mode refers to forcing all tasks to be executed on the CPU regardless of whether they are suitable for the NPU. This mode is mainly used to verify the correctness of the calculation by comparing the results of the CPU and NPU calculations to ensure the accuracy and consistency of the tasks. For scenarios that require high-fidelity verification, this mode can provide reliable support.

[0023] The user-defined mode refers to the ability of advanced users to manually control task allocation. Users can specify task allocation through a binary string, where each bit represents the task allocation position of an operator: "0" indicates allocation to the CPU, and "1" indicates allocation to the NPU. This mode allows users to flexibly adjust the allocation strategy according to specific needs and hardware characteristics, and is used to evaluate the acceleration effect of CPU and NPU under different configurations, providing the possibility of in-depth analysis for performance optimization.

[0024] The default mode refers to automatically allocating tasks according to the capabilities of the NPU. The scheduler checks whether each operation meets the NPU's limit conditions, and if it does, it is allocated to the NPU, otherwise it is allocated to the CPU. This mode makes full use of the computing power of the NPU to optimize system performance. In addition, users can also adjust the range of tasks supported by the NPU by modifying the "NPU implementable operator" list, thereby analyzing the performance of specific operations on the NPU. This mode is particularly suitable for users who want to balance performance and flexibility.

[0025] 2.2 In the MLIR dialect stage, encapsulate the CNN operators belonging to CPU tasks divided by the task scheduling function through the operator packaging technique, combine multiple operators into more efficient execution units to significantly reduce the overhead of operation switching between CPU and NPU, which includes:

[0026] ① Add custom operators: define new operators in the ONNX dialect, including name, input, output, and attributes, and ensure that they can be recognized and parsed by the compiler; reduce high-level operators to the MLIR dialect to optimize their representation to meet hardware requirements; convert operators to the LLVM dialect to make them compatible with the LLVM backend and generate efficient machine code. The packaged operator does not need to include detailed operator or parameter information, but needs to retain the information of input and output variables to ensure integrity during dialect conversion.

[0027] ② Introduce three custom operators in the ONNX dialect to handle operations with different numbers of input variables, and complete operator optimization through custom CNN compilers.

[0028] Taking the MNIST network as an example, as shown in the following table: Figure 3(b) shows that the input computation graph is composed of a series of ONNX operators (such as Maxpool, Reshape, Gemm, ReLU and Softmax). In the downgrading process of ONNX-MLIR, these operators are mapped to the ONNX dialect and represented as function calls, preserving the structure of the computation graph. In further optimization, operators such as Gemm and ReLU are packaged as NPU functions, while operators such as Maxpool and Reshape are assigned to the CPU. These operators are finally executed in sequence between the CPU and the NPU, achieving efficient data transmission through the system bus.

[0029] The operator optimization includes: for the CPU, the adjacent operators are combined into a function to reduce function call and data transmission overhead; for the NPU, operator packaging focuses on maximizing throughput, common combinations include: convolution + activation function or pooling operation; convolution + addition + activation or pooling operation for optimizing the execution efficiency of residual network; matrix multiplication + activation or data transformation.

[0030] 2.3 In the LLVM dialect stage, the memory management of the heterogeneous platform is unified through the memory management function, and the memory space is compressed through memory optimization to ensure the correctness of data interaction.

[0031] The memory optimization includes: a double-ended (DE) mechanism for configuring self-checking functions and a use-once-release (RU) mechanism for simplifying compilation requirements on certain networks.

[0032] As shown in Figure 3 (c) shows that, taking the 224x224x3 original input feature map of Resnet50 as an example, the output feature map of NPU1 is released at the end of NPU2 calculation, but during the calculation of NPU3, a read-write conflict occurs due to insufficient space, resulting in an error in the calculation result of NPU3.

[0033] The double-ended (DE) mechanism refers to: traversing the total storage space of all layer input feature map data and output feature map data, and setting the maximum value as the current feature map storage space. When a read-write conflict occurs, the initially allocated memory space size is updated through self-checking.

[0034] Step 3: Deep integration of CPU and NPU is achieved through RISC-V instruction extension, specifically: on the CPU side,

[0035] ONNX-MLIR generates effective RISC-V instructions for CPU tasks, supporting efficient scheduling. On the NPU side, the customized NPU

[0036] The compiler completes matching of the NPU task on the algorithm and the hardware and efficiently maps the operator to the NPU platform. In addition, the RISC-V instruction extension serves as a key link of the hardware layer and provides core support for unified operation of the heterogeneous instruction set.

[0037] Through specific experiments, the application is deployed in a CPU / NPU heterogeneous computing system in a Python environment, the heterogeneous instruction set fusion of the CPU and the NPU is realized through the RISC-V instruction extension, the generality of the CPU is combined with the high efficiency of the NPU, cross-platform collaborative calculation of a complex CNN operator is supported, the overall performance and the resource utilization rate of the heterogeneous architecture are improved, the problem that the CNN cannot be effectively run on the heterogeneous architecture (RISC-V CPU / NPU) is solved, and the computing performance is significantly improved.

[0038] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the application, the protection scope of the application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme within the scope is subject to the constraints of the application.

Claims

1.A method for compiling a convolutional neural network based on CPU / NPU cooperative computing, characterized in that, The application relates to a method for compiling a convolutional neural network (CNN) model in an ONNX file format, and belongs to the technical field of artificial intelligence. The method comprises the following steps: Step 1, quantizing and compressing various CNN models under different network frameworks to export the models in an ONNX file format; Step 2, completing an ONNX file compiling process through the fusion of an ONNX-MLIR tool chain and an NPU compiler and respectively generating a task list distributed to a CPU and an NPU, and specifically comprising the following steps: 2.1 in the ONNX dialect stage, distributing operations between the CPU and the NPU according to specific user requirements through a task scheduling function and three different modes; The three different modes comprise a full-CPU mode, a user-defined mode and a default mode; 2.2 in the MLIR dialect stage, realizing the encapsulation of CNN operators belonging to CPU tasks divided by the task scheduling function through an operator packaging technology, combining multiple operators into more efficient execution units to significantly reduce the switching overhead between the CPU and the NPU; 2.3 in the LLVM dialect stage, uniformly managing the memory of a heterogeneous platform through a memory management function and compressing the memory space through memory optimization to ensure the correctness of data interaction, Step 3, realizing the deep fusion of the CPU and the NPU through RISC-V instruction extension, and specifically comprising the following steps: on the CPU side, the ONNX-MLIR generates effective RISC-V instructions for CPU tasks, supports efficient scheduling, on the NPU side, a customized NPU compiler completes the matching of NPU tasks on the algorithm and hardware and efficiently maps operators to the NPU platform; 2.The method of claim 1, wherein, The full-CPU mode refers to forcing all tasks to be executed on the CPU; the user-defined mode refers to the ability of a high-level user to manually control task allocation; and the default mode refers to automatically allocating tasks according to the capability of the NPU, that is, the scheduler checks whether each operation meets the limitation condition of the NPU, if yes, the operation is allocated to the NPU, otherwise, the operation is allocated to the CPU. The memory optimization comprises a double-end mechanism of a self-checking function and a use-and-release mechanism for simplifying the compiling requirement on certain networks; The double-end mechanism refers to traversing the total storage space of all layer input feature map data and output feature map data and setting the maximum value as the current feature map storage space, and when a read-write conflict occurs, the initially allocated memory space size is updated through self-checking.

Images