Basic C library fusion neural network acceleration method and system
By designing the basic C library and fusing the neural network acceleration module, hardware acceleration is achieved using NEON and RISC-V vector extension instruction sets, the problem of low efficiency of neural network inference on embedded devices is solved, and efficient neural network inference and simplified development process is achieved.
Patent Information
- Application Number
- CN202411550856.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2025-03-11
AI Technical Summary
The prior art is difficult to efficiently implement neural network inference on embedded devices, and lacks software libraries with high versatility and widespread application.
Design a basic C library, integrates neural network acceleration modules, provides public basic modules and neural network acceleration interfaces, and uses NEON and RISC-V vector extension instruction sets to achieve hardware acceleration.
Improves the operation efficiency of neural networks on embedded devices, reduces memory usage requirements, and simplifies development and integration through a unified API interface.
Smart Images
Figure CN119599074A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of computer software, and proposes a method and system for integrating basic C library with neural network acceleration. Background Art
[0002] Embedded systems have become an integral part of our lives. From smartphones to home appliances, almost every device is equipped with embedded technology. With the rapid development of artificial intelligence, it is becoming more and more common to apply neural networks to embedded devices. Neural networks are a computing model that simulates the human brain and are widely used in image recognition, natural language processing, sound recognition and other fields. Traditionally, these tasks require a lot of computing resources, which are usually completed by cloud servers. However, since embedded system applications are constrained by multiple factors, such as resources, power consumption, diversity, etc., neural network reasoning through embedded processor cores has become a technological development need.
[0003] When users use the functions provided by the operating system, they need to call the API exposed by the system. In various Linux distributions, in order to make it easier for users to use the API, the system will further encapsulate the API into C library functions and make the C library functions available to users. The basic C library is the lowest-level function library in the operating system. It provides a large number of basic functions, such as standard I / O functions, character processing functions, string processing functions, mathematical functions, memory allocators, etc., which basically implements the functions specified in the C99 standard. The basic C library does not depend on any other library and does not call functions in other libraries. The only thing the basic C library depends on is the kernel, because some functions must be implemented by the kernel. For example, the standard I / O function must finally call open(2), read(2), write(2), close(2) and other system calls to implement it. Due to the increasing application of neural networks, the relatively complex embedded development, and the high customization of the software stack, developers lack a highly versatile and widely used software library to implement embedded artificial intelligence reasoning.
[0004] Computer SIMD (Single Intruction Multiple Data) instructions can operate on multiple data at one time, reducing the number of instruction executions, replacing a single loop that originally required multiple cycles. For multimedia applications and vector function libraries, SIMD technology can significantly improve performance, so the current mainstream instruction set architectures will provide SIMD extension instruction sets, such as the NEON instruction set of the ARM architecture, the MMX / SSE instruction set of the Intel architecture, and the vector instruction set of the RISC-V architecture. The NEON instruction set is an extended instruction set of the ARM processor, used to accelerate multimedia and signal processing algorithms. NEON (Neon) is an ARM Advanced SIMD (Single Instruction Multiple Data) extension that provides a wide range of instruction sets and independent hardware execution units, especially suitable for video encoding / decoding, 2D / 3D graphics, games, audio and speech processing, image processing technology and other fields. Similar to the ARM NEON instruction set, RISC-V vector instructions are also SIMD instructions, have independent register files, and support vector operations of 8-bit, 16-bit, 32-bit, 64-bit and other data types. However, compared with the ARM architecture's NEON instruction set, it has the following three characteristics: the hardware vector length is not only unknown at compile time, but may also change during program execution; vector register files can be configured to different data types and sizes through software; almost all vector instructions support mask operations.
[0005] In order to realize the acceleration of neural network reasoning for hardware kernels, the key technical path is to use SIMD technology to provide low-level machine learning hardware acceleration libraries for developers in the fields of image / video / multimedia / computer vision. There are four ways to use SIMD technology in programs: ① Use intrinsics implemented by the compiler in the program. This method is relatively simple and easy to maintain. Intrinsics will be directly converted into SIMD assembly instructions during compilation; ② Directly use function libraries that support SIMD instruction technology, such as Compute Library and Ne10 open source function libraries that support ARM NEON instruction sets. These function libraries provide C / C++ interfaces, so developers do not need to know the specific ARM NEON instructions, and these algorithm function libraries are often optimized and have high performance; ③ Use the auto-vectorization option when compiling the program to let the compiler automatically generate SIMD instructions. There is no need to modify the source code, and it is easy to port across platforms. Therefore, this is the simplest and most direct method, but for complex algorithms, the compiler's optimization effect is very poor; ④ Hand-written SIMD assembly, this method has poor portability and high difficulty. Summary of the invention
[0006] The purpose of this patent is to provide a method and system for integrating basic C library with neural network acceleration to solve the problems existing in the prior art.
[0007] The technical solution adopted by the present invention to achieve its purpose is as follows:
[0008] A method for accelerating neural network by integrating basic C library includes the following steps:
[0009] A lightweight C standard library is used as the basic C library, and an NN acceleration module for neural network acceleration is added to the basic C library; the basic C library provides a public basic module to encapsulate common neural network operation interfaces; the NN acceleration module includes three modules: core, graph, and runtime, which provide neural network acceleration interfaces;
[0010] When training models based on mainstream neural network frameworks, the trained models are quantized and converted to obtain deep learning models.
[0011] The NN acceleration module calls the graph and runtime modules to complete the construction of the computational graph of the deep learning model, and calls the core and runtime modules for inference calculations.
[0012] Furthermore, the Core.h header file is defined in the core module, declaring a unified neural network function and neural network support function API; the neural network function includes convolution function, pooling function, full connection function and activation function, and the neural network support function includes data type conversion function and activation table.
[0013] Furthermore, ARM and RISC-V platforms are distinguished through macro definition distinction and conditional compilation methods to achieve NEON and RISC-V vector extension optimization.
[0014] Furthermore, the NN acceleration module adopts NEON SIMD technology to provide a machine learning hardware acceleration library for the ARM platform.
[0015] Furthermore, the NN acceleration module adopts RVV SIMD technology to provide a machine learning hardware acceleration library for the RISC-V platform.
[0016] Furthermore, when compiling the basic C library and NN acceleration module, the platform architecture is configured by configuring the --host option.
[0017] Furthermore, when compiling the basic C library and the NN acceleration module, the aarch64 and riscv64 tool chains built using the basic C library are used for compilation.
[0018] Furthermore, the ARM platform uses NEON intrinsics and NEON inline assembly instructions to implement hardware accelerated programming for some algorithm interfaces defined in Core.h.
[0019] Furthermore, the RISC-V platform uses RVV intrinsics and RVV inline assembly instructions to implement hardware accelerated programming for some algorithm interfaces defined in Core.h.
[0020] A system integrating basic C library and neural network acceleration, including:
[0021] The basic C library uses a lightweight C standard library to provide common basic modules and encapsulate common neural network operation interfaces;
[0022] NN acceleration module provides a neural network acceleration interface, including three modules: core, graph and runtime. The core module defines the Core.h header file, declaring a unified neural network function and neural network support function API. The graph module is responsible for model conversion, parsing and quantization, and completes the construction and optimization of the computational graph. The runtime module implements memory allocation and multi-threaded operations, and provides convolution algorithm acceleration and image processing functions.
[0023] Beneficial effects achieved by the present invention:
[0024] 1. The present invention adopts a lightweight C standard library as the basic C library, which is designed as a substitute for GNU C library (glibc), uClibc or Android Bionic, and is used in embedded operating systems and mobile devices. The basic C library focuses on reducing code size and function call overhead as much as possible, improving neural network reasoning performance, and its memory allocation strategy reduces memory fragmentation and improves memory utilization, which is beneficial to embedded devices or distributed deployment.
[0025] 2. The present invention integrates neural network acceleration technology with the basic C library, providing fine-grained operator interfaces from bottom to top to provide model loading, parsing and reasoning capabilities. That is, a general NN acceleration module is introduced in the basic C library for ARM and RISCV processors, and the NEON instruction set and RISC-V vector instruction set are used to provide NN acceleration interfaces respectively. It is upwardly compatible with the basic C library to provide a unified interface to the outside world, and calls the kernel underlying interface downward to apply machine learning to embedded systems, providing high-performance neural network reasoning capabilities. The NN acceleration module is connected to the neural network application code through a unified API interface, and external applications can use the acceleration function without understanding the internal implementation details, thereby simplifying development and integration.
[0026] 3. The NN acceleration module has multiple new neural network functions for Arm CPU and RISCV CPU. By optimizing key operations such as convolution, pooling, full connection and activation, it can significantly improve the operating efficiency of neural networks on embedded devices. It supports the use of functions such as data type conversion and activation tables, which can effectively reduce memory usage requirements. It supports hardware acceleration using architecture assembly instructions, performs micro-architecture optimization for key neural network interfaces, and reduces memory overhead for complex networks. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 This is the overall architecture diagram of the basic C library with the NN acceleration module added.
[0028] Figure 2 This is the architecture diagram of the NN acceleration module.
[0029] Figure 3 This is the composition diagram of the Core module of the NN acceleration module.
[0030] Figure 4 This is the application ecology diagram of the NN acceleration module.
[0031] Figure 5 It is the execution flow graph when the NN acceleration module is running.
[0032] Figure 6 This is an application example diagram of the NN acceleration module. DETAILED DESCRIPTION
[0033] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.
[0034] The embodiment of the present invention discloses a method and system for integrating neural network acceleration with a basic C library. An NN (Neural Network) acceleration module is added to a lightweight basic C library for mainstream architecture processors. The basic C library provides a public basic module to encapsulate commonly used neural network operation interfaces, such as NEON intrinsics, RVV intrinsics, and standard I / O functions. The NN acceleration module provides a neural network acceleration interface from the basic C library level, such as Figure 1 The NN acceleration module is a collection of effective neural network kernels that maximizes the performance of neural networks on processors targeting embedded devices while minimizing memory usage.
[0035] like Figure 2The core components of the NN acceleration module shown include the core module, graph module and runtime (runtime library) module. The core module defines the Core.h header file, which declares a unified neural network common function and utility function API. The graph module implements functions such as model conversion, parsing and quantization, as well as computational graph construction and optimization. The runtime module implements memory allocation and multi-threaded operations. The runtime library is a wrapper for the core library and provides other additional functions, such as memory management, multi-threaded execution of workloads, convolution algorithm acceleration, and image processing.
[0036] like Figure 2 As shown in the figure, the NN acceleration module adopts SIMD technology and uses NEON and RVV instructions to implement hardware accelerated operations. For the ARM architecture platform, some algorithms are implemented using ARM NEON intrinsics or ARM NEON inline assembly instructions, where intrinsics are provided by arm_neon.h. Similarly, for the RISC-V architecture platform, RVVinstrinsics or RVV assembly instructions are used, where intrinsics are provided by riscv_vector.h. The NN acceleration module uses macro definitions and conditional compilation (ifdef) methods to achieve deep learning operator instruction-level optimization through standard NEON / RVV extensions, and implements a set of low-level machine learning functions optimized for the ARM / RISC-V series processors.
[0037] Different preprocessor macros are defined during the implementation process to distinguish the vector types and function interfaces defined by ARM and RISC-V intrinsics, and to distinguish different hardware acceleration implementations, for example:
[0038] ARM_NEON (represents the implementation of ARM NEON acceleration):
[0039] #ifndef__ARM_NEON_H__
[0040] #define__ARM_NEON_H__
[0041] #include<arm_neon.h>
[0042] #endif
[0043] RISCV_VECTOR (represents the implementation of RISC-V RVV acceleration):
[0044] #ifndef__RISCV_VECTOR_H__
[0045] #define__RISCV_VECTOR_H__
[0046] #include<riscv_vector.h>
[0047] #endif
[0048] During the compilation process, configure the architecture platform by configuring the "--host" option, for example:
[0049] configure --host=aarch64 (represents the 64-bit version of the ARM architecture)
[0050] configure --host=riscv64 (represents the 64-bit version of the RISC-V architecture)
[0051] like Figure 3 As shown, the Core module of the NN acceleration module contains two parts: neural network function functions (NNFunctions) and neural network support functions (NNSupportFunctions).
[0052] (1) Neural network function functions include APIs for implementing common neural network operations. These functional interfaces are connected to the neural network application code to perform various neural network operations. Specifically, they include:
[0053] ① Convolution function: implements the convolution operation, which is a key operation in neural networks (especially convolutional neural networks CNN) and is used for feature extraction.
[0054] ②Pooling function: used to reduce dimensionality, reduce the amount of data and computational burden, while retaining important features.
[0055] ③Fully-connected function: connects all input nodes and output nodes to achieve the mapping between features and classification results.
[0056] ④Activation function: Implement activation function operations, such as ReLU, Sigmoid, etc., to introduce nonlinearity.
[0057] (2) The neural network support function includes different practical functions. The support function further improves the overall performance of the system by optimizing data processing and accelerating calculations. Specifically, it includes:
[0058] ①Data type conversion function: responsible for converting different data formats to ensure that the neural network can run on different hardware platforms.
[0059] ②Activation tables: Provides pre-calculated activation function value tables to accelerate the calculation of activation functions.
[0060] Figure 4 It is an application ecological diagram of the NN acceleration module. The complete ecology of the NN acceleration module is divided into hardware layer, driver layer, software infrastructure layer and industry application layer from bottom to top. Since the NN acceleration library enables ARM and RISC-V processors for accelerated reasoning respectively, the hardware bottom layer depends on the CPU, NPU and other hardware of the AMR and RISC-V architectures. The driver layer provides hardware information and external communication capabilities, including basic drivers for the hardware platform and a series of Linux operating systems compatible with the NN acceleration library. As an important part of the software infrastructure layer, the NN acceleration module is integrated into the basic C library with reference to the open source acceleration library as the cornerstone, and provides a unified interface to the outside. In order to be compatible with the basic C library, the present invention replaces the underlying C library interface of the open source acceleration library with the public basic module interface of the basic C library. Based on the external interface of the NN acceleration module in the basic C library, users can develop customized AI application services on the embedded side, including vision, voice and signal applications.
[0061] Figure 5 The figure shows the execution flow of the NN acceleration module at runtime. The API of the NN acceleration module is directly connected to the underlying library of the CPU. An efficient NN acceleration module is the key to enabling reasoning based on ARM / RISC-V CPU. The NN acceleration module provides optimized functions to accelerate key neural network operators such as convolution, pooling, and activation.
[0062] Figure 6 The following is an application example of the NN acceleration module. Model training is performed based on mainstream neural network frameworks (TensorFlow / Caffe / Pytorch, etc.), and the trained model (such as the yolov3 model) is quantized and converted, so that the model is input to the loading function interface implemented by the NN acceleration module, and the graph and runtime module interfaces are called to complete the loading of the yolov3 model, and the model parsing function and the calculation graph construction function are called at the same time. At the same time, the core and runtime module interfaces of the NN acceleration module are called to build the calculation graph, implement the core steps of the example reasoning, implement the corresponding test data pre-processing and reasoning result post-processing interfaces of the yolov3 model, and compile using the aarch64 and riscv64 tool chains built with the basic C library, respectively. The underlying assembly programming interface is called to complete reasoning acceleration and optimization, improve reasoning accuracy, and enhance reasoning performance.
[0063] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.
Claims
1. A method for accelerating neural network by integrating basic C library, characterized in that: The following steps are involved: A lightweight C standard library is used as the basic C library, and an NN acceleration module for neural network acceleration is added to the basic C library; the basic C library provides a public basic module to encapsulate common neural network operation interfaces; the NN acceleration module includes three modules: core, graph, and runtime, which provide neural network acceleration interfaces; When training models based on mainstream neural network frameworks, the trained models are quantized and converted to obtain deep learning models. The NN acceleration module calls the graph and runtime modules to complete the construction of the computational graph of the deep learning model, and calls the core and runtime modules for inference calculations.
2. The method according to claim 1, characterized in that The Core.h header file is defined in the core module, declaring a unified neural network function and neural network support function API; the neural network function includes convolution function, pooling function, full connection function and activation function, and the neural network support function includes data type conversion function and activation table.
3. The method according to claim 1, characterized in that ARM and RISC-V platforms are distinguished through macro definition and conditional compilation methods to achieve NEON and RISC-V vector extension optimization.
4. The method according to claim 2, characterized in that The NN acceleration module uses NEON SIMD technology and provides a machine learning hardware acceleration library for the ARM platform.
5. The method according to claim 2, characterized in that The NN acceleration module uses RVV SIMD technology and provides a machine learning hardware acceleration library for the RISC-V platform.
6. The method according to claim 2, characterized in that When compiling the basic C library and NN acceleration module, configure the platform architecture by configuring the --host option.
7. The method according to claim 2, characterized in that When compiling the basic C library and NN acceleration module, compile using the aarch64 and riscv64 toolchains built with the basic C library.
8. The method according to claim 2, characterized in that The ARM platform uses NEON intrinsics and NEON inline assembly instructions to implement hardware accelerated programming for some algorithm interfaces defined in Core.h.
9. The method according to claim 2, characterized in that The RISC-V platform uses RVV intrinsics and RVV inline assembly instructions to implement hardware accelerated programming for some algorithm interfaces defined in Core.h.
10. A system integrating basic C library and neural network acceleration, characterized in that: include: The basic C library uses a lightweight C standard library to provide common basic modules and encapsulate common neural network operation interfaces; NN acceleration module, which provides neural network acceleration interface, including core, graph and runtime modules; The core module defines the Core.h header file, declaring a unified neural network function and a neural network support function API; the graph module is responsible for model conversion, parsing, and quantization, and completes the construction and optimization of the computational graph; the runtime module implements memory allocation and multi-threaded operations, and provides convolution algorithm acceleration and image processing functions.