An NPU instruction optimization system and method for OpenCL heterogeneous computing

Abstract

The application provides an NPU instruction optimization system and method for OpenCL heterogeneous computing. An OpenCL compiling and runtime layer is inserted between a host end and an NPU device. Standard OpenCL kernel code is deeply analyzed, and a neural network operator that can be optimized is identified and converted into a special intermediate representation designed by the NPU. The special intermediate representation is mapped to a special hardware unit and a kernel system of the NPU device to generate an instruction stream containing NPU special instructions. The application can efficiently map high-level operators and data patterns related to neural network computing that are implied in the standard OpenCL kernel code to the special hardware unit and the memory system of the self-developed NPU through identification, conversion and fusion technologies while supporting the OpenCL standard, so that the performance level of the NPU native private software stack is approached or even reached while the programming convenience of OpenCL is maintained.

Description

Technical Field

[0001] This invention relates to the field of NPU neural network processor technology, and more specifically, to an OpenCL heterogeneous computing NPU instruction optimization system and optimization method. Background Technology

[0002] With the explosive growth of artificial intelligence applications, neural network processors have become key hardware due to their high energy efficiency in highly parallel, computationally intensive tasks. However, NPU architectures are diverse, and each vendor typically provides proprietary software stacks and programming models (such as dedicated drivers, compilers, and runtime libraries), leading to serious software portability and lock-in issues for application developers. Code written for one NPU is difficult to reuse on other NPUs, significantly increasing development costs and fragmenting the ecosystem.

[0003] Against this backdrop, OpenCL, as an open, cross-platform heterogeneous computing standard, provides a unified framework for programming devices such as CPUs, GPUs, and DSPs. Introducing OpenCL into our self-developed NPU can theoretically create an open and easy-to-use programming environment, attracting a wider range of developers.

[0004] Existing general OpenCL-to-NPU mapping schemes cannot efficiently utilize the unique hardware acceleration units and parallel architecture of self-developed NPUs, resulting in performance (throughput, latency) and energy efficiency that are far lower than when directly using the native proprietary software stack of the NPU when running typical neural network workloads. Summary of the Invention

[0005] This invention addresses the technical problems existing in the prior art by providing an OpenCL heterogeneous computing NPU instruction optimization system and optimization method.

[0006] According to a first aspect of the present invention, an NPU instruction optimization system for OpenCL heterogeneous computing is provided, comprising a host, an OpenCL compilation and runtime layer, and an NPU device;

[0007] The standard OpenCL application on the host computer initiates a computation request by calling the OpenCL API;

[0008] The OpenCL compilation and runtime layer receives standard OpenCL kernel code, performs in-depth analysis on the standard OpenCL kernel code, identifies optimizable neural network operators, and converts the neural network operators into a dedicated intermediate representation for NPU design.

[0009] The dedicated intermediate representation is mapped onto the dedicated hardware unit and kernel architecture of the NPU device to generate an instruction stream containing NPU-specific instructions, and the instruction stream is sent to the NPU device.

[0010] The NPU device receives the instruction stream and executes computational tasks based on dedicated hardware units and kernel architecture.

[0011] Based on the above technical solution, the present invention can also be improved as follows.

[0012] Optionally, the OpenCL compilation and runtime layer includes a pattern recognition and kernel fusion engine, an intermediate representation layer, an optimized instruction generator, and an NPU driver adaptation layer;

[0013] The compiler front-end of the OpenCL compilation and runtime layer is used to compile OpenCL C source code into LLVM IR when standard OpenCL kernel code is received;

[0014] The pattern recognition and kernel fusion engine is used to identify computational subgraphs related to neural networks from the LLVM IR through static code analysis and data flow analysis, and to mark the computational subgraphs as high-level neural network operators;

[0015] The intermediate representation layer converts the high-level neural network operators into a dedicated intermediate representation designed for the NPU.

[0016] The optimized instruction generator is used to map the dedicated intermediate representation onto the dedicated hardware unit and kernel architecture of the NPU device, and generate low-level instructions based on the hardware characteristics of the NPU device.

[0017] The NPU driver adaptation layer is used to translate the underlying instructions into an optimized instruction stream containing NPU-specific instructions, and then send the optimized instruction stream to the NPU device.

[0018] Optionally, the pattern recognition and kernel fusion engine is used to identify computational subgraphs related to neural networks from the LLVM IR through static code analysis and data flow analysis, and to label the computational subgraphs as high-level neural network operators, including:

[0019] The pattern recognition and kernel fusion engine identifies target computation patterns from the LLVM IR through static code analysis and data flow analysis, and marks them as high-level neural network operators.

[0020] Optionally, the optimized instruction generator is used to map the dedicated intermediate representation onto the dedicated hardware units and kernel architecture of the NPU device, and generate low-level instructions based on the hardware characteristics of the NPU device, including:

[0021] Based on the hardware characteristics of the NPU device, the dedicated intermediate representation is deeply optimized:

[0022] Mapping advanced neural network operators to dedicated instructions for dedicated hardware units of the NPU device;

[0023] Based on the data access patterns of advanced neural network operators, a data flow strategy in the multi-level memory of the NPU device is planned to maximize data reuse and minimize off-chip access.

[0024] This involves fusing adjacent, fusionable high-level neural network operators into a single composite kernel at an intermediate representation level.

[0025] Optionally, identifying target computation patterns from the LLVM IR and labeling them as high-level neural network operators includes:

[0026] Each target computation mode is identified from the LLVM IR, and each specific technical mode corresponds to a high-level neural network operator. The name of each high-level neural network operator is then labeled.

[0027] Optionally, the optimized instruction generator is used to map the dedicated intermediate representation onto the dedicated hardware units and kernel architecture of the NPU device, and generate low-level instructions based on the hardware characteristics of the NPU device, including:

[0028] Map the dedicated intermediate representation of each advanced neural network operator to the dedicated instructions of the corresponding dedicated hardware unit of the NPU device;

[0029] And based on the data access mode of each advanced neural network operator, the computational data corresponding to each advanced neural network operator is mapped to the memory of the corresponding level.

[0030] According to a second aspect of the present invention, an NPU instruction optimization method for OpenCL heterogeneous computing is provided, comprising:

[0031] Standard OpenCL applications on the host machine initiate computation requests by calling the OpenCL API;

[0032] The OpenCL compilation and runtime layer receives standard OpenCL kernel code, performs in-depth analysis on the standard OpenCL kernel code, identifies optimizable neural network operators, and converts the neural network operators into a dedicated intermediate representation for NPU design.

[0033] The dedicated intermediate representation is mapped onto the dedicated hardware unit and kernel architecture of the NPU device to generate an instruction stream containing NPU-specific instructions, and the instruction stream is sent to the NPU device.

[0034] The NPU device receives the instruction stream and executes computational tasks based on dedicated hardware units and kernel architecture.

[0035] Optionally, a deep analysis is performed on the standard OpenCL kernel code to identify optimizable neural network operators, and the neural network operators are converted into a dedicated intermediate representation for NPU design, including:

[0036] When standard OpenCL kernel code is received, the OpenCL C source code is compiled into LLVM IR;

[0037] Used to identify computational subgraphs related to neural networks from the LLVM IR through static code analysis and data flow analysis, and to label the computational subgraphs as high-level neural network operators;

[0038] The advanced neural network operators are converted into a dedicated intermediate representation for NPU design;

[0039] The dedicated intermediate representation is mapped onto the dedicated hardware units and kernel architecture of the NPU device to generate an instruction stream containing NPU-specific instructions, including:

[0040] The dedicated intermediate representation is mapped onto the dedicated hardware unit and kernel architecture of the NPU device, and the underlying instructions are generated according to the hardware characteristics of the NPU device.

[0041] The underlying instructions are translated into an optimized instruction stream that includes NPU-specific instructions.

[0042] This invention provides an OpenCL heterogeneous computing NPU instruction optimization system and method. By inserting an OpenCL compilation and runtime layer between the host and the NPU device, it can efficiently map the high-level operators and data patterns related to neural network computing implicit in the standard OpenCL kernel code to the dedicated hardware units and memory system of the self-developed NPU through identification, conversion and fusion technologies, while supporting the OpenCL standard. This allows the system to maintain the convenience of OpenCL programming and approach or even reach the performance level of the NPU's native private software stack. Attached Figure Description

[0043] Figure 1 A schematic diagram of the structure of an OpenCL heterogeneous computing NPU instruction optimization system provided in one embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the working process of a pattern recognition and kernel fusion engine according to an embodiment of the present invention;

[0045] Figure 3This is a schematic diagram of a pattern recognition and kernel fusion engine recognizing neural network operators according to an embodiment of the present invention;

[0046] Figure 4 This is an architecture diagram of an NPU instruction optimization system for applying OpenCL heterogeneous computing on a self-developed NPU device, according to an embodiment of the present invention.

[0047] Figure 5 This is a flowchart illustrating an OpenCL heterogeneous computing NPU instruction optimization method according to an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0049] Before providing a detailed description of the solutions provided by this invention, the technical terms involved in this invention will be explained.

[0050] LLVM: An open-source compiler infrastructure project whose modular design makes it well-suited as a backend for building dedicated compilers.

[0051] Intermediate means a program representation used by the compiler between source code and machine code, which facilitates machine-independent optimization.

[0052] Tensor cores: Hardware units specifically designed to perform large-scale matrix multiplication and addition operations, commonly found in modern AI accelerators.

[0053] The core of this invention is inserting an OpenCL compilation and runtime layer between the host and the NPU device. The entire optimized system architecture and workflow can be found in [reference needed]. Figure 1 ,according to Figure 1 An embodiment of the present invention provides an NPU instruction optimization system for heterogeneous OpenCL computing, which mainly includes a host, an OpenCL compilation and runtime layer, and an NPU device.

[0054] The working principle of the NPU instruction optimization system is as follows:

[0055] Standard OpenCL applications on the host machine initiate computation requests by calling the OpenCL API;

[0056] The OpenCL compilation and runtime layer receives standard OpenCL kernel code, performs in-depth analysis on the standard OpenCL kernel code, identifies optimizable neural network operators, and converts the neural network operators into a dedicated intermediate representation for NPU design.

[0057] The dedicated intermediate representation is mapped onto the dedicated hardware unit and kernel architecture of the NPU device to generate an instruction stream containing NPU-specific instructions, and the instruction stream is sent to the NPU device.

[0058] The NPU device receives the instruction stream and executes computational tasks based on dedicated hardware units and kernel architecture.

[0059] Understandably, when an OpenCL computation request is initiated on the host side, the OpenCL compilation and runtime layer receives the OpenCL kernel code and performs in-depth analysis. It identifies operators related to neural networks from the kernel code and converts these operators into intermediate representations for the NPU design. Deep optimization of these intermediate representations primarily involves mapping them to the dedicated hardware units and kernel architecture of the NPU device, generating an instruction stream that the NPU device can recognize and execute. This instruction stream is then sent to the NPU device, which executes the computation task based on the dedicated hardware units and kernel architecture.

[0060] In one embodiment of the present invention, the OpenCL compilation and runtime layer includes a pattern recognition and kernel fusion engine, an intermediate representation layer, an optimized instruction generator, and an NPU driver adaptation layer.

[0061] The compiler front-end is used to compile OpenCL C source code into LLVM IR when it receives standard OpenCL kernel code;

[0062] The pattern recognition and kernel fusion engine is used to identify computational subgraphs related to neural networks from the LLVM IR through static code analysis and data flow analysis, and to mark the computational subgraphs as high-level neural network operators;

[0063] The intermediate representation layer converts the high-level neural network operators into a dedicated intermediate representation designed for the NPU.

[0064] The optimized instruction generator is used to map the dedicated intermediate representation onto the dedicated hardware unit and kernel architecture of the NPU device, and generate low-level instructions based on the hardware characteristics of the NPU device.

[0065] The NPU driver adaptation layer is used to translate the underlying instructions into an optimized instruction stream containing NPU-specific instructions, and then send the optimized instruction stream to the NPU device.

[0066] Understandably, see Figure 1 As shown, the OpenCL compilation and runtime layer mainly includes a pattern recognition and kernel fusion engine, an intermediate representation layer, an optimization instruction generator, and an NPU driver adaptation layer.

[0067] In this process, the call is made at the application layer on the host side. Standard OpenCL applications initiate computation requests through OpenCL APIs (such as clCreateBuffer to create memory and clEnqueueNDRange to submit tasks).

[0068] When the compiler frontend receives the standard OpenCL kernel code, it compiles the OpenCL C source code into LLVM IR.

[0069] The pattern recognition and fusion engine performs deep analysis of LLVM IR, identifying and fusing optimizable neural network operators (such as convolution and pooling). The intermediate representation layer converts the identified neural network operators into intermediate representations designed specifically for the NPU, serving as a bridge between high-level abstractions and low-level hardware.

[0070] The instruction generator maps intermediate representations to the corresponding dedicated hardware units and kernel architecture of the NPU, and generates efficient low-level instructions based on hardware characteristics.

[0071] The NPU driver adaptation layer translates the underlying instructions into a device-executable instruction stream, which is then sent to the NPU device for execution by a dedicated matrix computing unit and an efficient memory system, thereby achieving high-performance computing.

[0072] The core idea of ​​this invention is to resolve the conflict between versatility and performance by inserting an intelligent and efficient compilation and runtime layer between the standard OpenCL interface and the dedicated NPU hardware.

[0073] In one embodiment of the present invention, the pattern recognition and kernel fusion engine identifies the target computation pattern from LLVM IR through static code analysis and data flow analysis, and marks it as a high-level neural network operator.

[0074] For details, please refer to Figure 2When the host calls clBuildProgram, the enhanced compiler front end not only compiles the OpenCL C code into hardware-independent LLVM IR, but also starts the pattern recognition and kernel fusion engine.

[0075] This pattern recognition and kernel fusion engine identifies various computational subgraphs related to neural networks in LLVM IR through static code analysis and data flow analysis, and labels the type of each computational subgraph. For example, it can identify a pattern consisting of nested loops, load instructions, and multiply-accumulate operations, and label it as a "convolution operator." Specifically, the pattern recognition and kernel fusion engine receives LLVM IR and uses static code analysis and data flow analysis techniques to find target computational patterns. For example, the pattern recognition and kernel fusion engine identifies target computational patterns in LLVM IR consisting of nested loops, memory load instructions, and multiply-accumulate operations.

[0076] In this embodiment of the invention, the target computation mode refers to a set of code snippets in LLVM IR that correspond to the core computation process of the neural network and have fixed semantics and data flow characteristics. The core includes two key modes:

[0077] 1) The underlying code patterns of basic neural network operators: These correspond to the smallest computational units such as convolution, matrix multiplication, activation functions (such as ReLU), and pooling. Static code analysis can identify their input and output tensor features, precision configuration (such as FP8 / INT8), and operational logic, which are the foundation for building advanced neural network operators.

[0078] 2) Fusionable combinatorial computing modes: refer to pipelined computing sequences formed by multiple consecutive minimum computing units (such as "convolution + batch normalization + activation"). Data dependencies and execution links can be identified through data flow analysis. After marking, they can be fused and optimized by the compiler's operators to reduce data transfer overhead and improve execution efficiency.

[0079] The core value of these patterns is to achieve the mapping from general LLVM IR to dedicated neural network operators, adapt to AI chip hardware architecture, and maximize the efficiency of computing power release. This is in perfect agreement with the compiler's technical path of "achieving multi-stage optimization and supporting automatic operator fusion generation through MLIR architecture".

[0080] This target computation mode corresponds to the original convolution source code. The pattern recognition and kernel fusion engine ultimately marks (abstracts) this series of low-level instructions into a semantically clear high-level neural network operator—CONV_2D_OPERATOR—carrying all the necessary computation parameters.

[0081] according to Figure 2This clearly demonstrates that the technology of this invention does not replace the standard compilation chain, but rather adds an intelligent recognition layer on top of it, thereby bridging the gap between general-purpose OpenCL code and dedicated high-efficiency hardware.

[0082] For the intermediate representation layer, the identified high-level neural network operators (such as CONV_2D, GEMM, POOL) are converted into a dedicated intermediate representation for the self-developed NPU architecture. NPU-IR is no longer a general instruction, but an abstract computational description carrying meta-information such as tensor dimensions, data layout, and fusion strategies.

[0083] The instruction generator receives the NPU-IR and performs deep optimization based on the hardware characteristics of the self-developed NPU. The deep optimization mainly includes the following aspects:

[0084] ① Computational mapping: Each identified high-level neural network operator is mapped to a dedicated instruction of the corresponding dedicated hardware unit of the NPU device. For example, the GEMM (Generalized Matrix Multiplication) operator is directly mapped to the dedicated instruction of the matrix multiplication and addition unit of the NPU, rather than being decomposed into mad instructions.

[0085] ② Memory optimization: Based on the data access pattern of the operator, plan the data flow strategy in the NPU multi-level memory (such as InputFeature Buffer, Weight Buffer, Output Buffer) to maximize data reuse and minimize off-chip access.

[0086] ③ Kernel Fusion: Adjacent, fusionable operators (such as CONV -> ReLU -> POOL) are fused into a composite kernel at the NPU-IR level. This way, only one task needs to be issued, and the data flows within the NPU, avoiding the need to write back and reload intermediate results.

[0087] For example, see the convolution operator. Figure 3 This clearly demonstrates the core working process of the pattern recognition and kernel fusion engine in this invention:

[0088] 1) Input: OpenCL C kernel code, which contains the necessary components for implementing convolution calculations:

[0089] 2) Nested loop structure: used to traverse convolution kernels.

[0090] 3) Memory read operation: Load input data and weights from global memory.

[0091] 4) Multiplication, accumulation, and addition operations: the core arithmetic logic for completing calculations.

[0092] 5) Processing: The intermediate pattern recognition and kernel fusion engine performs static analysis on the input code. It identifies that the above code patterns (loop, data reading, computation) together constitute a typical two-dimensional convolution operation.

[0093] 6) Output: The pattern recognition and kernel fusion engine ultimately outputs an abstract, high-level CONV_2D_OPERATOR (two-dimensional convolution operator). This operator is no longer a hardware-independent, fine-grained instruction, but a semantic operation description containing all necessary parameters (such as input, weights, stride, padding, etc.). This high-level description forms the basis for subsequent deep optimizations (such as mapping to tensor kernels and kernel fusion).

[0094] The instruction generator, based on the hardware characteristics of the NPU device, maps the identified neural network operators to the dedicated hardware units of the NPU device. The driver adaptation layer then sends the optimized instruction stream (containing NPU-specific instructions and memory configuration commands) to the NPU device. Based on this optimized instruction stream, the NPU device efficiently schedules its dedicated computing units and memory system to execute tasks.

[0095] This invention is particularly suitable for scenarios requiring the porting of AI inference applications across different brands of NPUs, such as smartphones, autonomous vehicles, and smart security systems. Developers can write OpenCL code once and achieve high performance on devices integrating this invention's self-developed NPU, without needing to rewrite and deeply optimize for each platform.

[0096] The specific architecture of the OpenCL heterogeneous computing NPU instruction optimization system provided by this invention can be found in [reference needed]. Figure 4 The OpenCL compilation and runtime layer is a core module of the OpenCL compiler for NPUs (Neural Processing Units), called the NPU OpenCL Compiler. It encapsulates a complete compilation process designed to efficiently compile general-purpose OpenCL code for execution on dedicated NPU hardware.

[0097] The NPU OpenCL compiler integrates several core components:

[0098] 1) Pattern recognition and kernel fusion engine: used for analyzing and extracting advanced operators.

[0099] 2) Intermediate representation layer and instruction optimization generator: responsible for the conversion and optimization of intermediate representation.

[0100] 3) Kernel Fusion Engine: Specifically optimized for kernel fusion.

[0101] 4) Code generator: ultimately generates the NPU instruction stream.

[0102] The core compilation process of the NPU OpenCL compiler consists of five key steps:

[0103] 1) Front-end compilation: First, the input OpenCL source code is converted into a general LLVM intermediate representation using the standard LLVM compiler front-end.

[0104] 2) Pattern Recognition and Operator Extraction: Next, the pattern recognition and kernel fusion engine analyzes the LLVM IR to identify and extract advanced operators suitable for execution on the NPU.

[0105] 3) Generate NPU-specific IR: Then, the intermediate representation layer transforms and optimizes these high-level operators into a dedicated intermediate representation for NPU hardware.

[0106] 4) Kernel Fusion Optimization: The kernel fusion engine will further optimize the NPU IR by merging multiple small kernels into a more efficient large kernel to reduce memory access and startup overhead.

[0107] 5) Code generation: Finally, the code generator compiles the optimized NPU IR to generate the final instruction stream program that can be executed directly on the NPU.

[0108] In this invention, a dedicated intermediate representation design for the NPU is implemented: a structure named NPU IR is defined, which is the core data structure for intermediate representation specifically designed for neural network processors. This structure is used to carry and describe the computational graph of the neural network during the compilation process.

[0109] The NPU IR mainly consists of three parts:

[0110] 1) Operator type enumeration: Defines the core neural network operator types supported by the NPU, such as 2D convolution (CONV_2D), general matrix multiplication (GEMM), pooling, activation function, and element-wise operation (ELEMENTWISE).

[0111] 2) Tensor Descriptor: This is a substructure called the tensor descriptor, used to precisely describe the properties of each input and output tensor in the neural network. It contains:

[0112] ① Dimensional information: An integer vector that defines the shape of the tensor (such as Batch, Channel, Height, Width in the NCHW format).

[0113] ② Data layout: Specifies the format in which data is arranged in memory.

[0114] ③ Preferred memory location: Specifies which memory buffer is more suitable for storing this tensor in order to optimize access efficiency.

[0115] 3) Computational Graph Structure: The core of NPU IR is a list of Operator structure instances, which fully represents the computational graph of the entire neural network. Each Operator structure describes an operator node in detail, which includes:

[0116] ① Operator type: Specifies the specific type of operation it is.

[0117] ② Input / output tensors: All input and output tensors are described by a set of tensor descriptors respectively.

[0118] ③ Fusion attributes: contain parameters related to operator fusion optimization.

[0119] ④ Hardware mapping information: Specifies how the operator should be mapped to a specific hardware unit of the NPU for execution.

[0120] In summary, through this hierarchical design, NPU IR encapsulates information such as the computation graph structure, operator types, tensor data, and hardware mapping strategies of the neural network into a unified data structure, providing complete and machine-processable model information for subsequent optimization and code generation stages.

[0121] The computation mapping strategy of the instruction optimization generator:

[0122] Define a C++ class for hardware-aware mapping optimization. The following is a text description of the code: Hardware-aware mapper class, whose main function is to map computational operations to specific hardware units and optimize them. It contains two core methods:

[0123] 1. Mapping from convolution operations to tensor kernels:

[0124] 1) Receive convolution operators as input parameters;

[0125] 2) First, select the optimal matrix partitioning strategy based on the convolution parameters;

[0126] 3) Then generate the instruction sequence specifically for the tensor kernel;

[0127] 4) Finally, the optimized instruction stream is returned;

[0128] 2. Memory access optimization:

[0129] 1) Optimize memory access for general operators;

[0130] 2) First, perform data reuse analysis to identify memory access patterns;

[0131] 3) Develop a multi-level cache allocation strategy based on the analysis results;

[0132] 4) Return the optimized memory planning scheme.

[0133] This class embodies the idea of ​​hardware-software co-design. By analyzing computational characteristics and hardware features, it enables efficient execution of computational tasks on specific hardware while optimizing the efficiency of memory hierarchy usage.

[0134] See Figure 5 This paper illustrates an embodiment of an OpenCL heterogeneous computing NPU instruction optimization method, applied to the aforementioned OpenCL heterogeneous computing NPU instruction optimization system. The method includes the following steps:

[0135] Step 1: The standard OpenCL application on the host machine initiates a computation request by calling the OpenCL API;

[0136] Step 2: The OpenCL compilation and runtime layer receives the standard OpenCL kernel code, performs in-depth analysis on the standard OpenCL kernel code, identifies the optimizable neural network operators, and converts the neural network operators into a dedicated intermediate representation for NPU design.

[0137] Step 3: Map the dedicated intermediate representation onto the dedicated hardware unit and kernel architecture of the NPU device to generate an instruction stream containing NPU-specific instructions, and send the instruction stream to the NPU device;

[0138] Step 4: The NPU device receives the instruction stream and executes the computing task based on the dedicated hardware unit and kernel architecture.

[0139] In step 2, the OpenCL compilation and runtime layer receives standard OpenCL kernel code, performs in-depth analysis on the standard OpenCL kernel code, identifies optimizable neural network operators, and converts the neural network operators into a dedicated intermediate representation for NPU design, including:

[0140] When standard OpenCL kernel code is received, the OpenCL C source code is compiled into LLVM IR;

[0141] Used to identify computational subgraphs related to neural networks from the LLVM IR through static code analysis and data flow analysis, and to label the computational subgraphs as high-level neural network operators;

[0142] The advanced neural network operators are converted into a dedicated intermediate representation for the NPU design.

[0143] Step 3: Map the dedicated intermediate representation onto the dedicated hardware unit and kernel architecture of the NPU device to generate an instruction stream containing NPU-specific instructions, including:

[0144] The dedicated intermediate representation is mapped onto the dedicated hardware unit and kernel architecture of the NPU device, and the underlying instructions are generated according to the hardware characteristics of the NPU device.

[0145] The underlying instructions are translated into an optimized instruction stream that includes NPU-specific instructions.

[0146] It is understood that the NPU instruction optimization method for OpenCL heterogeneous computing provided by this invention corresponds to the NPU instruction optimization system for OpenCL heterogeneous computing provided in the foregoing embodiments. The relevant technical features of the NPU instruction optimization method for OpenCL heterogeneous computing can be referred to the relevant technical features of the NPU instruction optimization system for OpenCL heterogeneous computing, and will not be repeated here.

[0147] The present invention provides an OpenCL heterogeneous computing NPU instruction optimization system and method, which has the following advantages:

[0148] (1) Performance improvement:

[0149] ① This scheme directly maps high-level operators to dedicated computing units of the NPU (such as matrix cores).

[0150] ② The computational parallelism is greatly improved, and the efficiency of instruction issuance and execution far exceeds that of general scalar / vector instruction mapping.

[0151] ③ It achieved computational performance close to that of the native SDK.

[0152] (2) Memory bandwidth saving:

[0153] ① This solution optimizes the layout and flow of data in the NPU on-chip memory based on operator semantics.

[0154] ② Significantly improved data reuse rate and reduced communication with off-chip DRAM.

[0155] ③ It alleviates the memory bandwidth bottleneck and improves overall energy efficiency.

[0156] (3) Balancing programming convenience and performance:

[0157] ① This solution is compatible with the standard OpenCL API, so developers do not need to learn the NPU's proprietary programming model to develop.

[0158] ② This solution features deep optimizations at the underlying level. Developers can achieve high performance without the need for tedious manual kernel fusion and memory optimization.

[0159] ③ It achieves an excellent balance between development efficiency and operational performance.

[0160] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0161] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0166] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An NPU instruction optimization system for OpenCL heterogeneous computing, characterized in that, This includes the host machine, the OpenCL compilation and runtime layer, and the NPU device; The standard OpenCL application on the host computer initiates a computation request by calling the OpenCL API; The OpenCL compilation and runtime layer receives standard OpenCL kernel code, performs in-depth analysis on the standard OpenCL kernel code, identifies optimizable neural network operators, and converts the neural network operators into a dedicated intermediate representation for NPU design. The dedicated intermediate representation is mapped onto the dedicated hardware unit and kernel architecture of the NPU device to generate an instruction stream containing NPU-specific instructions, and the instruction stream is sent to the NPU device. The NPU device receives the instruction stream and executes computational tasks based on dedicated hardware units and kernel architecture; The OpenCL compilation and runtime layer includes a pattern recognition and kernel fusion engine, an intermediate representation layer, an optimized instruction generator, and an NPU driver adaptation layer. The compiler front-end of the OpenCL compilation and runtime layer is used to compile OpenCL C source code into LLVM IR when standard OpenCL kernel code is received; The pattern recognition and kernel fusion engine is used to identify computational subgraphs related to neural networks from the LLVM IR through static code analysis and data flow analysis, and to mark the computational subgraphs as high-level neural network operators; The intermediate representation layer converts the high-level neural network operators into a dedicated intermediate representation designed for the NPU. The optimized instruction generator is used to map the dedicated intermediate representation onto the dedicated hardware unit and kernel architecture of the NPU device, and generate low-level instructions based on the hardware characteristics of the NPU device. The NPU driver adaptation layer is used to translate the underlying instructions into an optimized instruction stream containing NPU-specific instructions, and then send the optimized instruction stream to the NPU device.

2. The OpenCL heterogeneous computing NPU instruction optimization system according to claim 1, characterized in that, The pattern recognition and kernel fusion engine is used to identify computational subgraphs related to neural networks from the LLVM IR through static code analysis and data flow analysis, and to label the computational subgraphs as high-level neural network operators, including: The pattern recognition and kernel fusion engine identifies target computation patterns from the LLVM IR through static code analysis and data flow analysis, and marks them as high-level neural network operators. The target computation patterns include minimal computation patterns and combined computation patterns in the neural network. The combined computation pattern is formed by the fusion of multiple consecutive minimal computation patterns.

3. The OpenCL heterogeneous computing NPU instruction optimization system according to claim 2, characterized in that, The optimized instruction generator is used to map the dedicated intermediate representation onto the dedicated hardware units and kernel architecture of the NPU device, and to generate low-level instructions based on the hardware characteristics of the NPU device, including: Based on the hardware characteristics of the NPU device, the dedicated intermediate representation is deeply optimized: Mapping advanced neural network operators to dedicated instructions for dedicated hardware units of the NPU device; Based on the data access patterns of advanced neural network operators, a data flow strategy in the multi-level memory of the NPU device is planned to maximize data reuse and minimize off-chip access. This involves fusing adjacent, fusionable high-level neural network operators into a single composite kernel at an intermediate representation level.

4. The OpenCL heterogeneous computing NPU instruction optimization system according to claim 2, characterized in that, Identifying target computation patterns from the LLVM IR and labeling them as high-level neural network operators includes: Each target computation mode is identified from the LLVM IR, and each target computation mode corresponds to a high-level neural network operator. The type of each high-level neural network operator is then labeled.

5. The OpenCL heterogeneous computing NPU instruction optimization system according to claim 4, characterized in that, The optimized instruction generator is used to map the dedicated intermediate representation onto the dedicated hardware units and kernel architecture of the NPU device, and to generate low-level instructions based on the hardware characteristics of the NPU device, including: Map the dedicated intermediate representation of each advanced neural network operator to the dedicated instructions of the corresponding dedicated hardware unit of the NPU device; And based on the data access mode of each advanced neural network operator, the computational data corresponding to each advanced neural network operator is mapped to the memory of the corresponding level.

6. A method for optimizing NPU instructions in OpenCL heterogeneous computing, applied to the OpenCL heterogeneous computing NPU instruction optimization system as described in claim 1, characterized in that, The method includes: Standard OpenCL applications on the host machine initiate computation requests by calling the OpenCL API; The OpenCL compilation and runtime layer receives standard OpenCL kernel code, performs in-depth analysis on the standard OpenCL kernel code, identifies optimizable neural network operators, and converts the neural network operators into a dedicated intermediate representation for NPU design. The dedicated intermediate representation is mapped onto the dedicated hardware unit and kernel architecture of the NPU device to generate an instruction stream containing NPU-specific instructions, and the instruction stream is sent to the NPU device. The NPU device receives the instruction stream and executes computational tasks based on dedicated hardware units and kernel architecture; A deep analysis of the standard OpenCL kernel code was performed to identify optimizable neural network operators, and these operators were converted into a dedicated intermediate representation for NPU design, including: When standard OpenCL kernel code is received, the OpenCL C source code is compiled into LLVM IR; Used to identify computational subgraphs related to neural networks from the LLVM IR through static code analysis and data flow analysis, and to label the computational subgraphs as high-level neural network operators; The advanced neural network operators are converted into a dedicated intermediate representation for NPU design; The dedicated intermediate representation is mapped onto the dedicated hardware units and kernel architecture of the NPU device to generate an instruction stream containing NPU-specific instructions, including: The dedicated intermediate representation is mapped onto the dedicated hardware unit and kernel architecture of the NPU device, and the underlying instructions are generated according to the hardware characteristics of the NPU device. The underlying instructions are translated into an optimized instruction stream that includes NPU-specific instructions.

Images