A runtime neural network inference programming system and method

Abstract

The application relates to a runtime neural network inference programming system and method, and the system comprises a neural network algorithm management module, an IO data management module, an NPU kernel management module and an NPU inference machine management module; the neural network algorithm management module, the IO data management module and the NPU kernel management module are connected with the NPU inference machine management module respectively, and the NPU inference machine management module is connected with the IO data management module. According to the task demand and user configuration, the application can take the inference machine as a task unit core, bind suitable neural network algorithms, inference cores and input data, realize flexible scheduling and execution of the inference task of the multi-core neural network processor, and improve the operation efficiency of the multi-core neural network processor.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence, and particularly relates to a runtime neural network inference programming system and method. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, various complex application scenarios have put forward higher and higher requirements for the computing power of neural network processors, and single-core neural network processors have been difficult to meet the existing application requirements, so multi-core neural network processors have emerged. The multi-core neural network processor greatly improves the overall computing power by using multiple neural network cores in parallel inference. However, in the existing technology, there is a lack of effective runtime inference programming method for neural network processors with multiple cores, and it is difficult to fully exert the performance of multi-core neural network processors. SUMMARY

[0003] To solve the technical problems in the background art, the present application provides a runtime neural network inference programming system and method, which takes an inference machine as a task unit core, binds appropriate neural network algorithms, inference cores and input data, and realizes flexible scheduling and execution of multi-core neural network processor inference tasks through management of the inference machine, thereby improving the running efficiency of the multi-core neural network processor.

[0004] The technical solution of the present application is: the present application is a runtime neural network inference programming system, which is characterized in that: the programming system comprises a neural network algorithm management module, an IO data management module, an NPU core management module and an NPU inference machine management module; the neural network algorithm management module, the IO data management module and the NPU core management module are connected with the NPU inference machine management module, and the NPU inference machine management module is connected with the IO data management module.

[0005] Further, the neural network algorithm management module can realize loading and deleting of multiple neural network algorithms, and can select the required neural network algorithm for the inference machine in the NPU inference machine management module and bind it.

[0006] Further, the IO data management module can realize loading and deleting of multiple input inference data, and can bind the data required for inference with the inference machine in the NPU inference machine management module according to the inference task requirements, and output the obtained results after inference is completed.

[0007] Further, the NPU core management module can bind the NPU inference core required for inference for the inference machine according to the current inference core working state and the requirements of the inference machine in the NPU inference machine management module.

[0008] Further, the NPU inference machine management module can realize the creation, management and deletion functions of the inference machine, bind the neural network algorithm selected from the neural network algorithm management module, the inference data selected from the IO data management module, and the inference core selected from the NPU core management module after the creation of the inference machine, and send the inference result to the IO data management module for output after the inference is completed.

[0009] A method for implementing the above-mentioned runtime neural network inference programming system, wherein the method comprises the following steps:

[0010] 1) The NPU loads the neural network algorithm through the neural network algorithm management module and loads the inference data through the IO data management module;

[0011] 2) The NPU creates an inference machine through the NPU inference machine management module;

[0012] 3) The NPU inference machine management module binds the algorithm from the neural network algorithm management module, binds the inference data from the IO data management module, and binds the inference core from the NPU core management module for the inference machine;

[0013] 4) The NPU executes the inference task by taking the inference machine as a basic task unit;

[0014] 5) After the inference is completed, the NPU outputs the inference result through the IO data management module.

[0015] The advantages of the present application are that the runtime neural network inference programming system and method provided by the present application take the inference machine as a task unit core, bind appropriate neural network algorithms, inference cores and input data, realize the flexible scheduling and execution of the inference task of the multi-core neural network processor through the management of the inference machine, improve the running efficiency of the multi-core neural network processor, and have strong practical significance. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The figure is a system module diagram of the present application;

[0017] Figure 2 The figure is a method flowchart of the present application.

[0018] The reference signs are as follows:

[0019] 1, NPU inference machine management module; 2, neural network algorithm management module; 3, IO data management module; 4, NPU core management module. DETAILED DESCRIPTION

[0020] The technical solutions of the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0021] Referring to Figure 1 The structure of the embodiment of the runtime neural network inference programming system of the application comprises a neural network algorithm management module 1, an IO data management module 2, an NPU kernel management module 3 and an NPU inference machine management module 4; the neural network algorithm management module 1, the IO data management module 2 and the NPU kernel management module 3 are connected with the NPU inference machine management module 4 respectively, and the NPU inference machine management module 4 is connected with the IO data management module 2. Among them:

[0022] The neural network algorithm management module 1 can realize loading and deleting of multiple neural network algorithms, and can bind the required neural network algorithm for the inference machine in the NPU inference machine management module 4 according to the requirement of the inference task.

[0023] The IO data management module 2 can realize loading and deleting of multiple input inference data, and can bind the data required for inference with the inference machine in the NPU inference machine management module 4 according to the requirement of the inference task, and output the obtained result after the inference is completed.

[0024] The NPU kernel management module 3 can bind the NPU inference core required for the inference machine to execute inference according to the current inference core working state and the requirement of the inference machine in the NPU inference machine management module 4.

[0025] The NPU inference machine management module 4 can realize the functions of creating, managing and deleting the inference machine, and after the inference machine is created, the neural network algorithm selected from the neural network algorithm management module 1, the inference data selected from the IO data management module 2 and the inference core selected from the NPU kernel management module 3 are bound, and after the inference is completed, the inference result is sent to the IO data management module 2 for output.

[0026] Referring to Figure 2 The application further provides a method for realizing the runtime neural network inference programming system, which comprises the following steps:

[0027] 1) The NPU loads the neural network algorithm through the neural network algorithm management module 1 and loads the inference data through the IO data management module 2;

[0028] 2) The NPU creates the inference machine through the NPU inference machine management module 4;

[0029] 3) The NPU binds the algorithm for the inference machine from the neural network algorithm management module 1, binds the inference data from the IO data management module 2 and binds the inference core from the NPU kernel management module 3 through the NPU inference machine management module 4;

[0030] 4) The NPU executes the inference task by taking the inference machine as a basic task unit;

[0031] 5) After the reasoning is completed, the NPU outputs the reasoning result through the IO data management module 2.

[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A runtime neural network inference programming system, characterized in that: The programming system includes a neural network algorithm management module, an I / O data management module, an NPU kernel management module, and an NPU inference engine management module. These modules are connected to the NPU inference engine management module. The neural network algorithm management module can load and delete multiple neural network algorithms and, based on the inference task requirements, select and bind the necessary neural network algorithm to the inference engine in the NPU inference engine management module. The I / O data management module can load and delete multiple input inference data and, based on the inference task requirements, associate the data to be inferred with the NPU inference engine. The inference engine in the management module is bound, and the result is output after inference is completed. The NPU kernel management module can bind the NPU inference engine to the inference engine according to the current working status of the inference engine and the needs of the inference engine in the NPU inference engine management module. The NPU inference engine management module can realize the creation, management and deletion functions of inference engines. After the inference engine is created, it binds the neural network algorithm selected from the neural network algorithm management module, the inference data selected from the IO data management module, and the inference engine selected from the NPU kernel management module. After the inference is completed, the inference result is sent to the IO data management module for output.

2. A method for implementing the runtime neural network inference programming system of claim 1, characterized in that: The method includes the following steps: 1) The NPU loads neural network algorithms through the neural network algorithm management module and inference data through the IO data management module; 2) The NPU creates the inference engine through the NPU inference engine management module; 3) The NPU binds algorithms to the inference engine from the neural network algorithm management module, inference data from the IO data management module, and inference kernels from the NPU kernel management module through the NPU inference engine management module; 4) The NPU performs inference tasks using the inference engine as the basic task unit; 5) After inference is completed, the NPU outputs the inference results through the IO data management module.

Images