System and method for compiling convolutional neural network model for embedded device

Abstract

The present invention relates to the technical field of artificial intelligence devices, and provides a system and method for compiling a convolutional neural network model for an embedded device. The system comprises a model computation graph representation unit, a convolutional neural network fixed optimization module, a convolutional neural network automatic optimization module, an automatic optimization process module, a model compilation optimization unit, and a model deployment and reasoning unit. A convolutional neural network operator having the highest computational load is optimized, and only the edge cropping size and the loop unrolling step size of the convolutional neural network operator are optimized, so that the optimization space is reduced and the optimization time is shortened while optimizing the operator to the greatest extent; in addition, the entire process from model training output to embedded device reasoning is established, and rapid optimization and deployment of convolutional neural network models are achieved. By means of the method, a high-performance convolutional neural network model can be rapidly deployed on an embedded device, so that the present invention has high practical value and innovative value.

Description

Convolutional neural network model compiling system and method for embedded device TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence devices, and particularly relates to a convolutional neural network model compiling system and method for embedded device. BACKGROUND

[0002] With the continuous development of deep learning related technologies, neural network models are widely used in many industries and scenarios. Applying deep learning technology to the Internet of Things field, especially to embedded devices with a large number of devices, has huge market demand. At present, because there are many lightweight convolutional neural networks, the inference of deep learning models on embedded devices is mainly based on convolutional neural networks. Due to the limited computing resources of embedded devices, how to improve the inference performance is a key factor in determining whether to land. At present, there are few operator automatic optimization tools for embedded devices.

[0003] SUMMARY

[0004] The present application provides a convolutional neural network model compiling system for embedded device, which reduces the optimization space and shortens the optimization time, and breaks through the whole process from the trained model to the inference of embedded device, realizes the rapid optimization and deployment of convolutional neural network model, can quickly deploy high-performance convolutional neural network model on embedded device, and has high practical value and innovative value.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0006] In a first aspect, the present application provides a convolutional neural network model compiling system for embedded device, comprising: a model computation graph representation unit, a convolutional neural network fixed optimization module, a convolutional neural network automatic optimization module, an automatic optimization process module, a model compiling optimization unit, and a model deployment and inference unit.

[0007] The convolutional neural network automatic optimization module optimizes the convolutional neural network operator with the largest computation amount of model, and only optimizes the cut edge size and loop unrolling step of the convolutional neural network operator.

[0008] Further, the model computation graph representation unit uses a self-defined model inference framework to analyze the neural network model and according to the computation graph representation method of the framework.

[0009] Further, the convolutional neural network fixed optimization module optimizes the memory arrangement of model weights and activation values in the convolution operator implementation, so as to increase the calculation locality; for repeatedly used data, a local variable is used instead of repeated memory access, so as to reduce the memory access times; for the embedded device with SIMD, the support for the vector instruction set is increased.

[0010] Further, the convolutional neural network automatic optimization module implements a slice calculation module in the convolution operator implementation, and the slice size is used as an optimization variable; in each slice, a loop unrolling module is implemented, and the loop unrolling step is used as an optimization variable; the slice number, size and loop unrolling step range constitute a to-be-optimized variable space; in the to-be-optimized variable space, an optimal set of variables is selected; after the set of variables is specified, the performance of the model on the embedded device is actually tested; through a certain optimization strategy, a set of variables with the optimal performance is selected.

[0011] Further, the automatic optimization process module selects an optimal set of variables in the to-be-optimized variable space; after the set of variables is specified, the performance of the model on the embedded device is actually tested; through a certain optimization strategy, a set of variables with the optimal performance is selected.

[0012] Further, the model compilation optimization unit selects a suitable to-be-optimized variable space range and an optimization strategy based on the computation graph representation of the model and the automatic optimization module, and optimizes the convolutional neural network operator of the model.

[0013] Further, the model deployment and inference unit compiles the model based on the optimization result and performs inference on the embedded device.

[0014] In a second aspect, the application provides a convolutional neural network model compilation method for an embedded device, comprising:

[0015] A model computation graph representation is constructed, a neural network model is parsed using a custom model inference framework, and a computation graph representation of the model is constructed according to the computation graph representation mode of the framework;

[0016] A convolutional neural network fixed optimization module is constructed, and the memory arrangement of model weights and activation values is optimized in the convolution operator implementation, so as to increase the calculation locality; in addition, for repeatedly used data, a local variable is used instead of repeated memory access, so as to reduce the memory access times; for the embedded device with SIMD, the support for the vector instruction set is increased;

[0017] An automatic optimization module of a convolutional neural network is constructed, in the implementation of a convolution operator, a slice calculation module is implemented, and the slice size is taken as a variable for optimization; in each slice, a loop unrolling module is implemented, and the loop unrolling step is taken as a variable for optimization; the slice number, size and loop unrolling step range constitute a variable space to be optimized;

[0018] An automatic optimization process module is constructed, that is, in the variable space to be optimized, an optimal optimization configuration is selected; after a group of variables is specified, the performance of the actual test model on the embedded device is tested; through a certain optimization strategy, a group of variables with the optimal performance is selected;

[0019] Model compilation optimization, based on the calculation graph representation of the model and the automatic optimization module, a suitable variable space to be optimized range and an optimization strategy are selected to optimize the model convolutional neural network operator;

[0020] Model deployment and inference, based on the optimization result, the model is compiled and inferred on the embedded device.

[0021] Compared with the prior art, the advantages and positive effects of the present application are that:

[0022] The present application provides a convolutional neural network model compilation system and method for embedded devices, which optimizes the convolutional neural network operator with the largest model calculation amount, and only optimizes the slice size and loop unrolling step of the convolutional neural network operator, thereby optimizing the operator to the maximum extent while reducing the optimization space and shortening the optimization time. In addition, the entire process from the trained model to the inference on the embedded device is connected, realizing the rapid optimization and deployment of the convolutional neural network model. In the above manner, a high-performance convolutional neural network model can be quickly deployed on the embedded device, which has high practical value and innovative value. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0024] Fig. 1 is a flowchart of the convolutional neural network model compilation method for embedded devices of the present application. DETAILED DESCRIPTION

[0025] In order to more clearly illustrate the above-mentioned purposes, features and advantages of the present application, the following will further illustrate the present application with reference to the drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0026] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure aspects of the present application.

[0027] Embodiment 1, the application provides a convolutional neural network model compiling system of embedded equipment, comprising: a model computation graph representation unit, a convolutional neural network fixed optimization module, a convolutional neural network automatic optimization module, an automatic optimization process module, a model compiling optimization unit, a model deployment and reasoning unit.

[0028] The model computation graph representation unit uses a custom model inference framework to parse a neural network model, and computes a computation graph representation according to a computation graph representation manner of the framework, the computation graph representation comprising computation graph data, node data, activation value data, and node and activation value relationship data, wherein the computation graph data comprises computation graph input, output, node quantity, node list, activation value list, and node and activation value relationship list. The node data comprises node operator type, weight data, and configuration data. The activation value data comprises activation value data type, shape, and actual data. The node and activation value relationship data comprises data of direct relationship between nodes and activation values.

[0029] The convolutional neural network fixed optimization module optimizes model weight and activation value memory arrangement manner in convolution operator implementation, so as to increase calculation locality. In addition, for repeatedly used data, a local variable is used to replace repeated memory access, so as to reduce memory access times. For embedded equipment with SIMD, support for a vector instruction set is increased. Memory arrangement manner, local variable replacement of repeated memory access, SIMD instruction support, and DSP operator support are each regarded as a pass. For different hardware, supported pass types are set. Through early configuration and registration, the above optimizations are automatically implemented in the model compiling stage.

[0030] The automatic optimization module of the convolutional neural network realizes a slice calculation module in the implementation of the convolution operator, and the slice size is taken as a variable for optimization; in each slice, a loop unrolling module is realized, and the loop unrolling step is taken as a variable for optimization; the slice number, size, and loop unrolling step range constitute a variable space to be optimized; in the variable space to be optimized, an optimal set of variables is selected; after a set of variables is specified, the performance of the actual test model on the embedded device is tested; through a certain optimization strategy, a set of variables with the optimal performance is selected; the slice calculation and loop unrolling need to optimize the slice number, size, and loop unrolling step according to the specific data situation. First, the slice range and rules need to be preset, including but not limited to average slice and random slice, and the loop unrolling step range and rules are also preset. The specific rules can be determined and optimized according to the actual situation. Second, the evaluation model needs to be specified, and the actual running operator time is used for evaluation. The evaluation model is used to evaluate the pros and cons of the specific rules. Third, the number of search attempts is specified.

[0031] The automatic optimization process module selects an optimal set of variables in the variable space to be optimized; after a set of variables is specified, the performance of the actual test model on the embedded device is tested; through a certain optimization strategy, a set of variables with the optimal performance is selected. The automatic optimization process is to select the optimal optimization configuration in the variable space to be optimized. After a set of variables is specified, the performance of the actual test model on the embedded device is tested. Through a certain optimization strategy, a set of variables with the optimal performance is selected. The optimization strategy supports but is not limited to Grid Search and Random Search.

[0032] The model compilation optimization unit selects a suitable variable space range to be optimized and an optimization strategy based on the computation graph representation of the model and the automatic optimization module, and optimizes the model convolutional neural network operator; according to the optimization configuration, each optimization of the fixed optimization module is executed. In addition, at this time, the automatic optimization configuration of each operator has been determined, and according to the specific configuration, each optimization of the automatic optimization module is executed.

[0033] The model deployment and inference unit includes: based on the above optimization result, the model is compiled and inferred on the embedded device.

[0034] Embodiment 2

[0035] As shown in FIG. 1, the present application also provides a convolutional neural network model compilation method of an embedded device, comprising:

[0036] First, the model computation graph representation is constructed. The neural network model is parsed using a custom model inference framework, and the computation graph representation of the model is constructed according to the computation graph representation method of the framework.

[0037] Secondly, a fixed optimization module of the convolutional neural network is constructed. In the implementation of the convolution operator, the memory arrangement mode of the optimization model weight and the activation value is optimized to increase the calculation locality. In addition, for the repeatedly used data, a local variable is used instead of repeated memory access to reduce the memory access times. For the embedded device with SIMD, the support for the vector instruction set is increased.

[0038] Thirdly, an automatic optimization module of the convolutional neural network is constructed. In the implementation of the convolution operator, a slice calculation module is implemented, and the slice size is used as a variable that can be optimized. Inside each slice, a loop unrolling module is implemented, and the loop unrolling step is used as a variable that can be optimized. The slice number, size and loop unrolling step range constitute the variable space to be optimized.

[0039] Fourthly, an automatic optimization process module is constructed. The automatic optimization process is to select the optimal optimization configuration in the variable space to be optimized. After a set of variables is specified, the performance of the actual test model on the embedded device is tested. Through a certain optimization strategy, the set of variables with the optimal performance is selected. The optimization strategy supports but is not limited to GridSearch and Random Search.

[0040] Fifthly, model compilation optimization. Based on the computation graph representation of the model and the automatic optimization module, the appropriate variable space range to be optimized and the optimization strategy are selected to optimize the convolutional neural network operator of the model.

[0041] Sixthly, model deployment and inference. Based on the optimization results described above, the model is compiled and inferred on the embedded device.

[0042] The above is only the preferred embodiment of the present application, and does not limit the form of the present application. Any skilled person in the art can use the disclosed technical content to make changes or modifications into equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments without departing from the technical solution content of the present application still belongs to the protection scope of the present application.

Claims

1. A convolutional neural network model compilation system for embedded devices, the system comprising: The application relates to a model calculation graph representation unit, a convolutional neural network fixed optimization module, a convolutional neural network automatic optimization module, an automatic optimization process module, a model compilation optimization unit and a model deployment and reasoning unit. The convolutional neural network automatic optimization module optimizes the convolutional neural network operator with the largest calculation amount, and only optimizes the cutting edge size and the loop unrolling step length of the convolutional neural network operator. The model calculation graph representation unit uses a self-defined model reasoning framework to analyze a neural network model and to perform calculation graph representation in the manner of the framework. 2.The system of claim 1, wherein, The convolutional neural network fixed optimization module optimizes the memory arrangement mode of model weights and activation values in the convolutional operator implementation, so as to increase calculation locality; uses a local variable to replace repeated memory access for repeatedly used data, so as to reduce memory access times; and supports a vector instruction set for an embedded device with SIMD.

3. The system of claim 1, wherein, The convolutional neural network automatic optimization module realizes a slice calculation module in the convolutional operator implementation, and the slice size is used as an optimization variable; a loop unrolling module is realized in each slice, and the loop unrolling step length is used as an optimization variable; the slice number, size and loop unrolling step length range constitute a to-be-optimized variable space; an optimal variable group is selected in the to-be-optimized variable space; after the variable group is specified, the performance of the model on an embedded device is actually tested; and an optimal variable group is selected through a certain optimization strategy.

4. The system of claim 1, wherein, The automatic optimization process module selects an optimal variable group in a to-be-optimized variable space; after a variable group is specified, the performance of the model on an embedded device is actually tested; and an optimal variable group is selected through a certain optimization strategy.

5. The system of claim 1, wherein, The model compilation optimization unit selects a suitable to-be-optimized variable space range and an optimization strategy based on the calculation graph representation of the model and the automatic optimization module, and optimizes the convolutional neural network operator of the model.

6. The system of claim 1, wherein, The model deployment and reasoning unit compiles the model based on the optimization result, and reasons on the embedded device.

7. The system of claim 1, wherein, The application relates to a model calculation graph representation unit, a convolutional neural network fixed optimization module, a convolutional neural network automatic optimization module, an automatic optimization process module, a model compilation optimization unit and a model deployment and reasoning unit. 8.A method for compiling a convolutional neural network model of an embedded device, the method comprising: The convolutional neural network fixed optimization module optimizes the memory arrangement mode of model weights and activation values in the convolutional operator implementation, so as to increase calculation locality; uses a local variable to replace repeated memory access for repeatedly used data, so as to reduce memory access times; and supports a vector instruction set for an embedded device with SIMD. The convolutional neural network automatic optimization module realizes a slice calculation module in the convolutional operator implementation, and the slice size is used as an optimization variable; a loop unrolling module is realized in each slice, and the loop unrolling step length is used as an optimization variable; the slice number, size and loop unrolling step length range constitute a to-be-optimized variable space; an optimal variable group is selected in the to-be-optimized variable space; after the variable group is specified, the performance of the model on an embedded device is actually tested; and an optimal variable group is selected through a certain optimization strategy. ​ ​ An automatic optimization process module is constructed, and the automatic optimization process is to select an optimal optimization configuration in a variable space to be optimized. After a group of variables is specified, the performance of an actual test model on an embedded device is tested. Through certain optimization strategies, a group of variables with optimal performance is selected. Model compilation optimization is performed based on the computation graph representation of the model and the automatic optimization module. An appropriate variable space range to be optimized and an optimization strategy are selected to optimize the model convolutional neural network operator. Model deployment and inference are performed based on the optimization results to compile the model and perform inference on the embedded device.

Images