A neural network accelerator model conversion method and apparatus

Abstract

The application discloses a neural network accelerator model conversion method and device, the method comprises the following steps: obtaining a neural network model to be converted, analyzing a model network structure file to obtain all network layers of the model, reconstructing the network layers, mapping the network layers into operator nodes supported by a neural network accelerator, and finally serializing the converted operator nodes and model weights according to a network topology structure to generate a target file; the device comprises a neural network model construction module, a reconstruction module, a mapping module and a serialization module; the application solves the multi-adaptation difficulty problem of deploying multiple format models on a neural network accelerator device, can efficiently convert the models, and generates a model format suitable for the neural network accelerator.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a neural network accelerator model conversion method and device. BACKGROUND

[0002] The current neural network technology develops very fast, and various machine learning frameworks emerge as the times require. Various machine learning frameworks have their own advantages and disadvantages, so in the academic and industrial circles, various machine frameworks coexist, which leads to different file formats of the models trained by different frameworks. For neural network accelerators and other ASIC (Application Specific Integrated Circuit) devices, various model file formats need to be adapted, which greatly hinders the deployment efficiency of neural network models on ASIC devices.

[0003] The existing model format file must be based on the framework or protocol corresponding to the format to read the file. For example, the model file of TensorFlow uses Protocol Buffers. If you want to quickly read and obtain the network layer information in the model file, you must use the TensorFlow software package, which is very cumbersome on hardware. At the same time, most machine learning frameworks save the model format containing a large amount of information useless for neural network accelerator running models (such as learning rate, batch data size, etc. required for training), which is useless information for neural network accelerators (mainly used for model inference). In summary, the traditional neural network accelerator must rely on the machine learning framework program to use the model file saved by it, and using the framework will have the problem of needing to adapt to different frameworks for the neural network accelerator, which has great limitations. SUMMARY

[0004] To solve the problems of the prior art, the present application generates a model format suitable for a neural network accelerator by analyzing, reconstructing and mapping different formats of model files, thereby achieving the purpose of improving the deployment efficiency of models on a neural network accelerator. The present application adopts the following technical solutions:

[0005] A neural network accelerator model conversion method, comprising the following steps:

[0006] S1, obtaining a neural network model to be converted, including a network structure file and a model weight file; the current common machine learning frameworks include TensorFlow, Pytorch, PaddlePaddle, MXNet, etc., each of which has a unique model file format;

[0007] S2, parse the network structure file, extract the network layer operator and operator attribute, construct the network layer and network topology structure of the neural network model;

[0008] S3, reconstruct the network layer, fuse the network layer, keep only the core operator parameters of part of the network layer as the attribute of the related network layer, replace the original network layer with the fused network layer, and update the network topology structure, thereby simplifying the model structure, realizing the compression of the model, reducing the model inference time, mapping the reconstructed network layer to the operator node supported by the neural network accelerator through the mapping table, and obtaining all the operator nodes corresponding to the network layer; the operator node type supported by the neural network accelerator is limited, part of the network layer (operator with parameters) can be realized by using one or several operators of the neural network accelerator, and some typical operators will be operated on the last dimension of the high-dimensional array by default, and the index of this dimension is usually included in the general model file, therefore, the attribute information of the general network layer is first simplified, only the core operator operation parameters are kept, and part of the network layer can be realized by using the same operation process in hardware implementation, multiple operations will finally be replaced by the same operator supported by the neural network accelerator;

[0009] S4, according to the updated network topology structure, serialize the operator node and its corresponding weight file to generate a target file, and the serialization is to save the file in binary format, which can be conveniently deployed on the neural network accelerator.

[0010] Further, the network layer constructed in S2 is recorded by a data structure dictionary to record the operator type and attribute information of all network layers.

[0011] Further, in S3, the network layer fused with the core operator parameters is simplified through a dictionary format, and the fused core operator parameters are added as a new key-value pair in the dictionary of the network layer.

[0012] Further, in S4, the operator node type and attribute information are saved in the order of the corresponding updated network topology structure, and the weight value of the operator node corresponding to the network layer is serialized and saved.

[0013] Further, the network topology structure constructed in S2 is constructed by recording the input layer index value of each layer of the network layer; the update of the network topology structure in S3 is to update the input layer index value of the fused network layer; and in S4, the input layer index value of the network layer corresponding to the operator node is serialized.

[0014] Further, in the S2, the magnification type abstract operation in the neural network model is replaced with an up-sampling layer and mapped to an up-sampling operator node of the neural network accelerator.

[0015] Further, the fusion in the S3 is fusion of the convolution layer and the batch normalization layer and / or the convolution layer and the LeakyRelu activation layer.

[0016] Further, the mapping in the S3 includes mapping the splicing layer and / or the copy operation to a routing operator node.

[0017] Further, the operator nodes supported by the neural network accelerator in the S3 include a convolution operator node, a pooling operator node, a residual operator node, a routing operator node, and an up-sampling operator node.

[0018] A neural network accelerator model conversion device includes a neural network model construction module, a reconstruction module, a mapping module, and a serialization module.

[0019] The neural network model construction module acquires a neural network model to be converted, including a network structure file and a model weight file, parses the network structure file, extracts network layer operators and operator attributes, constructs a network layer and a network topology structure of the neural network model.

[0020] The reconstruction module fuses network layers, retains only core operator parameters of part of the network layers as attributes of the related network layers, replaces original network layers with the fused network layers, and updates the network topology structure.

[0021] The mapping module maps the reconstructed network layers to operator nodes supported by the neural network accelerator through a mapping table.

[0022] The serialization module serializes the operator nodes and corresponding weight files according to the updated network topology structure to generate a target file.

[0023] The advantages and beneficial effects of the present application are as follows:

[0024] Firstly, the neural network model to be converted is acquired, then the network structure file of the model is parsed, the abstract operation in the network structure is replaced, all network layers and network topologies of the model are constructed, the network layers are fused to reduce the number of network layers, and the network topology is reconstructed, then the network layers are mapped to operator nodes supported by the neural network accelerator according to the mapping table, and finally the model network structure and weights are serialized according to the reconstructed network topology to generate a target file. The method can convert neural network models of different formats, and the model network layers are fused and processed during the model conversion process, realizing the compression of the model and reducing the model inference time. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flowchart of the method of the present application;

[0026] Figure 2 is an example diagram of part of the network structure of TensorFlow training in the present application.

[0027] Figure 3 is an example diagram of the reconstructed network structure in the present application.

[0028] Figure 4 is an example diagram of the convolution layer serialization storage in the present application. DETAILED DESCRIPTION

[0029] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0030] The present application adds an agent between the machine learning framework and the neural network accelerator. For the machine learning framework, the network layer information is read using software for the files saved by several mainstream machine learning frameworks. For the neural network accelerator, the read network layer information can be saved in a data format and supported operator type that is convenient for the neural network accelerator to read the network layer information. Thus, the neural network model parameter files of different formats are converted into a file format suitable for the neural network accelerator (including data format, dictionary, operator node operation sequence, and index to record topology information).

[0031] In view of the problem that the existing model file formats are too many, resulting in the model cannot be efficiently deployed on the neural network accelerator and other devices, by converting the neural network models of different formats, the model file suitable for the neural network accelerator is obtained, and the neural network model can be efficiently deployed on the neural network accelerator and other ASIC devices.

[0032] As Figure 1As shown, a neural network accelerator model conversion method is shown, taking a YOLOv3 model trained by a TensorFlow convolutional neural network as an example to illustrate the key points and advantages of the model conversion of the application. The YOLOv3 model contains 75 convolutional layers, 23 residual layers, 2 routing layers, and 2 upscaling operations. Most of the convolutional layers are followed by batch normalization layers and LeakyRelu activation layers. Due to the large number of layers in YOLOv3, only one subnetwork structure is introduced here to illustrate the model conversion process. The specific implementation steps are as follows:

[0033] Step one: obtain the subnetwork structure and weights of the YOLOv3 model, as shown in Figure 2 The subnetwork structure includes 3 Conv2D convolutional layers, 2 LeakyRelu activation layers, 1 Concatenation concatenation layer, and 1 ResizeBilinear upscaling operation.

[0034] Step two: parse the subnetwork structure, abstract the upscaling operation as an upsampling layer, then construct all other convolutional layers, activation layers, and concatenation layers, record the input layer index of each layer, and use the Identity layer as the end layer, i.e. complete the construction of all network layers and network topology.

[0035] Parsing is to extract operators and corresponding operator attribute information from other format model files. Construction refers to using dictionaries and other data structures to record all network layers. The dictionary contains the operator type and other attribute information of the network layer. The Identity layer, i.e. the identity layer, is used as the end layer, indicating that the current layer is one of the output layers of the network and has no operation process.

[0036] Step three: fuse the obtained convolutional layer followed by the activation layer module to obtain a new convolutional layer, and add an attribute to mark the use of the LeakyRelu activation function, then map all network layers to operator nodes supported by the neural network accelerator, as shown in Figure 3 The fused convolutional layer is mapped to a convolutional layer containing a leaky attribute (i.e. using the LeakyRelu activation function), the upscaling operation after the upscaling layer is replaced by the upscaling layer, and the concatenation layer is mapped to the routing layer. At the same time, due to the change of the network topology structure caused by the fusion of the network layer, the network topology structure needs to be updated.

[0037] Fusion is to replace the convolution layer and the activation layer with a new convolution layer, and add the activation function type and other attributes in the new convolution layer. The neural network accelerator supports a limited number of operator node types, and some network layers (operators with parameters) can be implemented using one or more operators of the neural network accelerator, while some typical operators, such as the routing layer, are operated on the last dimension of the high-dimensional array by default, and the model file usually contains the index of this dimension. Therefore, for general network layers, first simplify the attribute information and only keep the most core operator operation parameters, and some network layers can be implemented using the same operation process in hardware implementation, for example, the splicing layer and the copy operation, which will eventually be replaced by the routing operator supported by the neural network accelerator.

[0038] Mapping can be understood as a conversion of one data structure to another data structure. For example, the data structure in the TensorFlow model file uses the Protocol Buffers format, which is relatively complex and can be simplified using the dictionary format.

[0039] For network topology updates, such as in a convolutional neural network model, the input of a convolutional layer is the activation layer in front of it, and the index of the activation layer is assumed to be 3. Since we have performed fusion operations, the previous activation layer and another convolutional layer (index 2) in front of the activation layer have been fused, so the original activation layer with index 3 no longer exists. At this time, the index of the convolutional layer needs to be changed to 3, and the input layer index is also modified to 2, and other network layers are similar.

[0040] Step four: according to the new network topology structure and all the obtained operator nodes, perform serialization output, that is, save all operator node types and self-attribute information in the node topology order, and also save the input node index of each operator node. In this way, the network structure topology information is preserved. In addition, the weight values of the convolutional layer are also serialized and saved. Thus, the TensorFlow model is converted into a model format target file suitable for the neural network accelerator. Node serialization is to save it as a binary format file, as shown in Figure 4

[0041] ​To sum up, the application designs a scheme for model conversion of a neural network accelerator, analyzes a network structure of a neural network model trained by a common machine learning framework, obtains a complete network layer and network topology structure, fuses the network layer by reconstruction, reduces the model network layer, maps the obtained model network layer, obtains an operator node supported by the neural network accelerator, and finally serializes all the operator nodes and model weights according to the obtained network topology structure to generate a target file. The several steps included in the whole process have good decoupling, can efficiently convert model files trained by different machine learning frameworks, greatly improves the deployment efficiency of different format models on the neural network accelerator, and the good decoupling can also conveniently adapt and extend the model conversion of newly emerging frameworks.

[0042] The above examples are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for converting neural network accelerator models, characterized in that... The method comprises the following steps: S1, obtaining a neural network model to be converted, including a network structure file and a model weight file; S2, parsing the network structure file, extracting network layer operators and operator attributes, and constructing network layers and network topology of the neural network model; the constructed network topology is constructed by recording input layer index values of each layer of the network layer; S3, reconstructing the network layer, fusing the network layer, retaining only core operator parameters of part of the network layer as attributes of the related network layer, replacing the original network layer with the fused network layer, updating the network topology, and mapping the reconstructed network layer to an operator node supported by a neural network accelerator through a mapping table; For the routing layer, by default, operation is performed on the last dimension of a high-dimensional array, and the model weight file contains the index of this dimension. For a general network layer containing this dimension index, only the most core operator operation parameters are retained, and part of the network that uses the same operation process to implement multiple operations on the hardware is replaced by the same operator supported by the neural network accelerator in the end; The update of the network topology is to update the input layer index values of the fused network layer; S4, serializing the operator nodes and their corresponding weight files according to the updated network topology to generate a target file; The serialization of the operator nodes is to serialize the input layer index values of the network layer corresponding to the operator nodes.

2. The method of claim 1, wherein The network layer constructed in S2 is recorded by a data structure dictionary to record the operator types and attribute information of all network layers.

3. The method of claim 2, wherein In S3, the mapping is to simplify the network layer fused with the core operator parameters in a dictionary format, and add the fused core operator parameters as a new key-value pair in the dictionary of the network layer.

4. The method of claim 2, wherein In S4, the operator node types and attribute information are saved in the order of the updated network topology, and the weight values of the network layer corresponding to the operator nodes are serialized and saved.

5. The method of claim 1, wherein In S2, the abstract operation of the magnification type in the neural network model is replaced by an upsampling layer and mapped to an upsampling operator node of the neural network accelerator.

6. The method of claim 1, wherein The fusion in S3 is to fuse the convolution layer and the batch normalization layer, and / or the convolution layer and the activation layer.

7. The method of claim 1, wherein The mapping in S3 includes mapping the concatenation layer and / or the copy operation to a routing operator node.

8. The method of claim 1, wherein The operator nodes supported by the neural network accelerator in S3 include a convolution operator node, a pooling operator node, a residual operator node, a routing operator node, and an upsampling operator node.

9. A neural network accelerator model conversion apparatus employing the neural network accelerator model conversion method of claim 1, comprising: The neural network model construction module, the reconstruction module, the mapping module, and the serialization module are characterized by: The neural network model construction module obtains a neural network model to be converted, including a network structure file and a model weight file, parses the network structure file, extracts network layer operators and operator attributes, and constructs network layers and network topology of the neural network model; The reconstruction module fuses network layers, retains only core operator parameters of part of the network layers, and takes the core operator parameters as attributes of the network layers, replaces the original network layers with the fused network layers, and updates a network topology structure; The mapping module maps the reconstructed network layers to operator nodes supported by a neural network accelerator through a mapping table; The serialization module serializes the operator nodes and corresponding weight files according to the updated network topology structure, and generates a target file.

Images