Model conversion device

Abstract

This model conversion device includes: a model input unit for receiving layer information representing a layer operator used in each layer of an input model that is a neural network; a removal unit whereby continuous layer operators that can be processed collectively by a processing engine and include layer operators not included in the received layer information are removed from pattern table information in which the continuous layer operators are stored; and a conversion unit that converts the continuous layer operators included in the layer information into a batch processing operator in the processing engine using the pattern table information after removal has been performed thereon.

Description

Model conversion device

[0001] The disclosed technology relates to a model transformation device.

[0002] AI (Artificial Intelligence) inference processing such as object detection is executed on various processing engines such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), and a dedicated ASIC (Application Specific Integrated Circuit). These AI inference processes are primarily performed by learning in an environment using a GPU to create a model, which is then converted into a model format suitable for operation on each platform during inference processing. In recent years, advances in hardware technology have made it possible to efficiently perform AI processing using heterogeneous processing hardware equipped with multiple processing engines. Each processing engine is connected via external or internal memory and executes processing instructed by a control unit. When an inference model is executed on such processing hardware, it is possible to determine in advance which processing engine will execute each layer in the model based on specifications such as layer information that each processing engine can process. In this way, allocating a certain inference model to a processing engine in a manner suitable for the target hardware, and performing quantization, model weight reduction, and optimization is generally called a DNN (Deep Neural Network) compiler. When using multiple processing engines, the challenge is how to allocate the processing of each layer of the inference model (CNN (Convolutional Neural Network) model).

[0003] The open-source DNN compiler TVM (Non-Patent Document 1) provides a mechanism called BYOC (Bring Your Own Codegen) to utilize multiple processing engines within inference processing hardware and to easily support new processing engines provided by users. TVM's BYOC provides a mechanism for determining which layers in a CNN model are assigned to processing engines. The term "layer" here refers to an operator that represents layer processing, such as "Conv2D," a two-dimensional convolution operation, or "Relu," an activation function (hereinafter, an operator that represents layer processing will be referred to as a "layer operator").

[0004] Depending on the processing engine, it may be possible to combine multiple consecutive layers and treat them as a single layer. Combining (integrating) multiple consecutive layers is also called Layer Fusion. For example, consecutive layer operators such as Conv2D+Relu and Conv2D+Add+Relu may be processed together, or consecutive layer operators such as Conv2D+Conv2D may be processed together. In this case, the processing engine can integrate and process multiple layers, allowing for more efficient processing than processing each layer individually. This is because, when processing with Layer Fusion, temporary data of intermediate calculation results is stored in a generally small but high-speed memory such as an SRAM or register closer to the processing engine, thereby reducing access to external RAM (DRAM). Layer Fusion enables more efficient calculations when as many layers as possible can be integrated, but the extent to which layers can be integrated and processed varies depending on the processing engine. Therefore, when performing model conversion compatible with Layer Fusion, each processing engine registers the corresponding layer operator sequence in a pattern table in advance, and performs pattern matching sequentially to see if a layer operator sequence corresponding to that pattern appears in the model to be converted. If a pattern match is found, that pattern is adopted as the operator to be processed collectively, and this process is repeated. When pattern matching is performed, matching is often performed in descending order of the number of layers to be integrated in order to maximize the effect of Layer Fusion.

[0005] "Apache TVM", internet search <URL: https: / / tvm.apache.org / > "How to Bring Your Own Codegen to TVM", internet search<URL: https: / / tvm.apache.org / 2020 / 07 / 15 / how-to-bring-your-own-codegen-to-tvm>

[0006] In conventional methods, when converting a model, each processing engine performs pattern matching using a greedy algorithm to determine whether predefined Layer Fusion patterns can be applied to the model to be converted. The greater the number of Layer Fusion patterns and the greater the number of layers in the model to be converted, the more frequently pattern matching is performed, which poses a problem of taking an extremely long time for pattern matching.

[0007] Furthermore, in a model conversion device, Layer Fusion patterns are defined solely based on the conditions that the processing engine can handle, so there is a problem in that matching is performed on patterns that include layer operators that are not necessarily included in the model to be converted, which requires a long processing time.

[0008] Furthermore, even if there is no effect on the matching determination in Layer Fusion, the attribute information of the layer operator itself (kernel size, stride amount, etc. in the case of Conv2D) and differences in combinations of layer operators are distinguished and registered as patterns in the pattern table, which ultimately increases the number of pattern matches and increases processing time. For example, even if there is a limit to the number of Conv2Ds allowed as a Layer Fusion condition, Conv2D+Relu and Conv2D+Add+Relu are registered as separate patterns. In addition, activation functions are often used in combination with Conv2D, but in reality there are multiple types such as "Relu", "leaky_relu", "mish", etc. For example, when up to n Conv2Ds can be Layer Fusioned and there are m activation functions, the combination of Conv+activation function (Activation) is m. ｎ Depending on the conditions of Layer Fusion, there may be many combinations of other layer operators, but performing pattern matching for all of these increases the time required for model conversion processing.

[0009] The disclosed technology has been made in consideration of the above points, and aims to provide a model conversion device that can convert consecutive layer operators in the layer information of an input model that can be processed collectively by a processing engine into batch processing operators while suppressing increases in processing time.

[0010] A first aspect of the present disclosure is a model conversion device that includes: a model input unit that receives layer information representing layer operators used in each layer of an input model that is a neural network; a removal unit that removes consecutive layer operators, including layer operators not included in the received layer information, from pattern table information that stores consecutive layer operators that can be processed collectively by a processing engine; and a conversion unit that uses the pattern table information after removal to convert the consecutive layer operators included in the layer information into batch processing operators in the processing engine.

[0011] According to the above aspect, the model conversion device of the present disclosure can convert consecutive layer operators that can be processed collectively by a processing engine, among the layer information of an input model, into batch processing operators, while suppressing an increase in processing time.

[0012] 1 is a block diagram showing an example of a hardware configuration of a model conversion device. FIG. 2 is a diagram for explaining the functions of the model conversion device. FIG. 3 is a diagram showing an example of pattern table information. FIG. 4 is a diagram showing an example of pattern table information. FIG. 5 is a diagram showing an example of layer information. FIG. 6 is a diagram showing an example of pattern table information. FIG. 7 is a diagram showing an example of pattern table information after removal. FIG. 8 is a diagram showing an example of an activation function that can be subjected to LUT processing and an activation function that cannot be subjected to LUT processing. FIG. 9 is a diagram showing an example of pattern table information before and after replacement. FIG. 10 is a diagram showing an example of pattern table information before and after replacement. FIG. 11 is a diagram showing an example of a model graph before replacement. FIG. 12 is a diagram showing an example of a model graph after replacement. FIG. 13 is a diagram showing an example of subgraphing of a model graph. FIG. 14 is a diagram showing an example of restoring a replaced arithmetic operator to the original layer operator. FIG. 15 is a diagram showing an example of a table showing the correspondence between layer operators before replacement and arithmetic operators after replacement. FIG. 16 is a flowchart showing the flow of model conversion processing. FIG. 17 is a diagram showing an example of replacing a pattern consisting of a plurality of consecutive layer operators or a repetition of a plurality of consecutive layer operators with the arithmetic operator.

[0013] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0014] [First Embodiment] [Configuration of a Model Conversion Device According to the First Embodiment] A model conversion device according to this embodiment converts consecutive layer operators that can be processed collectively by a processing engine, among layer operators used in each layer of an input model that is a convolutional neural network, which is an example of a neural network, into batch processing operators that can be processed collectively by the processing engine, and generates code for the processing engine. The model conversion device 10 according to this embodiment will be described below.

[0015] First, the hardware configuration of a model transformation device 10 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the hardware configuration of the model transformation device 10. As shown in Fig. 1, the model transformation device 10 includes processing engines 50A and 50B, a CPU 91, a ROM (Read Only Memory) 92, a RAM (Random Access Memory) 93, storage 94, an input unit 95, a display unit 96, and a communication I / F (Interface) 97. Each component is connected to each other via a bus 99 so that they can communicate with each other.

[0016] The CPU 91 is a central processing unit that executes various programs and controls each part. That is, the CPU 91 reads programs from the ROM 92 or the storage 94 and executes the programs using the RAM 93 as a work area. The CPU 91 controls each of the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 92 or the storage 94.

[0017] The ROM 92 stores various data and programs. The RAM 93 temporarily stores programs or data as a working area. The storage 94 is configured by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including the operating system and various data.

[0018] The input unit 95 includes, for example, a pointing device such as a mouse, a keyboard, etc., and is used to input various types of information. The display unit 96 is, for example, a liquid crystal display, and displays various types of information. The display unit 96 may also function as the input unit 95 by employing a touch panel system.

[0019] The communication I / F 97 is an interface for communicating with other devices, for example, using a wired communication standard such as Ethernet (registered trademark) or FDDI (Fiber Distributed Data Interface), or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

[0020] The processing engines 50A and 50B perform CNN processing such as convolution processing and activation function processing using code transferred from the ROM 92 or storage 94. The processing engines 50A and 50B according to this embodiment are configured as hardware designed specifically for CNN. The processing engines 50A and 50B are configured as dedicated electrical circuits, such as a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing, such as a CPU, GPU, or FPGA, or an ASIC, which is a processor having a circuit configuration designed specifically for executing specific processing. The processing engines 50A and 50B may each have a different configuration. Note that the number of processing engines designed specifically for CNN may be one or three or more.

[0021] As shown in FIG. 2 , the model transformation device 10 functionally comprises a model input unit 60, a pattern table storage unit 62A, a pattern table storage unit 62B, a graph transformation unit 64, a removal unit 66, a first replacement unit 68, a second replacement unit 70, a transformation unit 72, a graph generation unit 74, a restoration unit 76, and a code generation unit 78.

[0022] The model input unit 60 receives layer information input by the input unit 95, which indicates layer operators used in each layer of the input model, which is a convolutional neural network.

[0023] Here, the layer operator represented by the layer information refers to the layer operator that constitutes the model (for example, "Conv2D" which is a two-dimensional convolution operation, or "Relu" which indicates an activation function). Although the operator names differ depending on the model format, the intended operation basically refers to the same thing. For example, what is represented as layers.Conv2D in the keras framework is represented as the Conv operator in onnx.

[0024] Layer information is available in various formats, such as ONNX (Open Neural Network Exchange), darknet, TensorFlow, and keras, and is composed of a series of information representing layer operators defined in each format. The layer information also lists each layer in order and includes information indicating the input / output connection relationships between layers and parameters to be used (such as filter coefficient information for convolution operations).

[0025] The pattern table storage unit 62A stores pattern table information that stores layer operators that can be processed by the processing engine 50A and consecutive layer operators that can be processed collectively by the processing engine 50A.

[0026] The pattern table storage unit 62B stores pattern table information that stores layer operators that can be processed by the processing engine 50B and consecutive layer operators that can be processed collectively by the processing engine 50B.

[0027] The pattern table information is defined based on the specifications of each processing engine 50A, 50B, and registers information about layer operators that can be processed by each processing engine 50A, 50B. Each processing engine 50A, 50B may register a single layer operator, or, if the processing engine 50A, 50B can process multiple consecutive layer operators together, a series of layer operators may be registered. Since pattern table information is prepared for each processing engine 50A, 50B, there are as many pattern table information as there are processing engines, and the pattern table information registered by each processing engine 50A, 50B is different. Examples of pattern table information are shown in Figures 3A and 3B. It is also possible to register pattern table information that is differentiated based on attribute values ​​set for the layer operators (such as kernel size and stride amount for "Conv2D").

[0028] 3A shows an example in which information on layer operators that can be processed by the processing engine 50A and information on consecutive layer operators that can be processed collectively by the processing engine 50A are stored in the pattern table information. Also, FIG. 3A shows an example in which the types and number of layers that can be integrated are restricted by the processing engine specifications. When a processing engine integrates and processes multiple layers, it cannot integrate layers infinitely due to issues such as internal memory capacity.

[0029] Many CNN inference models have a layer structure in which two-dimensional convolution operations appear consecutively: Conv2D → Conv2D → ... → Conv2D. Some processing engines are equipped with a mechanism for efficiently processing these consecutive Conv2D operations (two-dimensional convolution operations) by collectively handling them. For example, data transfer time can be reduced by storing the results of intermediate layers in the hardware's built-in memory and reading and writing data to external memory only at the beginning and end. Figure 3A shows an example in which the upper limit n of the number of layers that can be integrated is 4, and up to n layers of Conv2D can be processed collectively.

[0030] FIG. 3B shows an example in which information on layer operators that can be processed by the processing engine 50B and information on consecutive layer operators that can be collectively processed by the processing engine 50B are stored in the pattern table information.

[0031] The graph conversion unit 64 analyzes the layer information of the input model and converts it into a model graph. Conversion to a model graph is typically performed within a DNN compiler, where each layer of a convolutional neural network serves as a node, and a graph is constructed based on the layer connection information. For example, in the DNN compiler TVM, this is expressed as a Relay of the graph IR (Intermediate Representation). When a model is generated for each processing engine using a DNN compiler or the like, optimization (e.g., deletion of unnecessary layers, quantization, layer fusion, etc.) is performed based on this graph information. When a layer operator is assigned to a processing engine other than the default (e.g., a CPU), that layer portion is treated as a subgraph and converted into code for the assigned processing engine. The processing engine to which each layer is assigned is determined by pattern matching with the model graph of the input model using pattern table information prepared for each processing engine. If multiple consecutive layer operators, rather than just individual layers, can be executed on the processing engine side, the consecutive layer operators are registered in the pattern table information as matching patterns. This allows successive layer operators in the model graph of the input model to be subgraphed together.

[0032] The removal unit 66 removes layer operators that are not included in the layer information of the input model from the pattern table information for the processing engines 50A and 50B, and removes consecutive layer operators that include layer operators that are not included in the layer information of the input model, thereby making it possible to exclude unnecessary patterns in advance during pattern matching.

[0033] For example, assume that the layer information of the input model includes "Conv2D," "Relu," "leaky_relu," "Prelu," and "Sigmoid" as shown in Fig. 4A. In this case, nothing is removed from the pattern table information of the processing engine 50A as shown in Fig. 4B, but "TransposedConv2D," "BatchNormalization," and "Resize" are removed from the pattern table information of the processing engine 50A as shown in Fig. 4C.

[0034] The first replacing unit 68 replaces, in the pattern table information after removal, multiple types of layer operators that can be processed by the same arithmetic operator in the processing engines 50A and 50B with the arithmetic operator.

[0035] Specifically, if the pattern table information contains layer operators that allow different layer operators to be treated as similar layer processing in the processing engines 50A and 50B, these are replaced with one arithmetic operator that performs the layer processing.

[0036] For example, consider a case where a processing engine in the target hardware is capable of processing some activation functions in a unified manner using a look-up table (LUT). In the case of such a processing engine, for example, three functions such as Conv2D+Relu, Conv2D+leaky_relu, and Conv2D+prelu are processed as Conv2D+Activation (LUT) by this processing engine. Differences in activation functions are expressed by information set in the LUT table. Therefore, activation functions that can be processed by the LUT (e.g., leaky_relu, prelu, relu) are replaced with an arithmetic operator called "Activation (LUT)," and activation functions that cannot be processed by the LUT (e.g., softmax, mish, HardSwish) are replaced with an arithmetic operator called "Activation (non-LUT)" (see Figures 5A and 5B).

[0037] In this way, the Conv2D-related layer operators used in combination with various activation functions are narrowed down to two: Conv2D+Activation (LUT) and Conv2D+Activation (non-LUT) (see FIG. 5C).

[0038] In the above example, various types of activation functions used in combination with Conv2D layer operators are replaced with arithmetic operators, but the present invention is not limited to this. For example, as shown below, a case will be described in which layer operators (Padding(1,1), Padding(2,2)) for padding (filling with zeros) according to the kernel size and stride amount when performing a Conv2D convolution operation are consecutively present in the model graph of the input model. In this case, if the padding amount can be automatically calculated according to the kernel size and stride amount on the processing engine, it can be replaced with the replaced expression "Padding."

[0039] (Before replacement) Padding (1, 1) + Conv2D Padding (2, 2) + Conv2D

[0040] (After replacement) ・Padding + Conv2D

[0041] Here, Padding(1,1) indicates an operator that pads one pixel on the top, bottom, left, and right of the input feature map. Padding(2,2) indicates an operator that pads two pixels on the top, bottom, left, and right of the input feature map. In the replaced Padding, the amount of padding can be automatically calculated from the attribute information of Conv2D.

[0042] By this replacement, layer operators that can be handled uniformly for internal processing even if they are different for the target processing engine are integrated into a single arithmetic operator, thereby reducing the number of patterns in the pattern table information. Figure 5B shows an example in which the number of processable layer operators in the pattern table information of processing engine 50A has been reduced from six to four. Figure 5C shows an example in which the number of batch processing operators that are collectively processed by processing engine 50A has been reduced from five to three.

[0043] The second substitution unit 70 similarly substitutes the layer operators included in the layer information with arithmetic operators in accordance with the results of the substitution performed by the first substitution unit 68. The second substitution unit 70 also creates information indicating the correspondence between the layer operators before substitution and the arithmetic operators after substitution.

[0044] Specifically, the second replacement unit 70 similarly replaces layer operators with arithmetic operators on the model graph side of the input model in accordance with the results of replacement by the first replacement unit 68 (see Figures 6A and 6B).

[0045] 6A and 6B show an example in which the layer operator "Relu" at node number "2" in the model graph, the layer operator "leaky_relu" at node number "4", and the layer operator "Prelu" at node number "6" are each replaced with the arithmetic operator "LUT". Also shown is an example in which the layer operator "Sigmoid" at node number "8" in the model graph is replaced with the arithmetic operator "non-LUT".

[0046] At this time, to enable the original layer operator to be restored later, the layer operator information before and after the replacement is linked using a unique ID (e.g., subgraph name) determined by the node number, node name, or attribute value of the node. Alternatively, a table showing the correspondence between the layer operators before replacement and the arithmetic operators after replacement is created (see FIG. 7). The unique ID and the table showing the correspondence are examples of information showing the correspondence.

[0047] The conversion unit 72 converts the consecutive layer operators included in the layer information into batch processing operators using the pattern table information after the removal.

[0048] Specifically, the conversion unit 72 uses the post-removal pattern table information replaced by the first replacement unit 68 to convert consecutive layer operators contained in the post-replacement layer information replaced by the second replacement unit 70 into batch processing operators (see Figure 6C).

[0049] For example, pattern matching is performed based on the pattern table information after replacement by the first replacement unit 68 and the model graph of the input model, and conversion to a batch processing operator is performed. If consecutive layer operators that can be processed collectively and are registered in the pattern table information appear in the model graph of the input model, the corresponding consecutive layer operators are converted into a subgraph. As described above, since the removal unit 66 and the first replacement unit 68 perform each process on the pattern table information, it is expected that the number of attempts to perform pattern matching will be reduced compared to the conventional method. Note that pattern matching is performed using pattern table information defined for each of the processing engines 50A and 50B.

[0050] For example, in the case of the activation function mentioned above, if there are m types of activation functions following Conv2D and the Layer Fusion condition is that Conv2D can be integrated up to n layers, the number of trials for pattern matching this condition against a certain part of the input model is m. ｎ If m = 3 and n = 4, then 3 ４ = 81 ways. If the activation function is expressed as two types, Activation (LUT) and Activation (non-LUT), the number of pattern matchings is 2 ４ = 16 possibilities.

[0051] 6C shows an example in which the successive layer operators “Conv2D” and “Relu” are converted into the batch processing operator “Conv2D+Activation (LUT)”, the successive layer operators “Conv2D” and “leaky_relu” are converted into the batch processing operator “Conv2D+Activation (LUT)”, and the successive layer operators “Conv2D” and “Prelu” are converted into the batch processing operator “Conv2D+Activation (LUT)”. Also shown is an example in which the successive layer operators “Conv2D” and “Sigmoid” are converted into the batch processing operator “Conv2D+Activation (non-LUT)”.

[0052] The graph generation unit 74 generates a model graph after conversion by the conversion unit 72, and also creates subgraphs of the layer operators assigned to each of the processing engines 50A and 50B based on the pattern table information (see FIG. 6C).

[0053] 6C shows an example in which node number "1" of the converted model graph is a subgraph "Conv2D_LUT_2_4_6" and includes three batch processing operators "Conv2D+Activation(LUT)". Also, an example in which node number "2" of the converted model graph is a subgraph "Conv2D_nonLUT_8" and includes a batch processing operator "Conv2D+Activation(non-LUT)".

[0054] The restoration unit 76 restores the calculation operators included in the subgraphed model graph to their original layer operators using information representing the correspondence between the layer operators before replacement and the calculation operators after replacement, for the results converted by the conversion unit 72.

[0055] Here, pattern matching is performed using the arithmetic operators, and after the pattern matching is completed, the replaced arithmetic operators are restored to the original layer operators. This allows the subsequent code generation unit 78 to generate processing code for the processing engine in question for performing processing corresponding to each of the original layer operators. For example, during pattern matching in the conversion unit 72, {(1) Padding + Conv2D + BN + Activation, (2) Conv2D + BN + Activation, (3) Conv2D + BN, (4) Conv2D + Activation} are treated without distinction, but when generating subsequent processing code, processing corresponding to each of Padding, Conv2D, BN, and Activation is required, so the original configuration must be restored.

[0056] For example, as shown in FIG. 6D, the replaced arithmetic operators (e.g., "LUT" and "non-LUT") are restored to their original layer operators. The restoration is performed based on the node names, node number information, and tables of the optimized graph (for each subgraph). This allows the layer operators originally used in the input model to be obtained.

[0057] 6D shows an example in which three batch processing operators “Conv2D+Activation (LUT)” included in the subgraph “Conv2D_LUT_2_4_6” are restored to the batch processing operators “Conv2D+Relu,” “Conv2D+leaky_relu,” and “Conv2D+Prelu,” respectively. Also, an example in which the batch processing operator “Conv2D+Activation (non-LUT)” included in the subgraph “Conv2D_nonLUT_8” is restored to the batch processing operator “Conv2D+Sigmoid.”

[0058] The code generator 78 generates processing codes for the processing engines 50A and 50B based on the restored model graph, and obtains the final conversion model for the accelerator. In generating the processing codes, for example, if the activation functions can be processed using an LUT, the table information of the LUT is set to values ​​for each activation function.

[0059] [Operation of the Model Transformation Device According to the First Embodiment] Next, the operation of the model transformation device 10 according to this embodiment will be described.

[0060] 8 is a flowchart showing the flow of model conversion processing by the model conversion device 10. The model conversion processing is performed by the CPU 91 reading a model conversion program from the ROM 92 or storage 94, expanding it into the RAM 93, and executing it. Furthermore, the pattern table storage unit 62A of the model conversion device 10 stores pattern table information related to the processing engine 50A, and the pattern table storage unit 62B stores pattern table information related to the processing engine 50B. Furthermore, it is assumed that layer information indicating layer operators used in each layer of the input model, which is a convolutional neural network, has been input to the model conversion device 10.

[0061] In step S100, the CPU 91, functioning as the model input unit 60, receives layer information input by the input unit 95, which indicates layer operators used in each layer of the input model, which is a convolutional neural network.

[0062] In step S102, the CPU 91 functions as the graph conversion unit 64 to analyze layer information of the input model and convert it into a model graph.

[0063] In step S104, the CPU 91, as the removal unit 66, deletes layer operators that are not included in the layer information of the input model from the pattern table information for the processing engines 50A and 50B, and also removes consecutive layer operators that include layer operators that are not included in the layer information of the input model.

[0064] In step S106, the CPU 91 functions as the first replacing section 68 to replace, in the post-removal pattern table information, multiple types of layer operators that can be processed by the same arithmetic operator in the processing engines 50A and 50B, with that arithmetic operator.

[0065] In step S108, the CPU 91, as the second substitution unit 70, similarly substitutes the layer operators included in the layer information with arithmetic operators in accordance with the results of the substitution performed by the first substitution unit 68. The second substitution unit 70 also creates information indicating the correspondence between the layer operators before substitution and the arithmetic operators after substitution.

[0066] In step S110, the CPU 91 functions as the conversion unit 72 and converts the consecutive layer operators included in the layer information into batch processing operators using the pattern table information after the removal.

[0067] In step S112, the CPU 91 functions as the graph generating unit 74 to generate a model graph after conversion by the converting unit 72, and also creates subgraphs of the layer operators assigned to each of the processing engines 50A and 50B based on the pattern table information.

[0068] In step S114, the CPU 91, as the restoration unit 76, restores the calculation operator to the original layer operator using information representing the correspondence between the layer operator before replacement and the calculation operator after replacement, for the result of conversion by the conversion unit 72.

[0069] In step S116, the CPU 91 functions as the code generation unit 78 to generate processing codes for the processing engines 50A and 50B, obtain a final conversion model for the accelerator, and then terminate the model conversion process.

[0070] As described above, the model conversion device 10 of this embodiment removes consecutive layer operators, including layer operators not included in the layer information of the received input model, from pattern table information storing consecutive layer operators that can be collectively processed by the processing engine, and converts the consecutive layer operators included in the layer information into batch processing operators in the processing engine using the pattern table information after removal. This makes it possible to convert consecutive layer operators that can be collectively processed by the processing engine in the layer information of the input model into batch processing operators while suppressing an increase in processing time. It also makes it possible to reduce the pattern matching processing time required to select which processing engine will process each layer in the CNN inference model.

[0071] Furthermore, by integrating layer operators that can be treated indistinguishably in internal processing even if the layer operators are different for the target processing engine into calculation operators and registering them in pattern table information, the number of pattern matchings can be reduced, and the processing time for pattern matching can be shortened.

[0072] Second Embodiment Next, a model transformation device according to a second embodiment will be described. Note that the configuration of the model transformation device according to the second embodiment is the same as that of the first embodiment, so the same reference numerals are used and the description thereof will be omitted.

[0073] The second embodiment differs from the first embodiment in that at least one type of pattern made up of consecutive layer operators that can be processed by the same arithmetic operator in the processing engine is replaced with an arithmetic operator.

[0074] The first replacement unit 68 of the model transformation device 10 according to the second embodiment replaces, in the post-removal pattern table information, multiple types of layer operators that can be processed by the same arithmetic operator in the processing engines 50A and 50B, with that arithmetic operator.

[0075] The first replacing unit 68 further replaces, in the pattern table information after removal, at least one type of pattern made up of consecutive layer operators that can be processed by the same arithmetic operator in the processing engines 50A, 50B, with that arithmetic operator.

[0076] For example, as shown in Fig. 9 , a pattern consisting of a plurality of consecutive layer operators or a repetition of a plurality of consecutive layer operators is replaced with the arithmetic operator. The example of Fig. 9 shows that, in the case of a processing engine capable of processing four layer operators, i.e., pooling, convolution, batch normalization, and activation function, different patterns such as: Padding + Conv2D + BatchNormalization + Activation; Conv2D + BatchNormalization + Activation; Conv2D + BatchNormalization; and Conv2D + Activation are grouped together into the arithmetic operator "Conv2D'". In this case, the name of the arithmetic operator after replacement may be determined according to the number of Conv2D layers before replacement. FIG. 9 shows an example in which the name of the arithmetic operator after replacement is “Consv2D″” when the number of Conv2D layers before replacement is two, and the name of the arithmetic operator after replacement is “Consv2D′″” when the number of Conv2D layers before replacement is three.

[0077] The second replacing unit 70 similarly replaces the layer operators included in the layer information with arithmetic operators in accordance with the results of the replacement by the first replacing unit 68. Similarly, the second replacing unit 70 replaces at least one type of pattern consisting of consecutive layer operators included in the layer information with arithmetic operators in accordance with the results of the replacement by the first replacing unit 68.

[0078] Specifically, the second replacement unit 70 similarly replaces a layer operator or a pattern consisting of consecutive layer operators with an arithmetic operator on the model graph side of the input model in accordance with the result of replacement by the first replacement unit 68.

[0079] The second replacement unit 70 also creates information that indicates the correspondence between the layer operator before replacement or a pattern made up of consecutive layer operators and the arithmetic operator after replacement.

[0080] The other configurations and operations of the model transforming device 10 according to the second embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0081] As described above, according to the model transformation device of the second embodiment, if the processing engine can process patterns of multiple consecutive layer operators using the same calculation operator, the patterns can be integrated into that calculation operator, thereby reducing the number of pattern matching operations.

[0082] The present invention is not limited to the device configuration and operation of the above-described embodiment, and various modifications and applications are possible within the scope of the gist of the present invention.

[0083] For example, the first replacement unit 68 may replace in advance, in the pattern table information, at least one type of pattern consisting of consecutive layer operators that can be processed by the same arithmetic operator in the processing engines 50A and 50B, with that arithmetic operator.

[0084] In addition, the various processes executed by the CPU after reading software (programs) in the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Lattice Devices) whose circuit configuration can be changed after manufacturing, such as FPGAs, and dedicated electrical circuits, such as ASICs, which are processors with circuit configurations specifically designed to execute specific processes. The model conversion process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0085] In addition, in each of the above embodiments, the model conversion program is described as being pre-stored (installed) in the storage 94, but the present invention is not limited to this. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network. The present invention may also be applied to a program product.

[0086] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0087] The following additional notes are provided regarding the above-described embodiments.

[0088] (Supplementary Item 1) A model conversion device comprising: a memory; and at least one processor connected to the memory, wherein the processor receives layer information representing layer operators used in each layer of an input model which is a neural network; removes the consecutive layer operators, including layer operators not included in the received layer information, from pattern table information which stores consecutive layer operators that can be processed collectively by a processing engine; and converts the consecutive layer operators included in the layer information into batch processing operators in the processing engine using the pattern table information after removal.

[0089] (Addendum 2) A non-transitory storage medium storing a program that can execute model conversion processing by a computer, wherein the model conversion processing receives layer information that represents layer operators used in each layer of an input model that is a neural network, removes the consecutive layer operators, including layer operators that are not included in the received layer information, from pattern table information that stores consecutive layer operators that can be processed collectively by a processing engine, and converts the consecutive layer operators included in the layer information into batch processing operators in the processing engine using the pattern table information after removal.

[0090] REFERENCE SIGNS LIST 10 Model transformation device 50A, 50B Processing engine 60 Model input unit 62A, 62B Pattern table storage unit 64 Graph transformation unit 66 Removal unit 68 First replacement unit 70 Second replacement unit 72 Transformation unit 74 Graph generation unit 76 Restoration unit 78 Code generation unit 91 CPU 94 Storage 95 Input unit 96 Display unit

Claims

1. A model conversion device comprising: a model input unit that receives layer information representing layer operators used in each layer of an input model that is a neural network; a removal unit that removes consecutive layer operators, including layer operators not included in the received layer information, from pattern table information that stores consecutive layer operators that can be processed collectively by a processing engine; and a conversion unit that uses the pattern table information after removal to convert the consecutive layer operators included in the layer information into batch processing operators in the processing engine.

2. A model conversion device as described in claim 1, further comprising: a first replacement unit that replaces, in the pattern table information after removal, multiple types of layer operators that can be processed by the same arithmetic operator in the processing engine with the arithmetic operator; a second replacement unit that replaces the layer operators included in the layer information with the arithmetic operator in accordance with the replacement by the first replacement unit; and a restoration unit that restores the arithmetic operator to the layer operator for the result of conversion by the conversion unit, wherein the conversion unit converts the consecutive layer operators included in the layer information after replacement by the second replacement unit into the batch processing operator using the pattern table information after removal after replacement by the first replacement unit.

3. A model transformation device as described in claim 2, wherein the second replacement unit creates information representing the correspondence between the layer operator before replacement and the arithmetic operator after replacement, and the restoration unit restores the arithmetic operator to the layer operator using the information representing the correspondence.

4. A model transformation device as described in claim 2, wherein the first replacement unit further replaces at least one type of pattern consisting of consecutive layer operators in the pattern table information after removal, which can be processed by the same arithmetic operator in the processing engine, with the arithmetic operator.

Images