Processing unit, method for processing trained models, and program for processing trained models

Abstract

To easily and properly reduce the number of layers to execute by processing of a learned model.SOLUTION: In an inference device 100 for processing learned model data 51 including a plurality of processing layers, a CPU which executes a control routine 10 receives an input of learned model data 51, specifies a pair of plural processing layers which can be converted to an equivalent layer with less layers from the learned model data 51, and generates execution stipulation information for stipulating execution of the processing layer to execute the specified pair of processing layers as an equivalent layer.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technique for converting the configuration of a learned model, for example.

Background Art

[0002] In recent years, when performing recognition and classification of images, characters, etc., a neural network, which is a type of machine learning, has been used. A neural network is composed of various layers such as a convolutional layer, a pooling layer, and an activation function layer, and a model structure is defined by combining these. Since the defined model cannot perform recognition as it is, it is necessary to first set (learn) the parameters of each layer using input data for learning, label data, etc., to obtain a learned model. Once a learned model can be obtained, subsequently, by inputting the data to be recognized into the learned model and performing calculations (inferring) on each layer, a recognition result can be obtained.

[0003] An inference device that performs inference using a learned model generally includes a processing routine or circuit block (hereinafter sometimes collectively referred to as a routine) that executes processing corresponding to each layer, and a work area that temporarily stores the results. For example, when performing inference using a neural network composed of three layers, a convolutional layer A, a pooling layer B, and an activation function layer C, in the inference device, input data is given to the routine of the convolutional layer, the calculation result Ao of this convolutional layer is stored in the work area, the calculation result Ao is input to the routine of the pooling layer and the calculation result Bo is stored in the work area, and the calculation result Bo is input to the routine of the activation function layer to obtain the final result.

[0004] One format for expressing the structure of a neural network model is the ONNX model format. The ONNX model format is a general-purpose model format that can be used in multiple inference devices and programs, and specifications such as the file structure and the specifications of each layer are defined. Currently, there are more than 150 types of layers available in the ONNX model format.

[0005] Regarding techniques related to neural network processing, for example, Non-Patent Document 1 discloses methods for dealing with excessively slow neural network processing. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Hitachi's Autonomous Driving Research Division Announces Causes and Solutions for Slow CNN Processing [Accessed November 14, 2022], Internet<URL:https: / / xtech.nikkei.com / atcl / nxt / column / 18 / 01537 / 00465 / > [Overview of the project] [Problems that the invention aims to solve]

[0007] When running a multi-layered model on an inference system, routines corresponding to each layer of the model are called, and the results of the processing performed by each routine are stored in memory each time. Therefore, calculations and storage are performed for each layer of the neural network model, which leads to the problem of long processing times.

[0008] For example, as described in Non-Patent Document 1, it is conceivable to create a model optimized for a single layer that combines multiple layers, taking into account the processing of each layer in advance. However, it is not easy to identify how to optimize each layer and then create such a model.

[0009] Furthermore, when using a predetermined format such as the ONNX model format, there is a problem in that the number of layers available in the model is limited, making it impossible to optimize the layers.

[0010] This invention has been made in view of the above circumstances, and its purpose is to provide a technology that can easily and appropriately reduce the number of layers to be executed in processing a trained model. [Means for solving the problem]

[0011] To achieve the above objective, the processing device relating to one viewpoint is a processing device for processing a trained model including a plurality of processing layers, comprising: a receiving unit that receives input of the trained model; and a generating unit that identifies a plurality of sets of processing layers from the trained model that can be converted into an equivalent layer with fewer layers, and generates execution definition information that defines the execution of the processing layers so that the identified plurality of sets of processing layers are executed as the equivalent layer. [Effects of the Invention]

[0012] According to the present invention, the number of layers to be executed in processing a trained model can be easily and appropriately reduced. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a diagram showing the configuration of an inference device according to one embodiment. [Figure 2] Figure 2 is a diagram illustrating the functional configuration of an inference device according to one embodiment. [Figure 3] Figure 3 illustrates the configuration of a trained model according to one embodiment. [Figure 4] Figure 4 shows the format of trained model data according to one embodiment. [Figure 5] Figure 5 shows an example of trained model data according to one embodiment. [Figure 6] Figure 6 is a diagram illustrating the configuration of routine assignment information according to one embodiment. [Figure 7] Figure 7 is a diagram illustrating the configuration of matrix allocation information according to one embodiment. [Figure 8] Figure 8 is a flowchart of the inference process according to one embodiment. [Figure 9]FIG. 9 is a configuration diagram of an example of an execution procedure list in a pre-replacement learned model. [Figure 10] FIG. 10 is a diagram for explaining an example of a replaceable layer group according to an embodiment. [Figure 11] FIG. 11 is a configuration diagram of an example of an execution procedure list in a learned model after replacement. [Figure 12] FIG. 12 is a flowchart of layer process processing according to an embodiment. [Figure 13] FIG. 13 is a diagram for explaining another example of a replaceable layer group according to an embodiment. [Figure 14] FIG. 14 is a configuration diagram of the first half of a replaceability determination table. [Figure 15] FIG. 15 is a configuration diagram of the second half of a replaceability determination table. [Figure 16] FIG. 16 is a configuration diagram of known model structure correspondence information. [Figure 17] FIG. 17 is a diagram for explaining an example of a processing routine for executing the processing of an integration layer. [Figure 18] FIG. 18 is a diagram for explaining another example of a processing routine for executing the processing of an integration layer. [Figure 19] FIG. 19 is a diagram for explaining the configuration of a general-purpose replaceable layer group according to a modification example. [Figure 20] FIG. 20 is a diagram for explaining an example of the configuration of a general-purpose replaceable layer group according to a modification example. [Figure 21] FIG. 21 is a diagram for explaining an example of the configuration of a replaceable layer group in a general-purpose replaceable layer group according to a modification example. [Figure 22] FIG. 22 is a diagram for explaining an example of an arithmetic circuit for realizing an integration layer corresponding to a general-purpose replaceable layer group according to a modification example. [Figure 23] FIG. 23 is a flowchart of inference processing according to a modification example.

MODE FOR CARRYING OUT THE INVENTION

[0014] Embodiments will be described with reference to the drawings. Note that the embodiments described below are not intended to limit the invention as defined in the claims, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0015] Figure 1 is a diagram showing the configuration of an inference device according to one embodiment.

[0016] The inference device 100 is an example of a processing unit and is composed of computer devices such as a PC (Personal Computer) or a server. The inference device 100 includes, for example, a Central Processing Unit (CPU) 101, a main memory 102, a Graphics Processing Unit (GPU) 103, a reader / writer 104, a communication interface (communication I / F) 105, an auxiliary storage device 106, an input / output interface (input / output I / F) 107, a display device 108, and an input device 109. The CPU 101, main memory 102, GPU 103, reader / writer 104, communication I / F 105, auxiliary storage device 106, input / output I / F 107, and display device 108 are connected via a bus 110.

[0017] The CPU 101 reads the processing program stored in the auxiliary storage device 106 into the main memory 102 and executes it.

[0018] Main memory 102 is, for example, RAM, ROM, etc., and stores programs executed by the CPU 101 (such as trained model processing programs) and various information. Auxiliary storage device 106 is, for example, a non-temporary storage device (non-volatile storage device) such as a Hard Disk Drive (HDD) or Solid State Drive (SSD), and stores programs executed by the CPU 101 and various information.

[0019] The GPU 103 is a processor suitable for performing specific processes, such as image processing, and is suitable for performing parallel processes. In this embodiment, the GPU 103 performs predetermined processes according to instructions from the CPU 101.

[0020] The reader / writer 104 is capable of attaching and detaching the recording medium 111, and reads data from and writes data to the recording medium 111. Examples of recording mediums 111 include non-temporary recording media (non-volatile recording media) such as SD memory cards, FD (floppy disk: registered trademark), CDs, DVDs, BDs (registered trademark), and flash memory. In this embodiment, the trained model processing program may be stored in the recording medium 111, and the reader / writer 104 may read and use it. Alternatively, the recording medium 111 may be stored in trained model data or input data to be input to the trained model, and the reader / writer 104 may read and use it. Furthermore, the reader / writer 104 may store output data output from the trained model in the recording medium 111.

[0021] The communication interface 105 is connected to the network and transmits and receives data with other devices connected to network 50.

[0022] The input / output interface 107 is connected to an input device 109, such as a mouse or keyboard.

[0023] The display device 108 is, for example, a display device such as a liquid crystal display, and displays and outputs various information.

[0024] Next, the functional configuration of the inference device 100 will be described.

[0025] Figure 2 is a diagram illustrating the functional configuration of an inference device according to one embodiment.

[0026] The inference device 100 stores, for example, a control routine 10, multiple processing routines capable of executing processing layers of a trained model (convolutional layer routine 31, pooling layer routine 32, ReLU (Rectified Linear Unit) activation function layer routine 33, additive layer routine 34, multiplicative layer routine 35, etc.), and one or more integrated layer routines 40 capable of executing processing layers that integrate groups of multiple processing layers, in the main memory and / or auxiliary storage device 106. In this embodiment, each processing routine in the inference device 100 is executed by the CPU 101. At least one of the processing routines may be implemented by a dedicated arithmetic circuit.

[0027] The control routine 10, when executed by the CPU 101, reads the trained model data 51 and performs a process to convert a predetermined set of layers (replaceable layer group) in the trained model into an (equivalent) integrated layer (equivalent layer) corresponding to this set of layers. The control routine 10 also reads the input data 52 into the work area 20 when executed by the CPU 101. The control routine 10 also controls the CPU 101 to read and execute the processing routines corresponding to each layer according to the model to be executed. The control routine 10 also outputs the processing result of the model in the work area 20 as output data 53 when executed by the CPU 101. Here, the work area 20 may be, for example, a part of the main memory 102, or, if SRAM is provided in the CPU 101, it may be an area of ​​SRAM. Here, the receiving unit and the generating unit described in the claims are configured by the CPU 101 executing the control routine 10, and the inference unit is configured by the CPU 101 executing the control routine 10 to control other control routines.

[0028] The processing routine is executed by the CPU 101, which reads (loads) the data to be used for processing into the CPU 101, executes the processing using the read data, and stores the processing results from the CPU 101 in the work area 20. The processing routines, excluding the integration layer routine, are routines that execute the processing of the processing layer that is available in the specifications of the input trained model.

[0029] The convolution layer routine 31, executed by the CPU 101, performs convolution using each of the multiple filter data. The pooling layer routine 32, executed by the CPU 101, performs downsampling on the input data. The ReLU activation function layer routine 33, executed by the CPU 101, transforms the input data using the ReLU activation function (ramp function). The additive layer routine 34, executed by the CPU 101, performs addition on the input data. The multiplication layer routine 35, executed by the CPU 101, performs multiplication on the input data.

[0030] The integration layer routine 40, when executed by the CPU 101, performs integrated processing that combines the processing performed by the replaceable layer group. The integration layer routine 40 can perform integrated processing by being read once by the CPU 101. In this embodiment, this integration layer routine 40 is, for example, a processing layer that cannot be used in the specifications of the input trained model.

[0031] The integration layer routine 40 may be, for example, a routine that executes an integration layer that combines the ReLU activation function layer, the multiplication layer, and the addition layer.

[0032] Figure 3 illustrates the configuration of a trained model according to one embodiment.

[0033] The trained model 51A has a constant 61 (constant_2), a convolutional layer 62 (node_1), a ReLU activation function layer 63 (node_2), a multiplication layer 64 (node_3), an addition layer 65 (node_4), and a pooling layer 66 (node_5).

[0034] The convolutional layer 62 takes input data 52 (tensor_i) as input, performs a convolution operation, and outputs data (tensor_1).

[0035] The ReLU activation function layer 63 receives the data (tensor_1) output from the convolutional layer 62, performs processing using the ReLU activation function, and outputs the data (tensor_2).

[0036] The multiplication layer 64 takes a constant 61 (constant_2) and the data (tensor_1) output from the convolutional layer 62 as input, performs a multiplication operation, and outputs data (tensor_3).

[0037] The additive layer 65 receives the data (tensor_2) output from the ReLU activation function layer 63 and the data (tensor_3) output from the multiplier layer 64 as input, performs an addition operation, and outputs the data (tensor_4).

[0038] The pooling layer 66 receives the data (tensor_4) output from the additive layer 65, performs a downsampling process, and outputs output data 53 (tensor_o).

[0039] In this embodiment, for example, the number of layers in the trained model is reduced by converting a set of ReLU activation function layer 63, multiplication layer 64, and addition layer 65 into an integrated layer 70 that combines these processes.

[0040] Figure 4 shows the format of trained model data according to one embodiment.

[0041] The data representing the structure of a trained model (trained model data) is represented as a graph. The graph includes elements such as Input, which shows the input data for the entire model; Output, which shows the output data; Node, which shows each layer (node); Data, which shows constant data; and Tensor, which shows the matrices used. The names of the matrices representing the data are listed in Output and Input.

[0042] A Node contains elements such as Input (representing the input data for the layer), Output (representing the output data for the layer), Weight (representing the weights), Type (representing the type of layer), and Name (representing the name of the layer). Depending on the type of layer, it may also contain other elements.

[0043] Output, Input, and Weight contain the matrix names indicating the corresponding data in the layer. Type describes the type of layer; for example, Convolution for a convolutional layer, ReLU for a ReLU activation function layer, Mul for a multiplication layer, Add for an additive layer, Pool for a pooling layer, Tanh for a hyperbolic tangent function layer, Sin for a sine function layer, Cos for a cosine function layer, Matrix Mul for a matrix stack, Transpose for a matrix transpose layer, and Select for a selection layer. Name contains a name (layer name, node name) that can identify it as a layer of the same type in the trained model. For example, if the trained model contains two convolutional layers, conv_1 is written for one convolutional layer and conv_2 for the other convolutional layer.

[0044] The Data element contains two components: Name, which represents the weights used in the trained model, and Array, which represents the array of values ​​for that matrix. Name stores the name of the matrix that represents the weight data.

[0045] A Tensor contains two elements: Name, which indicates the matrix (tensor) used in the trained model, and Shape. Name stores the name of the matrix. Shape stores an array of the matrix's shape. For example, the array of the matrix's shape contains information such as the dimension and size; for a 3x3 2x2 matrix, it would be (2,3,3).

[0046] Figure 5 shows an example of trained model data according to one embodiment. Figure 5 shows trained model data 51B described in the format shown in Figure 4 for the trained model shown in Figure 3.

[0047] For example, according to section 51Ba of the description of the ReLU activation function layer 63 in the trained model data 51B, it is stated that the node name is node_2, the type is ReLU, the input is tensor_1, and the output is tensor_2.

[0048] This pre-trained model data 51B can appropriately represent the structure of the pre-trained model 51A shown in Figure 3.

[0049] Next, the information used in the processing of control routine 10 will be described. Control routine 10 uses routine allocation information 130 and matrix allocation information ID 140. Routine allocation information 130 and matrix allocation information ID 140 are stored and used, for example, in main memory 102.

[0050] Figure 6 is a diagram illustrating the configuration of routine assignment information according to one embodiment.

[0051] Routine assignment information 130 is set for each layer of the trained model. Routine assignment information 130 includes fields for routine assignment information ID 131, model layer information 132, routine name 133, input assignment information array 134, output assignment information array 135, local model structure pattern matching processed flag 136, known model structure correspondence information ID 137, and inference completed flag 138.

[0052] The routine assignment information ID 131 stores an ID (routine assignment information ID) that identifies routine assignment information 130. The model layer information 132 stores information that identifies the layer in the trained model associated with the routine (e.g., node name). The routine name 133 stores the routine name associated with the layer. The input assignment information array 134 stores one or more IDs (matrix assignment information IDs) assigned to the information (matrix) input to the routine corresponding to the layer. The output assignment information array 135 stores one or more IDs (matrix assignment information IDs) assigned to the information output from the routine corresponding to the layer. The local model structure pattern matching processed flag 136 stores a flag indicating whether or not pattern matching processing with the replaceable layer group has been performed for a set of multiple layers including the layer corresponding to routine assignment information 130. The known model structure correspondence information ID 137 stores an ID (known model structure correspondence information ID) indicating the known model information that was matched by the pattern matching processing. The inference flag 138 stores a flag indicating whether or not inference processing was performed for the layer corresponding to routine assignment information ID 131 in the inference processing according to the model.

[0053] Next, we will explain the matrix allocation information ID 140.

[0054] Figure 7 is a diagram illustrating the configuration of matrix allocation information according to one embodiment.

[0055] The matrix allocation information 140 is information provided in correspondence with the matrices used in the model, and includes fields for matrix allocation information ID 141, matrix name 142, local model structure pattern matching processed flag 143, storage location information 144, and matrix size 145.

[0056] The matrix assignment information ID 141 stores the ID (matrix assignment information ID) corresponding to matrix assignment information 140. The matrix name 142 stores the name of the corresponding matrix. The local model structure pattern matching processed flag 143 stores a processed flag indicating whether or not pattern matching processing has been performed for the layers related to the corresponding matrix. The storage location information 144 stores information indicating the storage location of the corresponding matrix on the work area 20. The matrix size 145 stores the size of the corresponding matrix.

[0057] Next, we will explain the inference process performed by the inference device 100.

[0058] Figure 8 is a flowchart of the inference process according to one embodiment.

[0059] The control routine 10 of the inference device 100 (more precisely, the CPU 101 that executes the control routine 10, which may hereafter be simply referred to as the control routine 10) reads the trained model data 51 (S11).

[0060] Next, the control routine 10 performs a model structure analysis process to analyze the model structure based on the trained model data 51 (S12). In this model structure analysis process, based on the trained model data 51, the connection relationships of each layer in the trained model and the input / output relationships of the matrices for each layer are identified as shown in Figure 3.

[0061] Next, the control routine 10 performs a routine assignment process (S13) to assign the corresponding routine to the layers defined in the trained model data. In the routine assignment process, the control routine 10 creates matrix assignment information 140 for each matrix in the trained model data. At this time, the control routine 10 assigns a new matrix assignment information ID to the matrix assignment information ID 141, stores the matrix name of the matrix information (Tensor) in the model data in the matrix name 142, and stores the shape of the matrix information matrix in the matrix size 145. The local model structure pattern matching processed flag 143 is left unset at this point. Next, the control routine 10 generates routine assignment information 130 for each layer of the trained model data. At this point, the control routine 10 assigns a new routine assignment ID to routine assignment information ID 131, stores information identifying the layer in the model in model layer information 132, sets the name of the corresponding processing routine in routine name 133, and stores the matrix assignment information ID of the assignment information assigned to the corresponding layer in the model data in input assignment information array 134 and output assignment information array 135. The local model structure pattern matching processed flag 136, known model structure correspondence information 137, and inferred flag 138 are left unset at this point.

[0062] Next, the control routine 10 uses the routine assignment information 130 to determine the procedures for reading the routine, reading the data used by the routine from the work area 20 to the CPU 101, and storing it from the CPU 101 back into the work area 20, and generates an execution procedure list (see Figure 9).

[0063] Figure 9 is a diagram illustrating an example of the execution procedure list in the pre-trained model before substitution.

[0064] The execution procedure list 80 is a list of execution procedures when the trained model data 51B shown in Figure 5 is executed as is. As shown in the execution procedure list 80, for each layer included in the trained model, a call to a processing routine that performs the processing of the layer occurs, as well as the loading of input data and the storage of output data used in this processing routine. If the processing is carried out according to the execution procedure list 80, the number of times the processing routine is called is 5, and the number of load and store operations on the work area 20 is 14.

[0065] Returning to the explanation of Figure 8, the control routine 10 then performs a local model pattern matching process (pattern matching process) to detect whether or not there is a structure in the trained model data that matches the pattern of a set of replaceable layers (a replaceable layer group), that is, whether or not there is a replaceable layer group in the trained model data (S14).

[0066] Here, we will explain the replaceable layer group.

[0067] Figure 10 illustrates an example of a replaceable layer group according to one embodiment.

[0068] Examples of replaceable layer groups (replaceable layer groups) include a group containing a ReLU activation function layer, a multiplier layer, and an additive layer; a group containing a multiplier layer and an additive layer; a group containing a multiplier layer, a hyperbolic tangent function layer, and a multiplier layer; and a group containing a subtraction layer, a cosine function layer, a sine function layer, and a multiplier layer. A replaceable layer group may also consist of a set of processing layers including at least one ReLU activation function layer or a hyperbolic tangent function layer, and at least one multiplier layer or an additive layer.

[0069] A set including a ReLU activation function layer, a multiplication layer, and an addition layer, for example as shown in Figure 10(A), has a first stage which includes a ReLU activation function layer that receives the output of input layer [0] and a multiplication layer that receives the output of input layer [0] and a constant [0], and a second stage which includes an addition layer that receives the internal matrix [0] output from the ReLU activation function layer and the internal matrix [1] output from the multiplication layer. Here, the first stage is a group of layers that process data output from the input layer as input, and the nth stage is a group of layers that process data output from the (n-1)th stage as input. Note that even if a layer processes data output from the input layer as input, if it is a processing layer that receives data from another stage, it will be the processing layer of the stage immediately following the latest stage that receives the data.

[0070] A set including a multiplier and an adder, for example as shown in Figure 10(B), has a multiplier in the first stage that receives the output of the input layer [0] and a constant [0] as inputs, and an adder in the second stage that receives the internal matrix [0] output from the multiplier and a constant [1] as inputs.

[0071] A multiplier layer, a hyperbolic tangent function layer, and a set including a multiplier layer, for example as shown in Figure 10(C), has a first stage multiplier layer to which the output of the input layer [0] and the constant [0] are input, a second stage hyperbolic tangent function layer to which the internal matrix [0] output from the multiplier layer is input, and a third stage multiplier layer to which the output of the input layer [0] and the internal matrix [1] output from the hyperbolic tangent function layer are input.

[0072] A set including a subtraction layer, a cosine function layer, a sine function layer, and a multiplication layer, for example as shown in Figure 10(D), has a first stage which is a subtraction layer that receives the output of input layer [0] and the output of input layer [1], and a cosine function layer that receives the output of input layer [1], a second stage which is a sine function layer that receives the internal matrix [0] output from the subtraction layer, and a third stage which is a multiplication layer that receives the internal matrix [2] output from the sine function layer and the internal matrix [1] output from the cosine function layer.

[0073] Next, we will explain an example of pattern matching.

[0074] First, we will explain how replaceable layer groups can be replaced with predetermined integration layers. The trained model will be explained using the model shown in Figure 3 as an example.

[0075] Here, if we define the multidimensional matrix X as a 2x2 two-dimensional matrix, and let xi be the i-th element of the multidimensional matrix X, then the matrix X will be expressed as shown in equation (1) below.

[0076]

number

[0077] Furthermore, when a function is executed on a matrix, the operation performed on each element shall be indicated by separating them with a │. For example, in the case of a matrix operation FUNC that performs an operation on each element of matrices A and B using func(a,b), if the resulting matrix is ​​C, it shall be represented as shown in equation (2) below. C=FUNC(A,B)│ci=func(ai,bi) ···(2)

[0078] Here, if we let X be the input matrix for the ReLU activation function layer, xi be an element in matrix X, Y be the output matrix, and yi be the i-th element in matrix Y, then it can be expressed as shown in equation (3) below.

[0079]

number

[0080] Furthermore, for the additive and multiplier layers, if we have two inputs, with one input matrix as X1 and the other as X2, and the output matrix as Y, and the elements of each matrix as x1i, x2i, and yi respectively, then the output matrix Y of the additive layer can be expressed as shown in equation (4), and the output matrix Y of the multiplier layer can be expressed as shown in equation (5). Y=ADD(X1,X2)│yi=x1i+x2i (4) Y=MUL(X1,X2)│yi=x1i×x2i (5)

[0081] Here, we consider the case where the set of ReLU activation function layer 63, multiplier layer 64, and adder layer 65 in Figure 3 is executed as an integrated layer 70. Let the matrix operation of the integrated layer 70 be FU_ACT, the input matrix to the integrated layer 70 be A, the output matrix be B, and the constant input matrix of the multiplier layer be C, with elements ai, bi, and ci respectively. Then the output matrix B can be expressed as shown in equation (6) below. B=FU_ACT(A,C)=ADD(ReLU(A),MUL(A,C))...(6) Here, the operation of each element in this process can be expressed as shown in equation (7) below.

[0082]

number

[0083]

number

[0084] Therefore, it can be seen that the set of ReLU activation function layer 63, multiplication layer 64, and addition layer 65 can be replaced with an integration layer 70 that performs the process of equation (8).

[0085] In this way, by replacing the set of ReLU activation function layer 63, multiplication layer 64, and addition layer 65 with an integration layer 70 that performs the processing of equation (8), it is possible to improve the efficiency of calculations, reduce the loading and storing of intermediate data matrices, thereby accelerating inference processing and reducing the amount of intermediate data.

[0086] Next, an example of pattern matching processing will be described. In this pattern matching processing, a condition (determination condition) for a predetermined replaceable layer group and a determination process to determine whether or not that condition is met are incorporated into the control routine 10, and pattern matching is performed by executing this process. Here, as an example, we will explain pattern matching that determines whether or not a structure matching a set of ReLU activation function layer, multiplier layer, and additive layer exists in the trained model shown in Figure 3. Note that the destination of the matrix is ​​the path in the direction of the model output (where the matrix is ​​connected to the input of the layer), and the source of the connection is the path in the direction of the model input (where the matrix is ​​connected to the output of the layer).

[0087] Here, it is assumed that routine assignment information 130 is constructed and initialized during routine assignment processing. In routine assignment information 130 corresponding to the node_1 layer, the local model pattern matching processed flag 136 is set to the processed flag. In routine assignment information 130 corresponding to the other layers, the local model structure pattern matching processed flag 136 is not set to the pattern matching processed flag. In matrix assignment information 140 for each matrix, the local model structure pattern matching processed flag 143 is not set to the pattern matching processed flag.

[0088] In this case, the control routine 10 searches for layers and matrices from the input side of the model structure and selects a matrix for which the local model structure pattern matching processed flag 143 of the matrix assignment information 140 is not set. In this example, since the flag is not set for matrix tensor_1, tensor_1 is selected.

[0089] Next, the control routine 10 enumerates the layers that utilize the selected matrix and determines whether the conditions of the first stage are met by checking whether there are two enumerated layers, one of which is a ReLU activation function layer and the other is a multiplication layer.

[0090] In the case of the trained model shown in Figure 3, there are two layers that utilize the matrix tensor_1, node_2 and node_3. Since node_2 is a ReLU activation function layer and node_3 is a multiplication layer, it is determined that the conditions for the first layer are met.

[0091] Next, the control routine 10 determines whether the conditions for the second stage are met by checking whether the following conditions are met: the matrix output by the ReLU activation function layer is connected to an adder layer, the matrix output by the multiplier layer is connected to an adder layer, and the adder layer to which the matrix output by the ReLU activation function layer is connected is the same as the adder layer to which the matrix output by the multiplier layer is connected. If these conditions are met, the second stage is identified as satisfying the conditions.

[0092] In the case of the trained model shown in Figure 3, the matrix tensor_2 output by the ReLU activation function layer node_2 is connected to the additive layer node_4, and the matrix tensor_3 output by the multiplicative layer node_3 is connected to the additive layer node_4. Since the connections of matrices tensor_2 and tensor_3 are both to the same node_4, the condition is satisfied, and thus the second condition is identified as being satisfied.

[0093] Thus, if the conditions of the first and second stages are met, the ReLU activation function layer node_2, the multiplication layer node_3, and the addition layer node_4 targeted in this process have the same pattern as the replaceable layer group, that is, they are a replaceable layer group and are replaceable with an integrated layer capable of performing the same processing as this set of layers. Therefore, the control routine 10 sets the pattern matching processing completed flags in the local model structure pattern matching processed flag 136 of the routine assignment information 130 for these layers and in the local model structure pattern matching processed flag 143 of the matrix assignment information 140 for the matrices input to these layers.

[0094] In the case of the trained model shown in Figure 3, the local model structure pattern matching processed flag 136 is set for the routine assignment information 130 corresponding to the multiple replaceable layers node_2, node_3, and node_4, and the local model structure pattern matching processed flag is set for the matrix assignment information 140 corresponding to tensor_1, tensor_2, and tensor_3.

[0095] On the other hand, if the conditions are not met during the pattern matching process, the processing for the selected matrix is ​​terminated at that point, a processed flag is set in the local model structure pattern matching processed flag 143 of the matrix assignment information 140 corresponding to the initially selected matrix, and a processed flag is set in the local model structure pattern matching processed flag 136 of the routine assignment information 130 corresponding to all layers to which this matrix is ​​connected.

[0096] In this process, we focused on matrices, selected a matrix, and performed pattern matching by traversing the layers to which that matrix is ​​connected. However, for example, we could focus on layers, select a layer, and perform pattern matching by traversing the layers connected to that layer. Specifically, for example, we could refer to routine assignment information 130, search for layers where the pattern matching flag has not been set, and then perform a determination process on the layers to which the matrix output by that layer is connected, based on whether they satisfy the conditions of the replaceable layer group. Also, in the above pattern matching process, we searched for conditions from the input side of the trained model, but we could also search from the output side.

[0097] Returning to the explanation of Figure 8, the control routine 10 executes a known model structure routine replacement process (S15) which replaces the multiple layers (replaceable layer group) detected as matching the replaceable layer group in the pattern matching process with the integrated layer corresponding to this set of multiple layers.

[0098] In the known model structure routine replacement process, the control routine 10 generates an execution procedure list (an example of execution specification information) by replacing the procedures for the replaceable layer groups in the execution procedure list created in the routine assignment process with the procedures for the integration layer corresponding to the replaceable layer groups, based on the replaceable layer groups identified in the pattern matching process.

[0099] Figure 11 is a diagram illustrating an example of an execution procedure list according to one embodiment.

[0100] Execution procedure list 81 is an execution procedure list created when the execution procedure rule created in the routine assignment process is execution procedure list 80 shown in Figure 9, and the ReLU activation function layer, multiplication layer, and addition layer are identified as a replaceable layer group. As shown in Figure 11, execution procedure list 81 is an execution procedure list in which the procedures for the ReLU activation function layer, multiplication layer, and addition layer (procedures 05-15 in Figure 9) in execution procedure list 80 are converted (substituted) to procedures for the integration layer (procedures 05-08). When processing is performed according to execution procedure list 81, the number of times the processing routine is called is 3, and the number of loads and stores to the work area 20 is 9. Compared to processing according to execution procedure list 80, the number of operations is reduced by 2 and the number of loads and stores to the work area 20 is reduced by 5, improving processing efficiency.

[0101] Returning to the explanation of Figure 8, the control routine 10 then executes processing according to the execution procedure list 81. Specifically, the control routine 10 reads the data to be input to the trained model (model input data) from, for example, the auxiliary storage device 106 and stores it in the work area 20 (S16).

[0102] Next, the control routine 10 reads the processing routine corresponding to each layer in the execution procedure list 81 and executes the processing of loop 1 (S17: layer process processing: see Figure 12).

[0103] Next, the layered process in step S17 will be described.

[0104] Figure 12 is a flowchart of a layered process according to one embodiment.

[0105] The processing routine read by the control routine 10 reads the data to be input to the layer (layer input data) from the work area 20 to the CPU 101 for the layer to be processed (S21). Next, the processing routine executes the arithmetic processing (part of the inference processing) described in the processing routine (S22), writes the data obtained from the arithmetic processing (layer output data) to the work area 20 (S23), and terminates the layer process processing.

[0106] Returning to the explanation of Figure 8, when the layer process processing by one layer is completed, the control routine 10 executes the layer process processing by the next layer. When the process processing of all layers in the execution procedure list 81 is completed, it exits the loop 1 process and writes the final calculation result (inference result) stored in the work area as model output data to, for example, the auxiliary storage device 106 (S18). The model output data written to the auxiliary storage device 106 is output to, for example, another device via the network.

[0107] The replaceable layer group is not limited to the configuration shown in Figure 10 above, but may also be configured as shown below.

[0108] Figure 13 illustrates another example of a replaceable layer group according to one embodiment.

[0109] Examples of replaceable layer groups include a set including a matrix stack and an addition layer that performs matrix multiplication, a set including a multiplication layer, an addition layer and a matrix stack, a set including a matrix transpose layer and an addition layer that performs matrix transposition, and a set including a selection layer and an addition layer that selects multiple inputs.

[0110] A set including a matrix stack and an adder layer, for example as shown in Figure 13(A), has a matrix stack in the first stage to which the output of the input layer [0] and a constant [0] are input, and an adder layer in the second stage to which the internal matrix [0] output from the matrix stack and a constant [1] are input.

[0111] A set including a multiplier layer, an additive layer, and a matrix stack, for example as shown in Figure 13(B), has a multiplier layer in the first stage that receives the output of the input layer [1] and a constant [0] as inputs, an additive layer in the second stage that receives the internal matrix [0] output from the multiplier layer and a constant [1] as inputs, and a matrix stack in the third stage that receives the output of the input layer [0] and the internal matrix [1] output from the additive layer as inputs.

[0112] A set including a matrix transpose layer and an adder layer, for example as shown in Figure 13(C), has a matrix transpose layer in the first stage to which the output of input layer [0] is input, and an adder layer in the second stage to which the internal matrix [0] output from the matrix transpose layer and the output of input layer [1] are input.

[0113] A set including a selection layer and an additive layer, for example as shown in Figure 13(D), has a first stage which is a selection layer that receives the output of input layer [0], the output of input layer [1], and the output of input layer [2], and a second stage which is an additive layer that receives the output of input layer [0] and the internal matrix [0] output from the selection layer.

[0114] Next, another example of pattern matching processing will be described. In this pattern matching processing, a replaceability determination table 120 (see Figures 14 and 15) for determining whether a layer group is replaceable, and known model structure correspondence information 150 (see Figure 16) are stored in the main memory 102 or auxiliary storage device 106, and these tables are used to perform pattern matching between the configuration of the trained model and the replaceable layer group.

[0115] First, before explaining the pattern matching process, we will describe the replaceability determination table 120 and the known model structure correspondence information 150.

[0116] Figure 14 is a diagram showing the configuration of the first half of the replaceability determination table. Figure 15 is a diagram showing the configuration of the second half of the replaceability determination table. The replaceability determination table 120 in Figures 14 and 15 is an example where the replaceable layer group consists of the four replaceable layer groups shown in Figure 10.

[0117] The replaceability determination table 120 includes 121 patterns and 122 fields for each pattern (122-1, 122-2, 122-3, 122-4).

[0118] The number of patterns (121) stores the number of replaceable layer groups (number of patterns). The pattern field (122) stores pattern information for each replaceable layer group. The pattern field (122) includes the number of stages (1221), the number of inputs (1222), the number of constant inputs (1223), the number of internal matrices (1224), the number of outputs (1225), the replacement block routine (1226), and the stage fields (1227, such as 1227-1, 1227-2, etc.).

[0119] The number of layers (1221) stores the number of layers in the pattern. The number of inputs (1222) stores the number of inputs in the pattern. The number of constant inputs (1223) stores the number of constant inputs in the pattern. The number of internal matrices (1224) stores the number of internal matrices in the pattern. The number of outputs (1225) stores the number of outputs in the pattern. The substitution block routine (1226) stores the type of integrated layer processing routine that can be substituted with the pattern.

[0120] The row-by-row field 1227 stores various information about the row in the pattern. For example, the row-by-row field 1227 includes the number of elements (layers) contained in the row, information about the matrix input to each element, information about the matrix output from each element, and information about the type of each element.

[0121] Figure 16 is a diagram showing the configuration of the known model structure correspondence information.

[0122] The known model structure correspondence information 150 is, for example, information provided in correspondence with a trained model, and includes fields for known model structure correspondence information ID 151, confirmation flag 152, number of candidates 153, and candidate-specific information 154.

[0123] The Known Model Structure Correspondence Information ID 151 stores an ID (Known Model Structure Correspondence Information ID) that identifies Known Model Structure Correspondence Information 150. The Confirmation Flag 152 stores a flag indicating whether or not it has been confirmed that a candidate pattern included in Known Model Structure Correspondence Information 150 is replaceable. The Number of Candidates 153 stores the number of candidates included in Known Model Structure Correspondence Information 150. The Candidate Information 154 stores information about the candidate patterns. The Candidate Information 154 is stored for each candidate pattern, and if a pattern that is not a candidate is found, the Candidate Information 154 for the corresponding pattern is deleted.

[0124] The candidate information 154 includes the number of inputs, the number of outputs, the number of elements, one or more inputs, one or more outputs, and one or more elements. The number of inputs stores the number of inputs in the corresponding candidate pattern. The number of outputs stores the number of outputs in the corresponding candidate pattern. The number of elements stores the number of elements in the corresponding candidate pattern. The inputs store the ID of the matrix assignment information of the data (matrix) input to the corresponding candidate pattern. The outputs store the ID of the matrix assignment information of the data (matrix) output from the corresponding candidate pattern. The elements store the position information of the element stored in the corresponding candidate pattern and the routine assignment information ID assigned to that element.

[0125] Next, we will discuss other examples of pattern matching.

[0126] In the example described above, pattern matching was performed by determining conditions. However, in this pattern matching process, the replaceable layer groups are registered in the replaceability determination table 120, and pattern matching is performed using the replaceability determination table 120 and the known model structure correspondence information 150.

[0127] Here, we will explain the pattern matching process using the trained model in Figure 3 and the replaceability determination table 120 shown in Figures 14 and 15 as an example.

[0128] First, the control routine 10 initializes the known model structure correspondence information 150 with the number of candidates = 0.

[0129] Next, the control routine 10 executes the first stage determination process. <First stage judgment process> The control routine 10 searches for layers and matrices from the input side of the trained model, selects a matrix for which the processed flag 143 of the local model structure pattern match processing information 140 is not set, and creates a list of input layer candidates that include the selected matrix. For example, in the case of the trained model in Figure 3, the processed flag 143 of the local model structure pattern match processing information 140 corresponding to matrix tensor_1 is not set, so tensor_1 is selected and the processed flag is set in the local model structure pattern match processing flag 143.

[0130] Next, the control routine 10 enumerates the layers to which the selected matrix should be connected, based on the structure of the trained model, and stores them in the main memory 102 as a set of candidate first-stage components. If the same layer is enumerated, the duplicate is removed before storage. For example, in the case of the trained model in Figure 3, node_2 (ReLU activation function layer) and node_3 (multiplication layer) are enumerated and stored in the set of candidate first-stage components.

[0131] Next, the control routine 10 determines whether the layers in the first-stage component candidate set are included in the first-stage components of each pattern in the replaceability determination table 120, and adds the partially matching replaceable layer group patterns as candidates to the candidate information 154 of the known model structure correspondence information 150. For example, in the case of the trained model in Figure 3, the ReLU / multiplication / addition layer group (the pattern in Figure 10(A)) is added as a candidate.

[0132] Next, the control routine 10 identifies the input matrices of each layer in the first stage candidate component set and determines whether or not there are matrices for which the processed flag is not set in the local model structure pattern matching processed flag 143 of the matrix assignment information 140.

[0133] As a result, if there is a matrix for which the local model structure pattern matching processed flag 143 does not have the processed flag set, the control routine 10 selects that matrix, identifies the layer to which this matrix is ​​connected, adds it to the first-stage component candidate set, and determines whether or not it is included in the first-stage components of each pattern recorded as a candidate in the known model structure correspondence information 150. If the layer added to the first-stage component candidate set is not included in any pattern, this layer is removed from the first-stage component candidate set.

[0134] Once the identification of candidate first-stage components is complete, the control routine 10 determines whether any of the patterns added as candidates to the known model structure correspondence information 150 partially match the components of the first stage, and removes any patterns that do not partially match from the candidates in the known model structure correspondence information 150. Furthermore, if there are any partially matching patterns that consist only of the first stage, the control routine 10 performs the confirmation determination process described later on those patterns.

[0135] If the confirmation flag 152 of the known model structure correspondence information 150 is not set, and there are candidates in the known model structure correspondence information 150, the control routine 10 further executes the second stage determination process described later. If there are no candidates, it executes the termination process described later.

[0136] For example, in the case of the trained model in Figure 3, the ReLU activation function layer and multiplication layer, which are the components of the first stage of the candidate ReLU / multiplication / addition layer group, perfectly match node_2 (ReLU activation function layer) and node_3 (multiplication layer) in the first stage component candidate set. Therefore, the ReLU / multiplication / addition layer group is maintained as a candidate. Also, since the ReLU / multiplication / addition layer group is not a pattern consisting only of the first stage, the determination process is not performed. The determination flag 152 in the known model structure correspondence information 150 does not have a determination flag set, and since a candidate exists, the second stage determination process is executed further.

[0137] Next, we will explain the second stage of the judgment process. <Second stage judgment process> The control routine 10 identifies the matrices output by each layer belonging to the first stage component candidate set, and for each identified matrix, selects a matrix for which the processed flag 143 of the matrix assignment information 140 local model structure pattern matching processed flag is not set, sets the processed flag, identifies the layer to which this matrix is ​​connected, and stores it in the main memory 102 as the second stage component candidate set.

[0138] For example, in the case of the trained model in Figure 3, the matrix tensor_2 output by node_2, one of the candidate components in the first stage, is identified, and node_4 (the summation layer) to which tensor_2 is connected is stored in the candidate component in the second stage. The matrix tensor_3 output by node_3, another candidate in the candidate component in the first stage, is identified, and node_4 (the summation layer) to which tensor_3 is connected is identified, but node_4 is not added because it is already stored in the candidate component in the second stage.

[0139] Next, the control routine 10 determines whether the set of candidate second-stage components partially matches the second-stage components of patterns registered as candidates in the known model structure correspondence information 150, and removes patterns that do not partially match from the candidates in the known model structure correspondence information 150.

[0140] For example, in the case of the trained model in Figure 3, it matches the second-stage component of the candidate ReLU / multiplication / addition layer group, and is therefore retained as a candidate.

[0141] Next, the control routine 10 identifies the input matrices of each layer belonging to the second stage component candidate set and determines whether or not there are matrices for which the processed flag is not set in the local model structure pattern matching processed flag 143 of the matrix assignment information 140.

[0142] As a result, if there is a matrix for which the local model structure pattern matching processed flag 143 does not have the processed flag set, the control routine 10 selects that matrix, identifies the layer to which this matrix is ​​connected, adds it to the second-stage component candidate set, and determines whether or not it is included in the second-stage components of each pattern recorded as a candidate in the known model structure correspondence information 150. If the layer added to the second-stage component candidate set is not included in any pattern, the control routine 10 removes this layer from the second-stage component candidate set.

[0143] Furthermore, the control routine 10 identifies the layer from which the matrix is ​​connected, adds the identified layer to the first-stage component candidate set, and determines whether or not it is included in the second-stage components of each pattern recorded as a candidate in the known model structure correspondence information 150. If the layer added to the first-stage component candidate set is not included in any pattern, the control routine 10 removes it from the first-stage component candidate set.

[0144] Once the identification of candidate second-stage components is complete, the control routine 10 determines whether any of the patterns added as candidates to the known model structure correspondence information 150 partially match the components of the second stage, and removes any patterns that do not partially match from the candidates in the known model structure correspondence information 150. Furthermore, if there are any partially matching patterns that consist only of stages that have already been processed (i.e., the first and second stages), the control routine 10 performs a confirmation determination process on those patterns.

[0145] If the confirmation flag 152 of the known model structure correspondence information 150 is not set, and there is a candidate in the known model structure correspondence information 150, the control routine 10 further executes the determination process for the next stage n (the nth stage determination process). If there is no candidate, it executes the termination process described later. Note that the nth stage determination process can be performed by replacing the second stage and the number of each stage related to it in the second stage determination process with the nth stage and the number of stages related to it.

[0146] For example, in the case of the trained model in Figure 3, the additive layer, which is a component of the second stage of the candidate RELU / multiply / additive layer group, partially matches node_4 (additive layer), which is a set of candidate components for the second stage. Therefore, the ReLU / multiply / additive layer group is maintained as a candidate. This ReLU / multiply / additive layer group then undergoes a determination process to determine if it is a pattern composed of the first and second stages.

[0147] Next, we will explain the termination process. <Cancellation process> If the number of candidates in the known model structure correspondence information 150 becomes 0, the control routine 10 releases the processed flag 136 of the local model structure pattern match processed flag 130 of the routine assignment information 130 and the processed flag 143 of the local model structure pattern match processed flag 143 of the matrix assignment information 140, which were set in the pattern determination process. However, the control routine 10 does not release the processed flag 143 of the local model structure pattern match processed flag 143 of the matrix assignment information 140 corresponding to the matrix that was initially registered in the input layer candidate list in the first stage determination process.

[0148] Next, we will explain the confirmation process. <Confirmation process> In the confirmation process, the control routine 10 identifies the input layer and output layer in each set of candidate components, and determines whether the candidate components match the pattern by determining the corresponding positions of these configurations with each layer and each element on the pattern.

[0149] The control routine 10 selects as input layer candidates any matrix from among the input layers enumerated in the first stage and the matrix identified when identifying candidate components of other stages, the matrix of the connecting layer not included in the candidate component set of any stage. The control routine 10 also selects as output layer candidates any matrix from among the matrices output by the layers in the candidate component set, the matrix of the connected layer not included in the candidate component set.

[0150] Next, the control routine 10 performs a matching determination process to determine whether the input layer candidates match the pattern, starting from the input layer candidates. In the matching determination process, the control routine 10 first determines the candidate components by prioritizing the matching of layers whose types match the components of the corresponding stages in the pattern. For example, if the corresponding stage in the pattern contains only one component of the same type, the correspondence between the candidate component and the pattern component can be determined based solely on the type.

[0151] The control routine 10 removes the confirmed component from the candidate component set and records it in the known model structure correspondence information 150. Specifically, if the confirmed component is an input / output layer, the control routine 10 sets the matrix assignment information ID in the corresponding field, and if it is a layer, it sets the element position of that layer (the row and element number) and the routine assignment information ID.

[0152] On the other hand, if there are multiple elements of the same type in the corresponding stage of the pattern, the control routine 10 determines the position by referring to the connection relationship with the elements determined in the preceding and succeeding stages. The input and output layers are processed similarly, assuming that the pattern layers are composed of elements of the types input and output.

[0153] The control routine 10 performs these processes from the input layer to the output layer, and repeats the same process until all correspondences with candidate patterns are determined, that is, until all candidate component sets for each stage are empty.

[0154] In this process, if none of the candidates are determined after executing the processing from the input layer to the output layer once, that is, if not a single element is removed from the candidate set of components in each stage, it means that there were elements that did not correspond to the pattern, so the control routine 10 terminates the process at that point.

[0155] In this way, once all candidates have been determined, the control routine 10 sets the determination flag 152 of the known model structure correspondence information 150 and terminates the pattern matching process.

[0156] In the pattern matching process described above, candidates were determined by tracing from the input side of the model. However, it is also possible to determine each candidate by tracing from the output side of the model. In that case, the layer outputting to the selected matrix should be selected and the process should proceed. Furthermore, in the pattern matching process, the model was interpreted as a tree structure consisting of layers and matrices, but it is also possible to interpret the model as a tree structure containing only layers and determine the candidates by checking for subtree matches.

[0157] Next, we will explain the processing of the integrated layer after replacing the replaceable layer group.

[0158] For example, let's describe the processing of the integration layer corresponding to the multiplication / addition layer group shown in Figure 10(B). Here, the function of processing in this integration layer (integration function FMA) is expressed as shown in equation (9) below, where matrix A is the input layer[0], matrix B is the constants [0] input to the multiplication layer, matrix C is the constants [1] input to the addition layer, and matrix D is the output layer[0]. D = FMA(A, B, C) = ADD(MUL(A, B), C) │di=ai×bi+ci ···(9)

[0159] The integrated layer processing routine can be implemented with two instructions: a multiplication instruction and an addition instruction. However, if the CPU 101 can execute fma, a dedicated instruction that integrates multiplication and addition, the integrated layer processing routine may be implemented using a single fma instruction. By executing using fma in this way, the processing by the processing routine can be sped up.

[0160] Next, we will explain the processing of the integration layer corresponding to the matrix multiplication / addition layer group shown in Figure 13(A).

[0161] Here, the BLAS (Basic Linear Algebra Subprograms) general-purpose matrix operation library provides a function called GEMM. GEMM is a function that transposes matrices A and B according to the transpose flag, then performs their matrix product, and finally adds matrix C. The result matrix Y from the GEMM operation is expressed as shown in equation (10) below.

[0162]

number

[0163] For example, if trA=0 and trB=0, the GEMM process can be represented as shown in Figure 17.

[0164] Here, the processing of the integration layer corresponding to the matrix multiplication / addition layer group shown in Figure 13(A) can be expressed using the GEMM function as shown in equation (11) below, where input layer[0] is matrix A, constant[0] is constant matrix B, constant[1] is constant matrix C, and output layer[0] is matrix D. D=GEMM(A,B,C,1,1,0,0)=AB+C (11)

[0165] Therefore, the processing of the integrated layer corresponding to the matrix multiplication / addition layer group shown in Figure 13(A) can be created as a processing routine using GEMM.

[0166] Next, we will explain the processing of the integrated layer corresponding to the multiplication / addition / matrix stacking group shown in Figure 13(B).

[0167] Here, if we let the input layer[0] in the multiplication / addition / matrix stacking group be matrix A, the input layer[1] be matrix B, the constant[0] be constant matrix C, the constant[1] be constant matrix D, and the output layer[0] be matrix E, then matrix E can be expressed as shown in equation (12) below.

[0168] E=A ADD(MUL(B,C),D) ···(12)

[0169] Here, the processing routine in the integration layer should execute the process shown in equation (12), but if the CPU 101 can execute the fma instruction, it can be configured to execute the processing routine shown in Figure 18. In this way, executing the fma instruction can improve the processing speed of the processing routine in the integration layer.

[0170] Furthermore, the processing of the multiplication / addition / matrix stacking group becomes a matrix multiplication of input layer[0] and input layer[1] or constant[0] when all elements of input layer[1] or constant[0] are 1 and all elements of constant[1] of the addition layer are 0. Therefore, the processing routine of the integrated layer corresponding to the multiplication / addition / matrix stacking group may be a process that performs the corresponding matrix multiplication. In addition, the processing of the multiplication / addition / matrix stacking group becomes a matrix multiplication of input layer[0] and constant[1] when all elements of input layer[1] or constant[0] are 0. Therefore, the processing routine of the integrated layer corresponding to the multiplication / addition / matrix stacking group may be a process that performs the corresponding matrix multiplication. In this way, output to the internal matrix becomes unnecessary in the processing routine, and memory access can be reduced.

[0171] Next, we will explain the processing of the integration layer corresponding to the matrix transpose / addition layer group shown in Figure 13(C).

[0172] Here, if we let the input layer[0] in the matrix transpose / addition layer group be matrix A, the input layer[1] be matrix B, and the output layer[0] be matrix C, then matrix C can be expressed as shown in equation (13) below. C = ADD(A^(-1), B) ... (13) In a typical adder layer, elements are accessed linearly. However, by providing a special adder layer that allows stride access (accessing elements with a predetermined number of bits left open) to the elements of matrix A in the same order as when matrix A is transposed, the processing of the integrated layer corresponding to the matrix transpose / adder layer group can be performed. By providing such an adder layer, memory access can be reduced.

[0173] Furthermore, the processing of the matrix transpose / addition layer group involves outputting input layer [1] if all elements of input layer [0] are 0. Therefore, the integrated layer processing routine corresponding to the matrix transpose / addition layer group may be configured to output input layer [1] as is if all elements of input layer [0] are 0. Also, if all elements of input layer [1] are 0, the process involves transposing input layer [0]. Therefore, the integrated layer processing routine corresponding to the matrix transpose / addition layer group may be configured to simply transpose input layer [0]. In this way, output to the internal matrix becomes unnecessary in the processing routine, reducing memory access.

[0174] Next, we will explain the processing of the integrated layer corresponding to the subtraction / cosine / sine / multiplication layer group shown in Figure 10(D).

[0175] If the input layer[0] in the subtraction / cosine / sine / multiplication layer group is matrix A, the input layer[1] is matrix B, and the output layer[0] is matrix C, then matrix C can be expressed as shown in equation (14) below. C = MUL(SIN(SUB(A,B)),COS(B)) │ci=sin(ai-bi)×cos(bi) ···(14)

[0176] Therefore, the processing routine in the integration layer should execute the processing routine of equation (14) (the first processing routine).

[0177] Here, if all elements of matrix A are π / 2, then the following relationship (15) is satisfied.

[0178]

number

[0179] Therefore, equation (14) can be expressed as shown in equation (16) below.

[0180]

number

[0181] Therefore, if all elements of matrix A are π / 2, the processing routine in the integration layer should execute the processing routine of equation (16) (the second processing routine). This second processing routine can perform processing faster than the first processing routine.

[0182] Therefore, when performing pattern matching, if all elements of matrix A are π / 2, the second processing routine may be used; otherwise, the first processing routine may be used.

[0183] Next, we will explain the processing of the integration layer corresponding to the selection / addition layer group shown in Figure 13(D).

[0184] In a selection / addition layer group, the selection layer receives two inputs and a selection input (1x1 matrix), and selects which of the input matrices to output depending on whether the value of the selection input is 0 or 1. Here, if the input layer[0] in the selection / addition layer group is matrix A, the input layer[1] is matrix B, the input layer[2] is matrix C (1x1 matrix), the internal matrix[0] which is the output of the selection layer is matrix D, and the output layer[0] is matrix E, then matrix D can be expressed as shown in equation (17) below, and matrix E can be expressed as shown in equation (18) below.

[0185]

number

[0186] For the integration layer processing corresponding to the selection / addition layer group, it is usually sufficient to execute the processes in equations (17) and (18).

[0187] Here, if the value of matrix C is 0, then E = A + A = 2A, and we can perform multiplication. If the value of matrix C is 1, then E = A + B, and we can perform addition.

[0188] Therefore, the processing routine in the integration layer may be configured to switch between addition and multiplication depending on the value of matrix C. This eliminates the need for output to an internal matrix in the processing routine, thus reducing memory access.

[0189] Next, the replaceable layer group may be, for example, a general-purpose replaceable layer group (general-purpose replaceable layer group) as shown below.

[0190] Figure 19 illustrates the configuration of a general-purpose replaceable layer group in a modified example.

[0191] The general-purpose replaceable layer group 190 has an input layer capable of taking multiple inputs, an N-stage (where N is 2 or more) processing layer, and an output layer.

[0192] Each of the 1st to N-1 stages includes one activation function layer or Eltwise layer (a layer that performs element-wise operations such as multiplication or addition), and zero or more activation function layers, Eltwise layers, or pass-through layers (a layer that outputs the input as is). The Nth stage includes one Eltwise layer that aggregates the results of the previous stage, or one pass-through layer that outputs the result as is if only one matrix was output in the previous stage. The general-purpose replaceable layer group 190 has connection paths between the preceding and succeeding stages that allow output from any layer to any layer.

[0193] By using the general-purpose replaceable layer group 190, multiple replaceable layer groups represented by the general-purpose replaceable layer group 190 can be used as a single representation. When managing the general-purpose replaceable layer group 190 in the replaceability determination table 120, the general-purpose replaceable layer group 190 may be managed as a single pattern, or all replaceable layer groups corresponding to all combinations represented by the general-purpose replaceable layer group 190 may be managed as patterns.

[0194] In the general-purpose replaceable layer group 190 shown in Figure 19, there are many replaceable layer groups included, so for example, a general-purpose replaceable layer group 191 with a limited number of members may be used.

[0195] Figure 20 illustrates the configuration of an example of a general-purpose replaceable layer group related to a modified example.

[0196] The general-purpose replaceable layer group 191 is defined as having N=3 in the general-purpose replaceable layer group 190 shown in Figure 19, with the first stage consisting of 3 layers and the second stage consisting of 2 layers, and there is no connection path between the third layer of the first stage and the first layer of the second stage.

[0197] Figure 21 illustrates the configuration of a replaceable layer group constructed using a modified general-purpose replaceable layer group.

[0198] Using the general-purpose replaceable layer group 191, the replaceable layer groups shown in Figures 10(A), 10(B), and 10(C) can be represented as shown in Figures 21(A), 21(B), and 21(C), respectively.

[0199] Next, we will describe the arithmetic circuit used when the processing of the integrated layer corresponding to the general-purpose replaceable layer group 190 is implemented using a dedicated arithmetic circuit.

[0200] Figure 22 illustrates an example of an arithmetic circuit that realizes an integrated layer corresponding to a general-purpose replaceable layer group in a modified example.

[0201] The arithmetic circuit 220 includes, as each of the 1st to N-1 stages, one circuit element capable of selectively executing a process corresponding to an activation function layer or an Eltwise layer, and M-1 circuit elements (where M is the maximum number assumed for one stage) capable of selectively executing a process corresponding to an activation function layer, an Eltwise layer, or a pass-through layer. The Nth stage includes one circuit element capable of selectively executing either an Eltwise layer that aggregates the results of the previous stage, or a pass-through layer that outputs the result as is if only one matrix was output in the previous stage. The arithmetic circuit 202 has connection paths that allow for the selective connection of each circuit element between the preceding and succeeding stages.

[0202] When using this arithmetic circuit 220, if the integrated layer processing corresponding to a specific replaceable layer group (specific replaceable layer group) represented by the general-purpose replaceable layer group 190 is executed, the control routine 10 selects and executes the processing of each circuit element of the arithmetic circuit 220 that corresponds to each layer of the specific replaceable layer group, and sets the connection circuits between the circuit elements to match the connection relationships of each layer in the specific replaceable layer group. This makes it possible to implement the integrated layer processing corresponding to the replaceable layer group represented by the general-purpose replaceable layer group 190 using the arithmetic circuit 220.

[0203] Furthermore, when executing the processing of the integrated layer corresponding to the general-purpose replaceable layer group 190 using a processing routine (program), a general-purpose processing routine that can set the connection relationships between program elements, along with program elements that perform the same processing as each circuit element in the arithmetic circuit 220, may be prepared in advance, and the integrated layer processing routine may be configured by setting this processing routine to correspond to the configuration of each layer and the connection relationships of each layer in the specific replaceable layer group. This makes it possible to realize the processing of the integrated layer corresponding to the replaceable layer group represented by the general-purpose replaceable layer group 190 with a single processing routine. Alternatively, routine components corresponding to the configuration of each layer may be prepared, and the routine components corresponding to each layer in the specific replaceable layer group may be combined to correspond to the connection relationships of each layer to generate and execute a processing routine for the specific replaceable layer group.

[0204] In the embodiment described above, an execution procedure list was created in which the replaceable layer groups in the trained model data were replaced with integrated layers before performing the inference operation. However, as shown below, the execution procedure may be created in which the replaceable layer groups were dynamically replaced with integrated layers immediately before performing the inference operation for each layer.

[0205] Figure 23 is a flowchart of the inference process related to the modified example. Steps similar to those in the inference process shown in Figure 8 are denoted by the same reference numerals.

[0206] In step S16, the control routine 10 starts executing the process according to the execution procedure list 80. Specifically, the control routine 10 reads the data to be input to the trained model (model input data) from, for example, the auxiliary storage device 106 and stores it in the work area 20.

[0207] Next, the control routine 10 causes the next processing layer in the execution procedure list at that time to execute the processing of loop 2 (S31, S32, S17).

[0208] First, the control routine 10 performs a local model pattern matching process (S31) to detect whether there is a structure that matches the pattern of the replaceable layer group in the unprocessed processing layer containing the next processing layer to be executed in the trained model data, that is, whether there is a replaceable layer group in the unprocessed processing layer. Step S31 differs from the pattern matching process in step S14 in that it targets unprocessed processing layers connected to the next unprocessed processing layer to be executed, and that already executed processing layers are processed as layers into which constants are input because the matrix is ​​actually determined, but otherwise the processing can be the same. The range of unprocessed layers to be pattern matched may be limited to a predetermined number (N) of layers immediately preceding the next processing layer to be executed. By limiting it in this way, the processing load can be reduced.

[0209] Next, if the control routine 10 detects in step S31 that there is a replaceable layer group, it executes an assigned routine replacement process in which it replaces the replaceable layer group in the execution procedure list with the integration layer corresponding to this replaceable group (S32). If there is no replaceable layer group, the control routine 10 proceeds to step S17.

[0210] In the assigned routine replacement process, the control routine 10 creates an execution procedure list by replacing the procedures for the replaceable layer groups in the execution procedure list with the procedures for the integration layer corresponding to the replaceable layer groups, based on the detected replaceable layer groups.

[0211] After step S32, the control routine 10 proceeds to step S17.

[0212] Control routine 10 executes the processing of loop 2 if there is a next unprocessed processing layer in the execution procedure list, and exits loop 2 if there is no next unprocessed processing layer in the execution procedure list.

[0213] According to the inference process described above, in step S31, processing layers that have already been executed can be processed as layers that input constants. Therefore, in the inference process shown in Figure 8, groups of processing layers that could not be determined to be a replaceable layer group may also be determined to be a replaceable layer group, allowing for more effective replacement of multiple processing layers with an integrated layer. This makes it possible to more effectively reduce the number of processing layers in the execution of the trained model. Consequently, the number of memory accesses can be reduced more effectively.

[0214] It should be noted that the present invention is not limited to the embodiments and modifications described above, and can be implemented with appropriate modifications without departing from the spirit of the invention.

[0215] For example, in the above embodiment, the inference device converted the replaceable layer group of the trained model into an integrated layer. However, the present invention is not limited to this. For example, in a processing device that does not perform inference using a trained model, the replaceable layer group in the trained model data may be converted into an integrated layer to create a converted trained model (an example of subject-defined information). In this case, the converted trained model is sent to the inference device, and the inference device can perform inference processing similar to the above-described inference processing by performing inference processing using the converted trained model. In this case, the inference device must be able to execute processing routines corresponding to the integrated layer.

[0216] Furthermore, although the above embodiment showed an example in which the processing routine is executed by the CPU, at least one processing routine may be executed by another processor such as a GPU. [Explanation of Symbols]

[0217] 10...Control routine, 20...Work area, 31...Convolutional layer routine, 32...Pooling layer routine, 33...ReLU activation function layer routine, 34...Adder layer routine, 35...Multiplier layer routine, 40...Integration layer routine, 100...Inference unit, 101...CPU, 102...Main memory, 108...Display device, 111...Recording medium

Claims

1. A processing apparatus for processing a trained model comprising a plurality of processing layers, A receiving unit that receives input from the aforementioned trained model, A generation unit identifies a set of processing layers from the aforementioned trained model that can be converted into an equivalent layer with fewer layers, and generates execution definition information that defines the execution of the processing layers so that the identified set of processing layers is executed as the equivalent layer. It comprises an inference unit capable of executing the trained model based on the execution specification information, The generation unit determines, when the inference unit executes a processing layer included in the trained model, whether a set of processing layers to be executed after the next processing layer can be converted into an equivalent layer, and generates the execution specification information to execute the set of processing layers that can be converted as an equivalent layer. The inference unit executes the equivalent layer by converting it into the multiple processing layers that can be converted, based on the execution specification information. Processing device.

2. When executing the processing layer, the generation unit considers the processing results of the processing layer that has already been executed, determines whether a set of processing layers to be executed after the next processing layer can be converted into an equivalent layer, and generates the execution specification information to execute the convertible set of processing layers as an equivalent layer. The apparatus according to claim 1.

3. The equivalent layer is a processing layer equivalent to a set of processing layers including at least one ReLU activation function layer or hyperbolic tangent function layer and at least one multiplication layer or addition layer. The apparatus according to claim 1.

4. The inference unit is comprised of a processor, The equivalent layer is a processing layer equivalent to a set of processing layers that execute processing that the processor can perform with a single instruction. The apparatus according to claim 1.

5. A processing apparatus for processing a trained model comprising a plurality of processing layers, A receiving unit that receives input from the aforementioned trained model, A generation unit identifies a set of processing layers from the aforementioned trained model that can be converted into an equivalent layer with fewer layers, and generates execution definition information that defines the execution of the processing layers so that the identified set of processing layers is executed as the equivalent layer. It comprises an inference unit capable of executing the trained model based on the execution specification information, The inference unit has an arithmetic circuit capable of selecting and executing processing of multiple equivalent layers, The aforementioned arithmetic circuit has multiple stages of circuit elements that can select and execute any one of the processing steps of two or more processing layers according to the settings, and the connection relationships of the circuit elements in each stage can be selected and set. The inference unit sets the processing to be performed by the circuit elements in the arithmetic circuit and the connection relationships of the circuit elements in each stage, according to the equivalent layer processing to be performed, and causes the arithmetic circuit to perform the equivalent layer processing. Processing device.

6. The aforementioned trained model is composed of a combination of processing layers that are provided in advance according to a predetermined standard. The equivalent layer is a processing layer that performs processing not provided in the predetermined standard. The apparatus according to claim 1.

7. The system has a storage unit that stores configuration information indicating the configuration of a set of multiple processing layers that can be converted to the equivalent layer, The generation unit identifies a set of processing layers that can be converted into an equivalent layer in the trained model, based on the configuration information. The apparatus according to claim 1.

8. The generation unit determines whether a set of processing layers can be converted to the equivalent layer by performing a determination process that includes a determination condition for whether or not the set of processing layers can be converted to the equivalent layer. The apparatus according to claim 1.

9. A method for processing a trained model using a processing unit that processes a trained model including multiple processing layers, The trained model accepts input, From the aforementioned trained model, identify a set of processing layers that can be converted into an equivalent layer with fewer layers, and generate execution definition information that defines the execution of the processing layers so that the identified set of processing layers is executed as the equivalent layer. Based on the execution rules information, the trained model is executed. When executing the processing layers included in the trained model, it is determined whether a set of processing layers to be executed after the next processing layer can be converted into an equivalent layer, and the execution specification information is generated to execute the convertible set of processing layers as an equivalent layer. Based on the execution specification information, the equivalent layer is executed in place of the multiple convertible processing layers. Methods for processing pre-trained models.

10. A trained model processing program that causes a computer to perform processing on a trained model that includes multiple processing layers, The aforementioned computer, A receiving unit that receives input from the aforementioned trained model, A generation unit identifies a set of processing layers from the aforementioned trained model that can be converted into an equivalent layer with fewer layers, and generates execution definition information that defines the execution of the processing layers so that the identified set of processing layers is executed as the equivalent layer. Based on the execution specification information, the trained model is made to function as an inference unit capable of execution. The generation unit determines whether a set of processing layers to be executed after the next processing layer when the inference unit executes the processing layers included in the trained model can be converted into an equivalent layer, and generates the execution specification information to execute the set of convertible processing layers as an equivalent layer. Based on the execution specification information, the inference unit is transformed into a set of convertible processing layers, and the equivalent layer is executed. A pre-trained model processing program that makes it function in this way.

Images