Processing device, processing method and processing program
By implementing in-place processing regulations in a processing device for neural network models, the work area required for processing is reduced, addressing the challenge of increasing memory demands with complex models.
Patent Information
- Application Number
- JP2023173215
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-10-04
- Publication Date
- 2025-04-16
AI Technical Summary
As the number of layers in neural network models increases, the required work area and memory capacity also increase, limiting the number of devices that can perform processing and increasing costs.
A processing device that includes an accepting unit for processing structures with multiple layers and a generating unit that creates execution regulations to enable in-place processing, where layers that can process input and output matrices using the same memory area are executed last and output matrices are written back to the input matrix memory area.
This approach effectively reduces the work area required for processing, allowing more devices to perform the processing without the need for increased memory capacity, thereby reducing costs.
Smart Images
Figure 2025063763000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for processing a processing structure such as a trained model that includes multiple processing layers. [Background technology]
[0002] In recent years, neural networks, a type of machine learning, have been used to recognize and classify images, characters, etc. Neural networks are composed of various layers, such as convolution layers, pooling layers, and activation function layers, and a model structure is defined by combining these. Since a defined model cannot perform recognition as is, it is first necessary to set (train) the parameters of each layer using input data for learning and label data, etc., to obtain a trained model. Once a trained model has been obtained, the data to be recognized can then be input into the trained model and each layer can be calculated (inferred) to obtain recognition results.
[0003] An inference device that performs inference using a trained model generally includes a processing routine or circuit block (hereinafter, sometimes collectively referred to as a routine) that executes processing corresponding to each layer, and a work area that temporarily stores the results. For example, when performing inference using a neural network that is composed of three layers, a convolution layer A, a pooling layer B, and an activation function layer C, the inference device performs the following processing: input data is provided to the convolution layer routine, the calculation result Ao of the convolution layer is stored in the work area, the calculation result Ao is input to the pooling layer routine, the calculation result Bo is stored in the work area, and the calculation result Bo is input to the activation function layer routine to obtain the final result.
[0004] The structure of neural networks is becoming more complex, and a wide variety of layers have been proposed and implemented, including a layer that updates part of the input matrix and outputs it.
[0005] In addition, inference systems include those that support in-place processing in which an input matrix is input to a single layer and the input matrix and output matrix are the same size, in which the input matrix is directly rewritten.
[0006] Here, a technique for the in-place method in a general program is disclosed in, for example, Non-Patent Document 1.
[0007] The ONNX model format is one of the formats for expressing the structure of a neural network model. The ONNX model format is a general-purpose model format that can be used by multiple inference devices and programs, and specifies the file structure and specifications for each layer. Currently, there are more than 150 types of layers that can be used in the ONNX model format. [Prior art documents] [Non-patent literature]
[0008] [Non-Patent Document 1] Ni Yongmao, "Algorithm Research Series - In-Place Method", Utsunomiya University Faculty of International Studies Research Papers, 2020, No. 50, pp. 101-107 Summary of the Invention [Problem to be solved by the invention]
[0009] When performing processing using a trained model composed of multiple layers, memory work areas are generally allocated to each of the input matrix and output matrix. As the number of layers increases, the required work area becomes larger, and the required memory capacity also increases.
[0010] For example, when an input matrix is input to a single layer, the work area can be reduced by the in-place method as described above. However, if there are only limited cases in a model where an input matrix is input to only a single layer, the effect of reducing the work area is small.
[0011] If the work area required for processing using a trained model becomes large, the number of devices that can execute the processing will be limited. On the other hand, increasing the memory capacity of the device to increase the work area will increase costs.
[0012] Furthermore, similar problems arise not only in processing using a trained model, but also in processing using a program structure composed of multiple layers.
[0013] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a technique capable of effectively reducing the work area used in processing. [Means for solving the problem]
[0014] In order to achieve the above-mentioned object, a processing device according to one aspect is a processing device that processes a processing structure including a plurality of processing layers, and has a receiving unit that receives input of the processing structure, and a generation unit that identifies, from the processing structure, an in-place processable layer that is a processing layer that can process a first input matrix among input matrices in the processing layer and a first output matrix among output matrices in the processing layer using the same memory area, and when there are a plurality of common input processing layers that are processing layers that use the first input matrix, executes the in-place processable layer last among the plurality of common input processing layers, and generates execution specification information that specifies that the first output matrix is output to the memory area of the first input matrix in the processing of the in-place processable layer. Effect of the Invention
[0015] According to the present invention, the work area used in processing can be effectively reduced. [Brief description of the drawings]
[0016] [Figure 1] FIG. 1 is a configuration diagram of an inference device according to an embodiment. [Diagram 2]FIG. 2 is a diagram illustrating the functional configuration of the inference device according to an embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an update layer routine according to one embodiment. [Figure 4] FIG. 4 is a diagram illustrating a configuration of a trained model according to an embodiment. [Diagram 5] FIG. 5 is a diagram illustrating a format of trained model data according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of trained model data according to an embodiment. [Figure 7] FIG. 7 is a configuration diagram of routine allocation information according to an embodiment. [Figure 8] FIG. 8 is a configuration diagram of matrix allocation information according to an embodiment. [Figure 9] FIG. 9 is a configuration diagram of layer specification information according to an embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of layer specification information according to an embodiment. [Figure 11] FIG. 11 is a flowchart of an inference control process according to one embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of an execution procedure list in an initial trained model. [Figure 13] FIG. 13 is a diagram showing an example of an execution procedure list after the in-place inference process. [Figure 14] FIG. 14 is a flow diagram of a layer process according to one embodiment. [Figure 15] FIG. 15 is a flowchart of a model generation process according to a modified example. [Figure 16] FIG. 16 is a diagram illustrating an example of trained model data generated according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] The following embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention, and all of the elements and combinations thereof described in the embodiments are not necessarily essential to the solution of the invention.
[0018] FIG. 1 is a configuration diagram of an inference device according to an embodiment.
[0019] The inference device 100 is an example of a processing device, and is configured by a computer such as a PC (Personal Computer) or a server. The inference device 100 includes, for example, a Central Processing Unit (CPU) 101, a main memory 102, a Graphics Processing Unit (GPU) 103, a reader / writer 104, a communication interface (communication I / F) 105, an auxiliary storage device 106, an input / output interface (input / output I / F) 107, a display device 108, and an input device 109. The CPU 101, the main memory 102, the GPU 103, the reader / writer 104, the communication I / F 105, the auxiliary storage device 106, the input / output I / F 107, and the display device 108 are connected via a bus 110.
[0020] The CPU 101 is an example of a processor, and reads out a processing program stored in the auxiliary storage device 106 into the main memory 102 and executes it.
[0021] The main memory 102 is an example of a storage unit, such as a RAM or a ROM, and stores programs (processing programs, etc.) executed by the CPU 101 and various information. The auxiliary storage device 106 is an example of a storage unit, such as a non-transitory storage device (non-volatile storage device), such as a Hard Disk Drive (HDD) or a Solid State Drive (SSD), and stores programs executed by the CPU 101 and various information.
[0022] The GPU 103 is a processor suitable for executing a specific process such as image processing, and is suitable for executing processes performed in parallel. In this embodiment, the GPU 103 executes a predetermined process according to an instruction from the CPU 101.
[0023] The reader / writer 104 is detachably attached to the recording medium 111, and reads data from the recording medium 111 and writes data to the recording medium 111. The recording medium 111 may be, for example, a non-temporary recording medium (non-volatile recording medium) such as an SD memory card, an FD (floppy disk: registered trademark), a CD, a DVD, a BD (registered trademark), or a flash memory. In this embodiment, a processing program may be stored in the recording medium 111, and the reader / writer 104 may read and use the processing program. Also, the recording medium 111 may store trained model data or input data to be input to the trained model, and the reader / writer 104 may read and use the processing program. Also, the reader / writer 104 may store output data output from the trained model in the recording medium 111.
[0024] The communication I / F 105 is connected to a network, and transmits and receives data to and from other devices connected to the network.
[0025] The input / output I / F 107 is connected to an input device 109 such as a mouse or a keyboard.
[0026] The display device 108 is, for example, a display device such as a liquid crystal display, and displays and outputs various information.
[0027] Next, the functional configuration of inference device 100 will be described.
[0028] FIG. 2 is a diagram illustrating the functional configuration of the inference device according to an embodiment.
[0029] The inference device 100 stores, for example, a control routine 10, a plurality of processing routines (such as a convolutional layer routine 31, a pooling layer routine 32, a ReLU (Rectified Linear Unit) activation function layer routine 33, an addition layer routine 34, and an update layer routine 35) capable of executing processing of a processing layer (also simply referred to as a layer) of a trained model (processing structure), and an inference instruction routine 50 in a main memory 102 and / or an auxiliary storage device 106. In this embodiment, in the inference device 100, each processing routine is executed by a CPU 101. At least one of the processing routines may be realized by a dedicated arithmetic circuit.
[0030] The control routine 10 is executed by the CPU 101 to read the input data 52 into the work area 20. The control routine 10 is also executed by the CPU 101 to control the execution of an inference process according to the trained model data 51 to be executed. The control routine 10 is also executed by the CPU 101 to output a processing result by the model in the work area 20 as output data 53. Here, the work area 20 may be, for example, a part of the area of the main memory 102, or may be an area of the SRAM if an SRAM is provided in the CPU 101. Here, the generation unit, allocation unit, and aggregation / allocation unit described in the claims are configured by the CPU 101 executing the control routine 10.
[0031] A processing routine, when executed by the CPU 101, loads data to be used for processing into the CPU 101, executes processing using the loaded data, and stores the processing results by the CPU 101 in the work area 20. A processing routine is a routine that executes processing of a processing layer that can be used in the specifications of an input trained model.
[0032] The convolution layer routine 31 is executed by the CPU 101 to perform convolution processing using each of a plurality of filter data. The pooling layer routine 32 is executed by the CPU 101 to perform downsampling processing on input data. The ReLU activation function layer routine 33 is executed by the CPU 101 to convert input data using a ReLU activation function (ramp function). The addition layer routine 34 is executed by the CPU 101 to perform addition operations on input data. The update layer routine 35 is executed by the CPU 101 to perform update processing on input data.
[0033] The inference instruction routine 50 receives various instructions related to the inference processing to be executed from the input device 109 or an external device, and passes them to the control routine 10. The received instructions include information indicating the trained model to be used, the input data to be used, etc. Here, the receiving unit described in the claims is configured by the CPU 101 executing the inference instruction routine 50.
[0034] FIG. 3 is a diagram illustrating an update layer routine according to one embodiment.
[0035] The update layer routine 35 is executed by the CPU 101 to configure the update layer 35a. The update layer 35a receives as input an input matrix to be updated and a submatrix for updating a part of the input matrix, and replaces the input matrix with the submatrix for a range starting from a specified position of the input matrix according to a specified position (parameter) indicating the update location of the input matrix, and outputs the result as an output matrix. In the example of FIG. 3, the input matrix to be updated is a two-dimensional matrix, but the input matrix may be a one-dimensional matrix or a three- or higher-dimensional matrix. The update layer 35a can be realized, for example, by simplifying the Scatter layer of ONIX.
[0036] FIG. 4 is a diagram illustrating a configuration of a trained model according to an embodiment.
[0037] The trained model 51A has input data 52-1 (tensor_i1), input data 52-2 (tensor_i2), a constant 61 (weight_1), a constant 62 (weight_2), a convolutional layer 63 (node_1), a constant 64 (weight_3), a convolutional layer 65 (node_2), a ReLU activation function layer 66 (node_3), a constant 67 (weight_4), a ReLU activation function layer 68 (node_4), an update layer 69 (node_5), a multiplication layer 70 (node_6), a multiplication layer 71 (node_7), an update layer 72 (node_8), a pooling layer 73 (node_9), and an addition layer 74 (node_10).
[0038] The convolution layer 63 receives the input data 52-1 (tensor_i1) and a constant 61 (weight_1), executes a convolution process, and outputs data (tensor_1).
[0039] The convolution layer 65 receives the input data 52-2 (tensor_i2) and a constant 62 (weight_2), executes a convolution process, and outputs data (tensor_2).
[0040] The ReLU activation function layer 66 inputs the data (tensor_1) output from the convolution layer 63, executes processing using the ReLU activation function, and outputs data (tensor_3).
[0041] The ReLU activation function layer 68 inputs the data (tensor_2) output from the convolution layer 65, executes processing using the ReLU activation function, and outputs data (tensor_4).
[0042] The update layer 69 receives the data (tensor_3) output from the ReLU activation function layer 66 and a constant 64 (weight_3), executes an update process, and outputs data (tensor_5).
[0043] The multiplication layer 70 inputs the data (tensor_4) output from the ReLU activation function layer 68 and the data (tensor_2) output from the convolution layer 65, performs a multiplication process, and outputs data (tensor_6).
[0044] The multiplication layer 71 receives the data (tensor_5) output from the update layer 69 and the data (tensor_6) output from the multiplication layer 70, executes a multiplication process, and outputs data (tensor_7).
[0045] The update layer 72 receives a constant 67 (weight_4) and the data (tensor_6) output from the multiplication layer 70, executes an update process, and outputs data (tensor_8).
[0046] The pooling layer 73 receives the data (tensor_6) output from the multiplication layer 70, performs downsampling processing, and outputs data (tensor_9).
[0047] The addition layer 74 inputs the data (tensor_7) output from the multiplication layer 71, the data (tensor_8) output from the update layer 72, and the data (tensor_9) output from the pooling layer 73, performs an addition process, and outputs output data 53 (tensor_o).
[0048] FIG. 5 is a diagram illustrating a format of trained model data according to an embodiment.
[0049] Data showing the structure of a trained model (trained model data) is expressed as a graph of the entire model. The graph includes the following elements: Input, which indicates the input data for the entire model; Output, which indicates the output data; Node, which indicates each layer (node); Tensor, which indicates the matrix to be used; and Data, which indicates constant data. The names of the matrices indicating the data are entered in Output and Input.
[0050] A Node includes the following elements: Input, which indicates the input data for the layer, Output, which indicates the output data for the layer, Weight, which indicates the weight (constant), Type, which indicates the type of the layer, Name, which indicates the name of the layer, and parameters and their values in the layer. Note that, depending on the type of layer, other elements may be included.
[0051] Output, Input, and Weight describe the matrix names (including constant names) that indicate the corresponding data in the layer. Type describes the type of layer, for example, Convolution if the layer is a convolution layer, ReLU if the ReLU activation function layer, Mul if the multiplication layer, Add if the addition layer, Pool if the pooling layer, and Update if the update layer. Name describes an identifiable name in the trained model (layer name, node name).
[0052] Tensor includes elements Name and Shape, which indicate the matrix (tensor) used in the trained model. Name stores the name of the matrix. Shape stores an array of matrix shapes. The array of matrix shapes is, for example, information on dimensions and sizes, and for a 3×3 quadratic matrix, it is (2,3,3), and for a 3×2×4 three-dimensional matrix, it is (3,3,2,4).
[0053] Data includes elements of Name, which indicates the constants (including the weights of the constants) used in the trained model, and Array, which indicates the array of the values of the constants (constant matrix). Name stores the name of the matrix indicating the constants.
[0054] Fig. 6 is a diagram illustrating an example of trained model data according to an embodiment. Fig. 6 illustrates trained model data 51B described in the format illustrated in Fig. 5 for the trained model illustrated in Fig. 4.
[0055] For example, description portion 51Ba for update layer 72 in trained model data 51B describes that the node name is node_8, the type is Update, the input is tensor_6, the constant is weight_4, the specified position (Position) is [4,0,1,0,0], and the output is tensor_8.
[0056] According to this trained model data 51B, the structure of the trained model 51A shown in FIG. 4 can be appropriately expressed.
[0057] Next, a description will be given of information used in the processing of the control routine 10. The control routine 10 uses routine allocation information 130, matrix allocation information 140, and layer specification information 150. The routine allocation information 130, matrix allocation information 140, and layer specification information 150 are stored in, for example, the main memory 102 and used.
[0058] FIG. 7 is a configuration diagram of routine allocation information according to an embodiment.
[0059] The routine assignment information 130 is set for each layer of the trained model. The routine assignment information 130 includes fields for a routine assignment information ID 131, model layer information 132, a routine name 133, an input assignment information array 134, an output assignment information array 135, an in-place inference possible flag 136, and an inferred flag 137.
[0060] The routine assignment information ID 131 stores an ID (routine assignment information ID) for identifying the routine assignment information 130. The model layer information 132 stores information (e.g., a node name) for identifying a layer in a trained model associated with a routine. The routine name 133 stores a routine name associated with a layer. The input assignment information array 134 stores one or more IDs (matrix assignment information IDs) assigned to information (matrix) input to a routine corresponding to a layer. The output assignment information array 135 stores one or more IDs (matrix assignment information IDs) assigned to information output from a routine corresponding to a layer. The in-place inference possible flag 136 stores a flag (in-place inference possible flag) indicating that in-place inference processing is possible for the layer corresponding to the routine assignment information 130. Here, the information stored in the in-place inference possible flag 136 (whether or not the in-place inference possible flag is stored) is an example of in-place processing possible information. The inference completion flag 137 stores a flag indicating that inference processing has been performed for the layer corresponding to the routine assignment information ID 131 in inference processing according to the model.
[0061] Next, the matrix assignment information ID 140 will be described.
[0062] FIG. 8 is a configuration diagram of matrix allocation information according to an embodiment.
[0063] Matrix allocation information 140 is information provided corresponding to the matrices used in the model, and includes fields for matrix allocation information ID 141 , matrix name 142 , in-place sharing flag 143 , storage location information 144 , and matrix size 145 .
[0064] Matrix allocation information ID 141 stores the ID (matrix allocation information ID) of the matrix corresponding to matrix allocation information 140. Matrix name 142 stores the matrix name specified in the model of the corresponding matrix. In-place sharing flag 143 stores a flag (in-place sharing flag) indicating that the work area (memory area) of the corresponding matrix is shared with other matrices. Storage location information 144 stores information indicating the storage location (memory area) of the corresponding matrix on work area 20. Matrix size 145 stores the size of the corresponding matrix.
[0065] Next, the layer specification information 150 will be described.
[0066] FIG. 9 is a configuration diagram of layer specification information according to an embodiment.
[0067] The layer specification information 150 is information provided corresponding to a layer that can be used in the model. The layer specification information 150 includes fields for a layer type name 151, an input matrix number 152, a weight matrix number 153, an output matrix number 154, an in-place input / output pair 155, and a parameter list 156. The layer type name 151 stores a name (identifier) indicating a type of a layer. The input matrix number 152 stores information on the number of matrices input to the corresponding layer. The weight matrix number 153 stores information on the number of weight matrices input to the corresponding layer. The output matrix number 154 stores information on the number of matrices output from the corresponding layer. The information on the number of matrices stored in the input matrix number 152, the weight matrix number 153, and the output matrix number 154 may be a specific number or may be information on a range of the number of matrices (for example, a predetermined number or more). The in-place input / output pair 155 stores information indicating a pair of an input matrix (first input matrix) and an output matrix (first output matrix) that can be in-placed. For example, when there are two input matrices and one output matrix, if the second input matrix and the output matrix can be processed in-place, that is, if the output matrix can be written to the memory area of the second input matrix, (2,1) is set in in-place input / output pair 155. Parameter list 156 stores the names of parameters (parameter names) that can be accepted in the corresponding layer.
[0068] Depending on the model format, the layer specification information 150 may include information other than the above. Also, although an example has been shown in which the layer specification information 150 is stored in the main memory 102, the layer specification information 150 may be hard-coded into the processing program, or may be realized on a circuit.
[0069] Fig. 10 is a diagram illustrating an example of layer specification information according to an embodiment. Fig. 10 illustrates layer specification information 150A described in the format illustrated in Fig. 9 for a layer related to the trained model illustrated in Fig. 4.
[0070] Layer specification information 150A indicates, for example, that a layer (update layer) whose layer type name is Update has 1 input matrix number, 1 weight matrix number, 1 output matrix number, the in-place input / output pair is (1,1), and the parameter list is Position.
[0071] Next, the processing by inference device 100 will be described.
[0072] FIG. 11 is a flowchart of an inference control process according to one embodiment.
[0073] The control routine 10 of the inference device 100 (more precisely, the CPU 101 that executes the control routine 10) reads the trained model data 51 (S11).
[0074] Next, the control routine 10 performs a model structure analysis process (S12) to analyze the model structure based on the trained model data 51. Here, in the model structure analysis process, based on the trained model data 51, the connection relationship of each layer in the trained model as shown in FIG. 4 and the relationship of input and output matrices for each layer are identified.
[0075] Next, the control routine 10 executes a routine assignment process to assign a corresponding routine to a layer defined in the trained model data (S13).
[0076] In the routine allocation process, the control routine 10 creates matrix allocation information 140 for each matrix (here, input data and constants are also included in the matrix) of the trained model data. At this time, the control routine 10 assigns a new matrix allocation information ID to the matrix allocation information ID 141, and stores the matrix name of the matrix (Tensor, Data) in the model data in the matrix name 142. Furthermore, the control routine 10 stores information on the storage location in the work area 20 where the matrix is stored in the storage location information 144, and stores the size of the matrix in the matrix size 145. Note that the in-place sharing flag 143 is left unset at this point.
[0077] Next, the control routine 10 generates routine assignment information 130 for each layer of the trained model data 51. At this time, the control routine 10 assigns a new routine assignment ID to the routine assignment information ID 131, stores information identifying a layer in the model in the model layer information 132, and sets the name of the corresponding processing routine to the routine name 133. Furthermore, the control routine 10 stores, in the input assignment information array 134 and the output assignment information array 135, the matrix assignment information ID of the matrix assignment information 140 assigned to the input matrix and output matrix of the corresponding layer in the model data. Note that the in-place inference possible flag 136 and the inference completed flag 137 are left unset at this point.
[0078] Next, the control routine 10 uses the routine allocation information 130 to determine procedures such as calling the routine, reading data used in the routine from the work area 20 to the CPU 101, and storing it from the CPU 101 to the work area 20, generates an execution procedure list 80 (see Figure 12), and reserves the necessary area on the work area 20 (input matrix, output matrix, constant matrix, area for the routine, etc.).
[0079] Fig. 12 is a diagram showing an example of an execution procedure list for an initial trained model. In Fig. 12, lines beginning with # are comments, and the same applies to Fig. 13 described later.
[0080] Execution procedure list 80 is a list of execution procedures when the trained model data 51B shown in Fig. 6 is executed as is. As shown in execution procedure list 80, for each layer included in the trained model, input data used in the processing routine is read (loaded), a processing routine that executes the processing of the layer is called (loaded), and output data is written (stored). When processing is performed according to execution procedure list 80, the processing routine is called 10 times, the work area 20 is loaded and stored 28 times, and the area required to store matrices in the work area 20 is 16, the same as the number of matrices used in processing.
[0081] Returning to the explanation of Figure 11, the control routine 10 then identifies input matrices and output matrices in the trained model data that can be processed using the same area of the work area 20, and performs an in-place inference application process that assigns the same area to these input matrices and output matrices (S14).
[0082] In the in-place inference application process, the control routine 10 refers to each routine allocation information 130, acquires the layer type of the corresponding layer on the model from the model layer information 132, and judges whether or not the layer of this layer type is compatible with in-place processing by itself by referring to the layer specification information 150 according to the layer type. Here, whether or not the layer is compatible with in-place processing can be judged by whether or not a pair is set in the in-place input / output pair 155 of the layer specification information 150. Note that the layer type name and information indicating whether or not the layer is compatible with in-place processing may be managed in association with each other, and whether or not the layer is compatible with in-place processing may be judged based on the layer type name. Alternatively, whether or not the layer is compatible with in-place processing may be judged by whether or not the input matrix and the output matrix in the layer corresponding to the layer type name are the same in size.
[0083] As a result, if it is determined that the layer of this layer type is compatible with in-place processing, the control routine 10 sets a flag indicating that this layer is compatible with in-place processing (in-place inference compatible flag) to the in-place inference compatible flag 136 of the determined routine allocation information 130.
[0084] Here, for example, when in-place processing is performed on a layer determined to support in-place processing, if the input matrix to be in-placed is input to multiple layers, it may affect other layers and prevent appropriate processing. Therefore, it is conceivable to perform in-place processing only when the input matrix to be in-placed is not input to other layers. In this case, the number of layers that can perform in-place processing is limited, and the effect of reducing the work area used is limited.
[0085] In contrast, in this embodiment, even when an input matrix to be in-placed is input to multiple layers, the following characteristic processing is performed to make it possible to execute in-place processing as much as possible. This processing is explained below.
[0086] First, the control routine 10 performs the processability determination process shown below for each piece of matrix allocation information 140 as a processing target (referred to as target matrix allocation information).
[0087] The control routine 10 lists the routine assignment information 130 for the layer (common input processing layer) that refers to (uses as input) the matrix corresponding to the target matrix assignment information. Next, the control routine 10 identifies the routine assignment information 130 that satisfies the conditions that the in-place inference possible flag is set and the input number of the in-place input / output pair in the layer specification information 150 corresponding to the layer indicated by the routine assignment information 130 matches the number in the input assignment information array 134 of the routine assignment information 130 for the matrix of the target matrix assignment information, and adds the routine assignment information ID of the identified routine assignment information 130 to the in-place target set created in the work area. On the other hand, the control routine 10 adds the routine assignment information ID of the routine assignment information 130 that does not meet the above condition to the in-place non-target set created in the work area.
[0088] Next, if there are no elements in the in-place target set, there is no need to perform in-place processing, and the control routine 10 ends the processing of this target matrix allocation information. Also, if there is one element in the in-place target set and no elements in the in-place non-target set, in-place processing is possible for the layer of the in-place target set (layer available for in-place processing), and so ends the processing of this target matrix allocation information.
[0089] On the other hand, if there is one or more elements in the in-place target set and one or more elements in the non-in-place target set, in-place processing may be possible for one of the layers of the in-place target set depending on the execution order of the layers, so the following detailed judgment process is performed.
[0090] First, the control routine 10 sets one layer of the elements of the in-place target set as the in-place judgment target layer. If there are multiple elements in the in-place target set, the following process is performed on the layers other than the in-place judgment target layer as layers of the in-place non-target set.
[0091] Next, the control routine 10 judges whether the in-place judgment target layer is reached by tracing the input in order for each layer of the non-in-place target set. Here, if the in-place judgment target layer is reached by tracing the input in order, this layer needs to be processed using the processing result of the in-place judgment target layer, which means that the in-place judgment target layer needs to be executed before this layer. If the in-place judgment target layer is executed before this layer, the subsequent processing will not be executed properly if in-place processing is executed. Therefore, this means that in-place processing cannot be executed for the in-place judgment target layer.
[0092] Therefore, when the input is traced in order and the in-place judgment target layer is reached, the control routine 10 clears the in-place inference possible flag in the routine allocation information 130 corresponding to the in-place judgment target layer, and ends the detailed judgment process.
[0093] On the other hand, if the in-place judgment target layer is not reached by tracing the input in order, the control routine 10 moves the procedures (reading input data, reading routines, writing output data) related to the in-place judgment target layer (in-place processable layer) in the execution procedure list 80 to after the procedures related to the layer of the in-place non-target set, and ends the detailed judgment process. Note that for layers other than the in-place judgment target layer in the layers of elements of the in-place target set, the in-place inference possible flag in the routine allocation information 130 is cleared.
[0094] In this detailed determination process, even if the input matrix is input to another layer, it is possible to identify a layer that can perform in-place processing by changing the execution order of the layers. In the above example, for example, for each layer of the in-place non-target set, it is determined whether or not the in-place determination target layer is reached by tracing the input in order, but if it is guaranteed that the in-place determination target layer is not reached by tracing the input in order for each layer of the in-place non-target set, it is also possible to perform processing for the case where the in-place determination target layer is not reached by tracing the input in order without making a determination.
[0095] Next, the control routine 10 refers to the routine allocation information 150 in which the in-place inference possible flag is set, and updates the matrix allocation information 140 so that the work areas allocated to the input matrix and the output matrix refer to the same location. At this time, which input matrix and which output matrix share the work area is determined by referring to the in-place input / output pair of the in-place input / output pair 155 of the layer specification information 150. In addition, the control routine 10 sets an in-place sharing flag in the in-place sharing flag 143 of the matrix allocation information 140 corresponding to each of the input matrix and the output matrix, and sets the storage location of the input matrix in the storage location information 144 of the matrix allocation information 140 for the output matrix.
[0096] According to the above-mentioned in-place inference application process, for a layer for which the in-place inference possible flag is finally set, i.e., a layer with one element in the in-place target set and no elements in the in-place non-target set, or a layer for which in-place processing is possible by changing the execution order of the layers, it becomes possible to process the input matrix and the output matrix using the same area of the work area, thereby reducing the work area used. This makes it possible to increase the available memory area, and also to reduce the cache area used, thereby improving the cache hit rate when using the cache function.
[0097] In the above example, the matrix allocation information 140 is referenced to list the routine allocation information 150, but for example, the matrix allocation information 140 may be referenced to list the routine allocation information 150.
[0098] In the above example, for multiple matrices that can share the same area, the storage location of the multiple matrices is set to the same storage location as the storage location assigned to one of the matrices, so that the work area is shared, but the present invention is not limited to this, and a new work area may be reserved and set as a storage location shared by multiple matrices. Furthermore, after the storage location of multiple matrices is shared, the work area required for all matrices may be reallocated and compacted to eliminate fragmentation of the work area.
[0099] In the above example, the work area is allocated to the matrix in the routine allocation process, but it is also possible to allocate the necessary work area to the matrix after the in-place inference application process is executed. In this way, fragmentation of the work area does not occur, and there is no need to perform reallocation or compaction.
[0100] Next, a specific example of the flow of processing when the in-place inference application processing is executed in the case where the execution procedure list 80 shown in FIG. 12 is created in step S13 will be described.
[0101] Fig. 13 is a configuration diagram of an example of an execution procedure list in a trained model after in-place inference application processing. Here, in Fig. 13, a matrix whose area is shared in the in-place inference application processing is shown in the format of tensor_A / B. tensor_A / B indicates that tensor_A shares an area with tensor_B, and the area of the assigned work area is the area assigned to tensor_B.
[0102] First, the control routine 10 refers to the routine allocation information 130, obtains the layer type of the corresponding layer on the model from the model layer information 132, and judges whether or not the layer of this layer type supports in-place processing by itself by referring to the layer specification information 150 according to the layer type. Here, in the case of the learning model of Fig. 4, since the ReLU activation function layer and the update layer support in-place processing by themselves, the control routine 10 judges that node_3, node_4, node_5, and node_8 support in-place processing, and sets the in-place inference possible flag 136 of the routine allocation information 130 corresponding to these.
[0103] Next, the control routine 10 performs a process feasibility determination process for each layer to determine whether or not in-place processing can be performed, taking into account the entire trained model.
[0104] First, the control routine 10 refers to all matrix allocation information and lists the routine allocation information that uses each matrix. Here, tensor_1, tensor_2, tensor_3, and tensor_6 associated with the routine allocation information for which the in-place inference possible flag is set are subject to subsequent processing, and other matrices associated with the routine allocation information for which the in-place inference possible flag is not set are not subject to subsequent processing.
[0105] The control routine 10 enumerates the routine assignment information 130 related to the layer connected to tensor_1. As a result, only the routine assignment information 130 corresponding to node_3 is enumerated, and routine assignment information corresponding to other layers is not enumerated. As a result, since node_3 means that in-place processing can be performed taking into account the entire trained model, the control routine 10 ends the processing for tensor_1.
[0106] Next, the control routine 10 enumerates the routine assignment information 130 related to the layers connected to tensor_2. As a result, the routine assignment information 130 corresponding to node_4 and node_6 is enumerated, and one in-place target set (node_4) and one in-place non-target set (node_6) are obtained.
[0107] Since the in-place target set has one or more elements and the in-place non-target set has one or more elements, the control routine 10 performs detailed determination processing.
[0108] In this case, tracing the input of node_6 leads to node_4 in the in-place target set. In this case, the result of node_4 is required for inference of node_6, so node_4 must be inferred first. If in-place processing is performed on node_4, the contents of tensor_2 will be updated, which will affect the result of node_6. Therefore, in-place processing cannot be performed on node_4. For this reason, control routine 10 deletes the in-place inference possible flag of in-place inference possible flag 136 in routine allocation information 130 of node_4, and ends the processing for tensor_2.
[0109] Next, the control routine 10 enumerates the routine assignment information 130 related to the layer connected to tensor_3. As a result, only the routine assignment information 130 corresponding to node_5 is enumerated, and routine assignment information corresponding to other layers is not enumerated. As a result, since node_5 means that in-place processing can be performed taking into account the entire trained model, the control routine 10 ends the processing for tensor_5.
[0110] Next, the control routine 10 enumerates the routine assignment information 130 related to the layers connected to tensor_6. As a result, the routine assignment information 130 corresponding to node_7, node_8, and node_9 is enumerated, resulting in one in-place target set (node_8) and two in-place non-target sets (node_7, node_9).
[0111] Since the in-place target set has one or more elements and the in-place non-target set has one or more elements, the control routine 10 performs detailed determination processing.
[0112] In this case, even if the inputs of node_7 and node_9 are traced, node_8 in the in-place target set will not be reached. In this case, by placing node_8 after node_7 and node_9 in the execution order, in-place processing can be performed without affecting other layers. Therefore, the control routine 10 moves the procedures related to node_8 (reading input data, reading the routine, writing output data) to after node_7 and node_9 in the execution procedure list 80, and ends the detailed determination process.
[0113] As a result, in-place processing is possible for node_3, node_5, and node_8, and the in-place inference possible flags 136 in the routine allocation information 130 corresponding to these layers are set.
[0114] Next, the control routine 10 refers to the routine allocation information 130 for which the in-place inference possible flag is set, and updates the matrix allocation information 140 so that the work areas assigned to the input matrix and the output matrix refer to the same location.
[0115] Specifically, the control routine 10 processes node_3, refers to layer specification information 150 using the layer type ReLU of node_3, and determines from the information of in-place input / output pair 155 that the first input and the first output are an in-place input / output pair. Therefore, an in-place sharing flag is set to in-place sharing flag 143 of matrix allocation information 140 corresponding to tensor_1 which is the first input and tensor_3 which is the first output, and the storage location of tensor_1 is set to storage location information 144 of matrix allocation information 140 corresponding to tensor_3.
[0116] Next, the control routine 10 processes node_5, refers to the layer specification information 150 using the layer type Update of node_3, and determines from the information of the in-place input / output pair 155 that the first input and the first output are an in-place input / output pair. Therefore, an in-place sharing flag is set to the in-place sharing flag 143 of the matrix allocation information 140 corresponding to the first input, tensor_3, and the first output, tensor_5, and the storage location of tensor_3 is set to the storage location information 144 of the matrix allocation information 140 corresponding to tensor_5.
[0117] Next, the control routine 10 processes node_8, refers to the layer specification information 150 using the layer type Update of node_8, and determines from the information of the in-place input / output pair 155 that the first input and the first output are an in-place input / output pair. Therefore, an in-place sharing flag is set to the in-place sharing flag 143 of the matrix allocation information 140 corresponding to the first input, tensor_6, and the first output, tensor_8, and the storage location of tensor_6 is set to the storage location information 144 of the matrix allocation information 140 corresponding to tensor_8.
[0118] According to the above process, the execution procedure list 80 becomes as shown in the execution procedure list 81 in FIG. 13. Specifically, the process related to node_8 is moved after the process of node_9. In addition, for tensor_1 (denoted as tensor_1 / 1 in the figure), tensor_3 (denoted as tensor_3 / 1 in the figure), and tensor_5 (denoted as tensor_5 / 1 in the figure), the storage location of tensor_1 is shared, thereby reducing the area of two matrices. In addition, for tensor_6 (denoted as tensor_6 / 6 in the figure) and tensor_8 (denoted as tensor_8 / 6 in the figure), the storage location of tensor_6 is shared, thereby reducing the area of one matrix.
[0119] Returning to the explanation of Fig. 11, the control routine 10 executes the processing of loop 1 (inference processing: S15 to S18) a necessary number of times (for example, the number of times a user repeatedly executes inference processing using the same trained model). Here, one processing of loop 1 corresponds to one execution of inference processing according to the trained model.
[0120] In the processing of loop 1, the control routine 10 waits to receive an inference request from the user from the inference instruction routine 50 (S15). Here, the inference request includes a specification of input data, a specification of an output destination of output data, and the like.
[0121] When the control routine 10 receives an inference request, it reads the input data specified in the inference request into the work area 20 (S16).
[0122] Next, the control routine 10 repeatedly executes the processing of loop 2 (layer process processing (FIG. 14): S17) for each layer in the trained model. In this embodiment, the control routine 10 executes the processing according to the execution procedure list 81 shown in FIG.
[0123] After executing the processing of loop 2 for all layers in the trained model, the control routine 10 proceeds to step S18.
[0124] In step S18, the control routine 10 writes the final calculation result (inference result) stored in the work area 20 as model output data, for example, to the auxiliary storage device 106. The model output data written to the auxiliary storage device 106 is output to another device, for example, via a network.
[0125] After executing the processing of loop 1 (inference processing: S15 to S18) a required number of times, the control routine 10 exits from the processing of loop 1 and ends the inference control processing.
[0126] Next, the layer process in step S17 will be described.
[0127] FIG. 14 is a flow diagram of a layer process according to one embodiment.
[0128] The control routine 10 reads data to be input to the layer (layer input data) for the layer to be processed from the work area 20 into the CPU 101 (S21). Next, the control routine 10 calls the processing routine for the layer to be processed and executes the calculation process (part of the inference process) described in the processing routine (S22). Then, the control routine 10 writes data obtained by the calculation process (layer output data) into the work area 20 (S23), and ends the layer process.
[0129] According to the above embodiment, even when the same input matrix is input to multiple layers, in-place processing can be performed if in-place processing is possible by changing the execution order of the layers. This makes it possible to reduce the amount of work area usage. In addition, since the amount of work area usage can be reduced, the cache hit rate can be improved when using cache technology.
[0130] In the above embodiment, the execution order of layers is changed in the execution procedure list to enable in-place processing, but as shown in the following modified example, the execution order of layers may be changed for a trained model and the setting of a matrix using the same region may be reflected. In the following description of the modified example, the same reference numerals are used for functional configurations similar to those of the above embodiment for convenience.
[0131] FIG. 15 is a flowchart of a model generation process according to a modified example.
[0132] The control routine 10 of the inference device 100 (more precisely, the CPU 101 that executes the control routine 10) reads the trained model data 51 (S31).
[0133] Next, the control routine 10 performs a model structure analysis process (S32) to analyze the model structure based on the trained model data 51. Here, in the model structure analysis process, based on the trained model data 51, the connection relationship of each layer in the trained model as shown in FIG. 4 and the relationship of input and output matrices for each layer are identified.
[0134] Next, the control routine 10 executes a routine assignment process to assign a corresponding routine to a layer defined in the trained model data (S33).
[0135] In the routine allocation process, the control routine 10 creates matrix allocation information 140 for each matrix (here, input data and constants are also included in the matrix) of the trained model data. At this time, the control routine 10 assigns a new matrix allocation information ID to the matrix allocation information ID 141, and stores the matrix name of the matrix (Tensor, Data) in the model data in the matrix name 142. Furthermore, the control routine 10 stores the size of the matrix in the matrix size 145. Note that the in-place sharing flag 143 and the storage location information 144 are left unset.
[0136] Next, the control routine 10 performs an in-place inference conversion process to identify input and output matrices in the trained model data that can be processed using the same region of the work area 20 (S34).
[0137] In the in-place inference conversion process, the control routine 10 identifies layers that can be processed in-place without changing the execution order, and input and output matrices that can share a storage location, by a process similar to that of the in-place inference application process. Also, the control routine 10 identifies layers that can be processed in-place by changing the execution order of the layers, and input and output matrices that can share a storage location.
[0138] Next, the control routine 10 renames, in the trained model data, input and output matrices that can share a storage location in a layer that can perform in-place processing without changing the execution order, and matrices that are identical to those matrices, to the same matrix names. In addition, for layers that can perform in-place processing by changing the execution order of the layers, the control routine 10 moves the layer description to an order position that corresponds to the execution order in which in-place processing can be performed, renames input and output matrices that can share a storage location, and matrices that are identical to those matrices, and further deletes matrices that are no longer necessary due to the use of the same matrix names, and writes (generates) new trained model data (S35).
[0139] According to the trained model data generated in this manner, in-place processing can be effectively performed without performing the in-place inference application processing (S14) in the inference control processing shown in Fig. 11. That is, in a device capable of performing inference using trained model data, the in-place processing can be effectively performed by executing the data as is.
[0140] Fig. 16 is a diagram showing an example of trained model data generated according to a modified example. Trained model data 51C in Fig. 16 is trained model data (updated trained model) generated by executing a model generation process on trained model data 51B shown in Fig. 6. Note that in Fig. 16, the changed parts are written in bold.
[0141] In the model generation process, the input matrix and output matrix of the element part (Node:{···}) for node_3 in the trained model data 51B and the matrix names of the same matrices (in this example, the matrix name of the output (Output) of node_1 and the matrix name of the input (Input) of node_5) are given the same matrix name. Furthermore, the input matrix and output matrix of the element part (Node:{···}) for node_5 in the trained model data 51B and the matrix names of the same matrices (in this example, the matrix name of the output (Output) of node_3 and the matrix name of the input (Input) of node_7) are given the same matrix name (tensor_1_3_5). Here, in the example of FIG. 16, the matrix names include the numbers of the original matrices that are given the same matrix names. For example, when tensor_1 and tensor_3 have the same name, they are named tensor_1_3, and when tensor_1, tensor_3, and tensor_5 have the same name, they are named tensor_1_3_5.
[0142] Also, in the model generation process, the element part for node_8 (Node:{···}) in the trained model data 51B is moved after the element part for node_9 (Node:{···}), and the input matrix and output matrix and the matrix names of the same matrices are given the same matrix name (tensor_6_8) (in this example, the matrix name of the output (Output) of Node_6, the matrix name of the input (Input) of Node_7, the matrix name of the input (Input) of Node_9, and the matrix name of the input (Input) of Node_10).
[0143] In addition, in the model generation process, the element part (Tensor:{···}) of tensor_1 in the trained model data 51B is changed to the same determined matrix name (tensor_1_3_5), and the element parts of tensor_3 and tensor_5 are deleted. Furthermore, in the model generation process, the element part (Tensor:{···}) of tensor_6 in the trained model data 51B is changed to the same determined matrix name (tensor_6_8), and the element part of tensor_8 is deleted.
[0144] As a result, the trained model data 51B becomes trained model data 51C.
[0145] The present invention is not limited to the above-described embodiment and modified examples, and can be appropriately modified and implemented without departing from the spirit of the present invention.
[0146] For example, in the above embodiment, an in-place inference application process may be executed before the routine allocation process, and the execution order of layers may be changed based on the results of that process, and the routine allocation process may be performed based on that.
[0147] In the above embodiment, the in-place inference application process is executed before the inference process is executed to determine the execution order of each layer, but the present invention is not limited to this. For example, when the process of loop 2 is executed for each layer according to the execution procedure list (initial execution rule information) without executing the in-place inference application process of step S14, the execution order may be dynamically determined and the execution procedure list (execution rule information) may be changed and executed. Specifically, for example, when the control routine executes a process similar to the in-place inference application process for the target layer in the process of loop 2 and determines that the in-place process can be executed by changing the execution order of the layers, the order of the target layer among the unprocessed layers may be changed and the in-place process may be executed. In this case, the CPU 101 that executes the control routine corresponds to the initial execution rule information generating unit and the process executing unit.
[0148] In addition, in the above embodiment, a trained model was used as an example of a processing structure including multiple processing layers, but the processing structure may be a program structure including multiple processing layers (subroutines).
[0149] Furthermore, in the above embodiment, an example has been shown in which the processing routines are executed by a CPU, but at least one of the processing routines may be executed by another processor such as a GPU. [Explanation of symbols]
[0150] 10... control routine, 20... work area, 31... convolution layer routine, 32... pooling layer routine, 33... ReLU activation function layer routine, 34... addition layer routine, 35... update layer routine, 100... inference device, 101... CPU, 102... main memory, 108... display device, 111... recording medium
Claims
1. 1. A processing apparatus for processing a processing structure including a plurality of processing layers, comprising: A reception unit that receives an input of the processing structure; a generation unit that identifies an in-place processable layer from the processing structure, which is a processing layer capable of processing a first input matrix among input matrices in the processing layer and a first output matrix among output matrices in the processing layer using the same memory area, and when there are a plurality of common input processing layers which are processing layers that use the first input matrix, generates execution specification information that specifies that the in-place processable layer is executed last among the plurality of common input processing layers, and that the first output matrix is output to the memory area of the first input matrix in the processing of the in-place processable layer; A processing device having
2. The generating unit identifies the in-place processable layer based on the type of the process layer. The processing device of claim 1 .
3. The processing layer further includes a storage unit that stores in-place processability information indicating whether the processing layer is an in-place processable layer, The generation unit identifies the in-place processable layer based on the in-place processability information. The processing device of claim 1 .
4. The generation unit identifies the in-place processable layer based on the sizes of an input matrix and an output matrix of the processing layer. The processing device of claim 1 .
5. an allocation unit that allocates memory areas to each of the processing layers, the input matrices, and the output matrices included in the processing structure based on the processing structure; The processing device according to claim 1 , further comprising an aggregation / allocation unit that aggregates memory areas to be allocated to the plurality of processing layers, the plurality of input matrices, and the plurality of output matrices included in the processing structure based on the execution specification information.
6. 2. The processing device according to claim 1, further comprising an allocation unit that allocates memory areas to the multiple processing layers, multiple input matrices, and multiple output matrices included in the processing structure based on the execution specification information, and at that time, allocates the same memory area to the first input matrix and the first output matrix.
7. the generation unit determines whether or not an input is traced to the in-place processable layer for a processing layer other than the in-place processable layer among the plurality of common input processing layers; When it is determined that the in-place processable layer is reached, execution rule information is generated that specifies not to output the first output matrix to the memory area of the first input matrix in the processing of the in-place processable layer. The processing device of claim 1 .
8. The processing structure is a trained model used for inference processing. The processing device of claim 1 .
9. The execution rule information is an updated trained model obtained by updating the trained model. The processing device of claim 1 .
10. an initial execution rule information generating unit that generates initial execution rule information that defines execution of the processing layer based on the processing structure; a process execution unit capable of executing the process structure based on the initial execution rule information; the generation unit determines whether a next processing layer, which is a processing layer scheduled to be executed next, is the in-place processable layer when the processing execution unit executes the processing layer based on the initial execution rule information, and generates the execution rule information such that, when the next processing layer is the in-place processable layer and there are a plurality of unprocessed common input processing layers, the in-place processable layer is executed last among the plurality of common input processing layers, and the first output matrix is output to a memory area of the first input matrix in the processing of the in-place processable layer; The process execution unit executes the process layer based on the execution rule information instead of the initial execution rule information when the execution rule information is generated. The processing device of claim 1 .
11. A processing method for processing a processing structure including a plurality of processing layers using a processing apparatus, comprising: Accepting an input of the processing structure; From the processing structure, an in-place processable layer is identified as a processing layer capable of processing a first input matrix among input matrices in the processing layer and a first output matrix among output matrices in the processing layer using the same memory area, and when there are a plurality of common input processing layers that are processing layers that use the first input matrix, execution specification information is generated that specifies that the in-place processable layer is executed last among the plurality of common input processing layers, and that the first output matrix is output to the memory area of the first input matrix in the processing of the in-place processable layer. Processing methods.
12. A processing program for causing a computer to execute processing on a processing structure including a plurality of processing layers, The computer A reception unit that receives an input of the processing structure; A processing program that functions as a generation unit that identifies, from the processing structure, an in-place processable layer, which is a processing layer that can process a first input matrix among input matrices in the processing layer and a first output matrix among output matrices in the processing layer using the same memory area, and, when there are multiple common input processing layers, which are processing layers that use the first input matrix, executes the in-place processable layer last among the multiple common input processing layers, and generates execution specification information that specifies that the first output matrix is output to the memory area of the first input matrix in the processing of the in-place processable layer.