Learning apparatus, method, and program

Abstract

To implement efficient pruning.SOLUTION: A learning apparatus includes an adaptation unit, a learning unit, and a pruning unit. The adaptation unit performs, on a neural network model, adaptation processing that includes at least one of insertion of an activation function and correction of the activation function. The learning unit generates a trained model by training the neural network model on which the adaptation processing has been performed. The pruning unit performs pruning on the trained model to generate a reconstructed model from which a parameter has been reduced.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Embodiments of the present invention relate to a learning device, method, and program. [Background technology]

[0002] Pruning is a technique for reducing the model size of a neural network. By pruning, the model size of the neural network can be reduced while maintaining inference accuracy by reducing the number of parameters (weight coefficients or channels) of the neural network. Specifically, pruning is performed on the parameters (weight coefficients and channels) of the hidden layer using activation functions such as ReLU (Rectified Linear Unit) and hyperbolic tangent. However, there is a problem in that efficient pruning cannot be performed on hidden layers that do not have an activation function, and the desired pruning results cannot be obtained. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 6951295 [Overview of the project] [Problems that the invention aims to solve]

[0004] This disclosure is made to solve the aforementioned problems and aims to provide a learning device, method, and program that can achieve effective pruning. [Means for solving the problem]

[0005] The learning device according to this embodiment includes an adaptation unit, a learning unit, and a pruning unit. The adaptation unit executes an adaptation process including at least one of inserting an activation function into a neural network model and modifying an activation function. The learning unit trains the neural network model that has executed the adaptation process to generate a learned model. The pruning unit performs pruning on the learned model to generate a reconstructed model with reduced parameters.

Brief Description of Drawings

[0006] [Figure 1] Block diagram showing the learning device according to this embodiment. [Figure 2] Flowchart showing an operation example of the learning device according to this embodiment. [Figure 3] Diagram showing a first example of the adaptation process according to this embodiment. [Figure 4] Diagram showing a second example of the adaptation process according to this embodiment. [Figure 5] Graph showing differences in the shapes of activation functions. [Figure 6] Diagram showing a third example of the adaptation process according to this embodiment. [Figure 7] Diagram showing a fourth example of the adaptation process according to this embodiment. [Figure 8] Block diagram showing an example of the hardware configuration of the learning device 10 according to the embodiment.

Modes for Carrying Out the Invention

[0007] Hereinafter, the learning device, method, and program according to this embodiment will be described in detail with reference to the drawings. In the following embodiments, parts denoted by the same reference numerals perform the same operations, and redundant descriptions will be omitted as appropriate.

[0008] The learning device according to this embodiment will be described with reference to the block diagram of FIG. 1. The learning device 10 according to this embodiment includes an acquisition unit 101, an adaptation unit 102, a learning unit 103, a pruning unit 104, a determination unit 105, and a storage unit 106.

[0009] The acquisition unit 101 acquires training data and a neural network model. The neural network is not limited to multiple fully connected layers such as a Multilayer Perceptron (MLP), but can be any network architecture, such as a Convolutional Neural Network (CNN) like VGG16, ResNet, or DenseNet, a Recurrent Neural Network, a Transformer, or a Graph Neural Network. The adaptation unit 102 performs an adaptation process that includes at least one of inserting an activation function into the neural network model and modifying an activation function. Specifically, the adaptation process according to this embodiment refers to a process that arranges an activation function suitable for pruning into the neural network model, such as inserting an activation function between layers of the neural network model, replacing an activation function with another activation function, and moving the position of an activation function to a different position. The learning unit 103 trains the neural network model that has undergone the adaptation process and generates a trained model.

[0010] The pruning unit 104 performs pruning on the trained model to generate a reconstructed model with reduced parameters. The determination unit 105 determines whether the reconstructed model needs to undergo another fitting process and whether the reconstructed model needs to be retrained. The memory unit 106 stores training data, neural network models, reconstruction models, and the like.

[0011] Next, an example of operation 10 of the learning device according to this embodiment will be described with reference to the flowchart in Figure 2. In this embodiment, we will explain using an example of training a neural network model that performs a two-class image classification task, classifying images as either dogs or cats. The training data is not limited to images; it may also be video, text, audio, sensing data, or other time-series data. Furthermore, the objective task inferred by the neural network model is not limited to classification; it can be similarly applied to other objective tasks such as object detection, semantic segmentation, regression, and prediction.

[0012] In step SA1, the acquisition unit 101 acquires the training data and the neural network model from the storage unit 106. In step SA2, the determination unit 105 determines whether or not activation function adaptation is necessary for the neural network model. Here, the determination is made based on whether or not channel-level pruning based on group sparsification by inactive channels and inactive nodes, as described in the prior art documents mentioned above, can be performed. In other words, all nodes and channels in the hidden layers of the neural network model are set to activation functions that include an interval of input values ​​in which the differential function is 0 or an interval of input values ​​in which the differential function asymptotically approaches 0. For example, the activation function is a function in which the differential function is greater than 0 for the interval of input values ​​on the positive side of a given input value and 0 or asymptotically approaches 0 for the interval of input values ​​on the negative side of a given input value, such as the Rectified Linear Unit (ReLU). Setting such an activation function increases the likelihood of inactive nodes and inactive channels occurring. Furthermore, when the neural network model is optimized using L2 regularization as the regularization method and Adam or an Adam-like optimizer as the optimizer, group sparsification is more likely to occur, and channel-level pruning can be performed, for example. In this way, the need for adaptation processing is determined by whether or not the optimization conditions of the prior art literature are met. For example, if there is no activation function between certain layers, it is determined that adaptation processing is necessary because inserting an activation function may advance pruning. Also, if an activation function exists in the neural network model, the activation function should be modified so that the magnitude of the parameter gradient with respect to the loss during training decreases. If activation function adaptation is required, proceed to step SA3; otherwise, proceed to step SA4.

[0013] In step SA3, the adaptation unit 102 performs the adaptation process for the activation function. In this embodiment, the adaptation process is assumed to be the insertion, substitution, and movement of the activation function as described above. The adaptation unit 102 may apply any one of the adaptation processes to the neural network model, or it may apply multiple processes. For example, the adaptation unit 102 may combine the processes of substituting an activation function included in the neural network model and moving it to a different position. Furthermore, multiple methods may be employed in the adaptation process, and training in the next step, SA4, may be performed on each of the adapted neural network models. For example, the adaptation unit 102 may perform two patterns: inserting a ReLU as an activation function at a position where no activation function exists, and inserting a hard sigmoid function as a different activation function, and training in the next step, SA4, may be performed on the neural network model for each pattern.

[0014] In step SA4, the learning unit 103 trains the neural network model that was fitted in step SA3 or the neural network model that was determined not to require fitting in step SA2 using training data, thereby generating a trained model.

[0015] The training of the neural network model may be performed, for example, by executing "supervised learning" using training data including correct data. Specifically, the input image x → i is set as \(x^{(i)} (i = 1, \cdots, N)\). N is a natural number of 2 or more. The upper arrow indicates that it is a set of vectors. Here, i represents the serial number of the training data and the number of training data. The input image x → i is a set of pixels with a horizontal width W and a vertical height T, and is regarded as a vector of W×T dimensions. Also, the teaching label t → i is a two-dimensional vector in which the element corresponding to the teaching label is 1 and the other elements are 0. Specifically, for example, when the input image x → i is a dog, it can be expressed as \((1, 0)\) T and when it is a cat, it can be expressed as \((0, 1)\). T Here, ( ) T indicates a column vector.

[0016] In the training of the neural network model, the output y i of the neural network model when the input image x i is input can be expressed as in Equation (1). Here, the output y → i is an estimated probability value. y → i = f(Θ → , x → i ) ··· (1) Here, the function f(Θ → , ) is a function of the neural network model that holds the parameter set Θ → and outputs a two-dimensional vector.

[0017] Regarding the learning error L i , it is assumed that the calculation formula of Equation (2) is used. Here, ln is the natural logarithm. L i = -t → i T ln(y →i )···(2) Instructional label t → i And the output y of the neural network model → i The calculation is performed using the cross-entropy of the learning error L. i The parameter set Θ of the neural network model is minimized using backpropagation and stochastic gradient descent so as to minimize the loss function L based on the weighted average of the parameters. → This is learned through iterative training using mini-batch learning. In this embodiment, the learning conditions are set as follows: for example, the optimizer is set to "Adam", the L2 regularization intensity λ to "0.001", the number of epochs to "100", and the mini-batch size to "64", and the learning unit 103 trains the neural network model. Of course, it is also possible to train under other learning conditions, such as different learning rates and different regularization intensities, or to train under multiple learning conditions and adopt the one with the best performance.

[0018] The termination condition for iterative learning is, for example, the learning error L. i Alternatively, the learning unit 103 can determine whether a criterion such as the absolute value or decrease of the output of the loss function L is below a threshold. If the criterion is below the threshold, the learning unit 103 can determine that the conditions for terminating iterative learning have been met. Alternatively, the learning unit 103 can determine whether the number of iterations has reached a predetermined number, and if the number of iterations has reached the predetermined number, it can determine that the conditions for terminating iterative learning have been met. Upon completion of iterative learning, the training of the neural network model is completed, and a trained model is generated.

[0019] Note that the neural network can be trained using any method, not limited to the examples above. The training error L is also a factor. i This may be calculated using binary cross-entropy. Furthermore, while the above binary classification of dog or cat assumes that the output layer includes a sigmoid function, for multi-class classification tasks that classify into three or more classes, a softmax function may be used in the output layer.

[0020] In step SA5, the pruning unit 104 performs pruning on the trained model. Pruning involves, for example, calculating the L2 norm of the weight coefficients of each channel in the hidden layer and setting a predetermined threshold (e.g., 10 -6 Only channels with an L2 norm greater than ) (hereinafter referred to as "remaining channels") are selected, and a reconstructed model is generated, which is a neural network model consisting of these remaining channels. Alternatively, only channels whose absolute values ​​of weight coefficients in the hidden layer are greater than or equal to a predetermined threshold may be selected, and a reconstructed model may be generated. In other words, using the method described in the prior art literature, channels where the absolute values ​​of all weight coefficients set in the channel are almost zero, i.e., channels that have become group-sparse, can be removed.

[0021] In step SA6, the determination unit 105 determines whether the reconstructed model requires further adaptation processing (re-adaptation processing). To determine whether re-adaptation processing is necessary, for example, by comparing the performance of the reconstructed model with the baseline performance, if the performance of the reconstructed model is equal to or better than the baseline performance, it is determined that re-adaptation processing is unnecessary. If the performance of the reconstructed model is lower than the baseline performance, it is determined that re-adaptation processing is necessary.

[0022] Baseline performance is an index of the performance of a trained model obtained when trained under different learning conditions than those in step SA4 described above, and is used as a benchmark to determine whether the performance of the reconstructed model obtained by pruning is superior. Baseline performance may use generally expected index values ​​depending on the task, or the learning unit 103 may create the baseline performance. If generally expected index values ​​are used, for example, for a binary classification task, index values ​​such as precision, recall, and F1 score may be used. On the other hand, if the learning unit 103 creates it, for example, the result of training a neural network model with the optimizer set to "SGD (Stochastic Gradient Descent)" and the L2 regularization strength set to "0" may be used as the learning conditions for generating the baseline performance. The determination unit 105 should compare the created baseline performance with the performance of the reconstructed model to determine whether re-adjustment processing is necessary.

[0023] Furthermore, even if the performance of the reconstructed model is lower than the baseline result, if the final output model meets the requirements, it may be determined that re-adjustment is unnecessary. Specifically, if the requirements such as "the reconstructed model can prune β% of the overall parameters while maintaining a precision of α% or higher" are met, it may be determined that re-adjustment is unnecessary. If it is determined that reconformation is necessary, proceed to step SA7 and repeat the same process. If it is determined that reconformation is not necessary, proceed to step SA8.

[0024] In step SA7, the adaptation unit 102 performs a re-adaptation process of the activation function of the reconstructed model. The re-adaptation process is the same as in step SA3, except that the target of adaptation is the reconstructed model, so the explanation is omitted. Note that in the re-adaptation process, a different adaptation process may be performed than the previous adaptation process. For example, if the activation function ReLU was inserted in the previous adaptation process, the re-adaptation process may perform a process such as replacing ReLU with GeLU. Alternatively, if the activation function ReLU was replaced with a hard sigmoid function in the previous adaptation process, the re-adaptation process may perform a process such as replacing the hard sigmoid function with a sigmoid function.

[0025] In step SA8, the adaptation unit 102 restores the adaptation process performed in step SA3 to its pre-adaptation state. For example, if an activation function has been inserted into the reconstructed model by the adaptation process in step SA3, the adaptation unit 102 executes a process to delete that activation function. If it has been replaced with another activation function, the adaptation unit 102 executes a process to restore it to the original activation function. Also, if the connection position of the activation function has been changed, the adaptation unit 102 executes a process to restore the activation function to its original connection position.

[0026] In step SA9, the determination unit 105 determines whether or not retraining is necessary for the reconstructed model. To determine whether or not retraining is necessary, for example, the performance of the refitted reconstructed model is compared with the baseline result, and if the performance of the refitted reconstructed model is higher than or equal to the baseline performance, it is determined that retraining is not necessary.

[0027] If retraining is necessary, return to step SA4 and repeat the same process. If retraining is not necessary, the pruning of the trained model is complete, the final reconstructed model is generated, and the reconstructed model is stored in the memory unit 106. The final reconstructed model may also be output to a deployment destination. In retraining, the training parameters may be initialized using the parameters obtained in the previous training instead of being initialized randomly. In other words, retraining may be performed by fine tuning. In the readjustment process in step SA7, if the activation function after the readjustment process is of a different type than the activation function from the previous adaptation process, the hidden layer parameters may be initialized by multiplying them by an appropriate constant (gain).

[0028] Note that restoring the activation function in step SA8 is not mandatory, and the pruning of the reconstructed model may be continued while the activation function adaptation process is still applied. For example, if an activation function is inserted into the neural network model through the adaptation process and training is performed, if the determination unit 105 determines that the reconstructed model performs better than the model before the activation function was inserted, the pruning process may be continued while maintaining the inserted activation function. Whether or not to restore the activation function adapted by the adaptation process can be determined using the same method as the determination for the re-adaptation process using baseline performance. Furthermore, if the activation function is replaced by the adaptation process, for example, if the hyperbolic tangent function is used as the activation function and it is replaced with a hard hyperbolic tangent function, the computational cost can be reduced because the hard hyperbolic tangent function is a function that has a linear interval. Therefore, if the computational cost of the deployment target can be reduced when deploying the finally generated reconstructed model, the pruning process may be continued while maintaining the replaced activation function. Moreover, if the neural network's performance can be made more generalized by changing the insertion position of the activation function, the insertion position of that activation function may be maintained.

[0029] Next, a first example of the adaptation process according to this embodiment will be described with reference to Figure 3. Figure 3 (left) shows a neural network in which a fully connected layer 31 and a fully connected layer 32 are connected, and a hidden layer without an activation function is connected to them. In this neural network, the nodes connecting the parameters of the fully connected layer 31 and the parameters of the fully connected layer 32 are not transformed by a nonlinear function, making it difficult to reduce the number of parameters during pruning.

[0030] Therefore, among the multiple hidden layers that make up the neural network model, an activation function is connected after the hidden layer to be pruned. Specifically, as shown in the right diagram of Figure 3, an activation function 35 is inserted between the fully connected layer 31 and the fully connected layer 32. For example, ReLU is used as the activation function 35. As a result, when the output from the fully connected layer 31 is input to the activation function 35, nodes and channels that are not activated by the activation function are expressed. This allows for the pruning of the parameters of the fully connected layer 31.

[0031] Furthermore, when the adaptation unit 102 inserts the activation function 35, it may determine the insertion position of the activation function 61 so as not to change the feature map obtained from the hidden layer before the insertion of the activation function 61.

[0032] Next, a second example of the adaptation process according to this embodiment will be described with reference to Figures 4 and 5. Figure 4 (left) shows a neural network containing an activation function in which a fully connected layer 31, a first activation function 41, and a fully connected layer 32 are connected. Here, we assume that the first activation function 41 is a hyperbolic tangent.

[0033] On the other hand, in Figure 4 (right), the first activation function 41 is replaced by a second activation function 45, which exhibits similar behavior but with a reduced magnitude of the parameter gradient with respect to loss. Here, the second activation function 45 is assumed to be a hard hyperbolic tangent function. As an example of substitution from the first activation function 41 to the second activation function 45, Figure 5 shows a comparison between the hyperbolic tangent function and the hard hyperbolic tangent function. Figure 5 is a graph showing the output values ​​(vertical axis) against the input values ​​(horizontal axis) for the hyperbolic tangent function 51 and the hard hyperbolic tangent function 52.

[0034] The hard hyperbolic tangent function 52 has a wider range of output values ​​("1" or "-1") compared to the hyperbolic tangent function 51, meaning it has more flat sections in its shape. Therefore, the hard hyperbolic tangent function 52 reduces the magnitude of the parameter gradients compared to the hyperbolic tangent function 51, allowing for pruning of more parameters. Furthermore, because the hard hyperbolic tangent function 52 can be calculated using a linear function, it also reduces computational costs, such as the amount of computation required.

[0035] The candidate for substitution from the first activation function 41 to the second activation function 45 may be determined by referring to a lookup table. For example, a lookup table may be prepared that associates activation functions with smooth function shapes with activation functions having a linear function shape, known as a hard version. The adaptation unit 102 can substitute the current first activation function 41 with a second activation function 45 suitable for pruning by referring to the lookup table.

[0036] Furthermore, when the adaptation unit 102 replaces the first activation function 41 with the second activation function 45, it may calculate the error between the first activation function 41 and candidate activation functions that could become the second activation function 45, and determine the activation function with the smallest error as the second activation function. This allows learning to continue without significantly changing the output from the first activation function of the original neural network model.

[0037] Next, a third example of the adaptation process according to this embodiment will be described with reference to Figure 6. Figure 6 (left) shows a neural network including a fully connected layer 31, an activation function 61, a batch normalization layer 62, and an activation function connected to the fully connected layer 32. The adaptation unit 102 moves the activation function to a different connection position than the current one. That is, as shown in Figure 6 (right), the position of the activation function 61 is moved after the batch normalization layer 62. As a result, the input to the activation function 61 becomes the output of the batch normalization layer 62, which stabilizes the distribution of inputs to the activation function 61 and allows for pruning of more parameters.

[0038] Next, a fourth example of the adaptation process according to this embodiment will be described with reference to Figure 7. Figure 7 (left) shows a neural network including an activation function connected to a fully connected layer 31, a batch normalization layer 62, an activation function 61, and a fully connected layer 32. Specifically, in addition to the connection flow between the fully connected layer 31, the batch normalization layer 62, and the activation function 61, Figure 7 (left) assumes that the input image is skip-connected, and the residual block 70, which is the sum of the output of the residual block 70 and the input image, becomes the input to the fully connected layer 32.

[0039] Even in the case of such residual blocks 70, the adaptation unit 102 moves and positions the activation function 61 after the residual block 70. This aligns the channels in the pruning of multiple residual blocks 70, allowing for pruning of more parameters.

[0040] As shown in Figures 6 and 7, when the activation function 61 is moved, the adaptation unit 102 may determine the destination of the activation function 61 so as not to change the feature map obtained from the hidden layer before the activation function 61 was moved.

[0041] In the case of classification tasks, the layer immediately preceding the output layer of the neural network model (reconstruction model) has a significant impact on inference, so the adaptation unit 102 does not need to insert or move the activation function.

[0042] According to the embodiment described above, after performing adaptation processes such as inserting, replacing, and moving activation functions on the neural network model, training and pruning are performed to generate a reconstructed model. Furthermore, it is determined whether adaptation is necessary for the reconstructed model, a re-adaptation process is performed, and then it is similarly determined whether relearning is necessary. The re-adaptation process and relearning process are repeated until the given conditions are met. This allows pruning to be performed on the weight coefficients and channels of layers that could not be pruned before the adaptation process of the activation functions, thereby achieving effective pruning.

[0043] Next, an example of the hardware configuration of the learning device 10 according to the above embodiment is shown in the block diagram of Figure 8. The learning device 10 includes a CPU (Central Processing Unit) 81, RAM (Random Access Memory) 82, ROM (Read Only Memory) 83, storage 84, display device 85, input device 86, and communication device 87, all of which are connected by a bus.

[0044] The CPU 81 is a processor that performs arithmetic and control processing according to a program. The CPU 81 uses a predetermined area of ​​the RAM 82 as a working area and, in cooperation with programs stored in the ROM 83 and storage 84, executes the processing of each part of the learning device 10 described above.

[0045] RAM82 is a type of memory such as SDRAM (Synchronous Dynamic Random Access Memory). RAM82 functions as a workspace for the CPU81. ROM83 is a memory that stores programs and various information in a non-rewritable format.

[0046] The storage device 84 is a device that writes and reads data from magnetic recording media such as HDDs (Hard Disk Drives), semiconductor storage media such as flash memory, or magnetically recordable storage media such as HDDs, or optically recordable storage media. The storage device 84 writes and reads data from the storage media in response to control from the CPU 81. The storage device 84 realizes the storage unit 106 of the learning device 10 described above.

[0047] The display device 85 is a display device such as an LCD (Liquid Crystal Display). The display device 85 displays various information based on display signals from the CPU 81. The input device 86 is an input device such as a mouse and a keyboard. The input device 86 receives information input by the user as an instruction signal and outputs the instruction signal to the CPU 81. The communication device 87 communicates with external devices via a network in response to control from the CPU 81.

[0048] The instructions shown in the processing procedure described in the above-described embodiment can be executed based on a software program. A general-purpose computer system can store this program in advance and, by reading this program, can obtain effects similar to those of the control operation of the learning device described above. The instructions described in the above-described embodiment are recorded as a program that can be executed by a computer on a magnetic disk (flexible disk, hard disk, etc.), optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, Blu-ray® Disc, etc.), semiconductor memory, or similar recording medium. Any storage format is acceptable as long as it is a recording medium that can be read by a computer or embedded system. The computer can read the program from this recording medium and, based on this program, have the CPU execute the instructions described in the program, thereby achieving operation similar to the control of the learning device in the above-described embodiment. Of course, when the computer acquires or reads the program, it may do so via a network. Furthermore, an operating system (OS) running on a computer, a database management software, a network, or other middleware (MW) operating on a computer, based on instructions from a program installed on a computer or embedded system from a recording medium, may execute some of the processes necessary to realize this embodiment. Furthermore, the recording medium in this embodiment is not limited to a medium independent of the computer or embedded system, but also includes recording media that store or temporarily store programs downloaded via LAN, the Internet, etc. Furthermore, the recording medium is not limited to one; even when the processing in this embodiment is performed from multiple media, these are also included as recording media in this embodiment, and the configuration of the media may be any configuration.

[0049] In this embodiment, the computer or embedded system is used to execute each process in this embodiment based on a program stored on a recording medium, and may be configured as any of the following: a single device such as a personal computer or microcontroller, or a system in which multiple devices are connected via a network. Furthermore, the term "computer" in this embodiment is not limited to personal computers, but also includes arithmetic processing units, microcontrollers, and the like included in information processing equipment, and refers collectively to any equipment or device capable of realizing the functions of this embodiment through a program.

[0050] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0051] 10...Learning device, 31,32...Fully connected layers, 35...Activation function, 41...First activation function, 45...Second activation function, 51...Hyperbolic tangent function, 52...Hard hyperbolic tangent function, 61...Activation function, 62...Batch normalization layer, 70...Residual block, 84...Storage, 85...Display device, 86...Input device, 87...Communication device, 101...Acquisition unit, 102...Adaptation unit, 103...Learning unit, 104...Pruning unit, 105...Decision unit, 106...Memory unit

Claims

1. An adaptation unit that performs an adaptation process including at least one of inserting an activation function into a neural network model and modifying an activation function, A learning unit that trains the neural network model on which the aforementioned adaptation process has been performed and generates a trained model, A pruning unit performs pruning on the aforementioned trained model to generate a reconstructed model with reduced parameters, A learning device equipped with the following.

2. The learning device according to claim 1, wherein the reconstruction model is characterized in that the processing of the adaptation unit, the learning unit, and the pruning unit is repeatedly performed until a given condition is met.

3. The learning device according to claim 1, wherein the adaptation unit connects the activation function after the hidden layer to be pruned among the plurality of hidden layers forming the neural network model, as insertion of the activation function.

4. The learning device according to claim 1, wherein the adaptation unit includes a process of replacing the first activation function included in the neural network model with a second activation function different from the first activation function, as a modification of the activation function.

5. The learning device according to claim 4, wherein the second activation function is a function that reduces the magnitude of the parameter gradient with respect to the loss calculated during model training, compared to the first activation function.

6. The learning device according to claim 1, wherein the adaptation unit connects the activation function included in the neural network model to a position different from the current connection position as a modification of the activation function.

7. The learning device according to claim 1, wherein the adaptation unit restores the activation function inserted or modified by the adaptation process to its state before the adaptation process.

8. The system further comprises a determination unit that determines whether the performance of the reconstructed model after the adaptation process is equal to or greater than the performance of the model before the adaptation process. The learning device according to claim 1, wherein the adaptation unit maintains the adaptation process on the reconstructed model if the performance of the reconstructed model after the adaptation process is equal to or better than the performance of the model before the adaptation process.

9. The adaptation unit performs an adaptation process which includes at least one of inserting an activation function into a neural network model and modifying an activation function, The learning unit trains the neural network model that has undergone the aforementioned adaptation process and generates a trained model. A learning method comprising a pruning unit that performs pruning on the trained model to generate a reconstructed model with reduced parameters.

10. Computers, Adaptation means for performing an adaptation process that includes at least one of inserting an activation function into a neural network model and modifying an activation function, A learning means for training a neural network model that has undergone the aforementioned adaptation process and generating a trained model, A learning program that functions as a pruning means to perform pruning on the aforementioned trained model and generate a reconstructed model with reduced parameters.

Images