Learning apparatus, method, and program

Abstract

To implement efficient pruning.SOLUTION: A learning apparatus includes a learning unit, a pruning unit, a reconstruction unit, and a determination unit. The learning unit trains a neural network model having a plurality of pathways and generates a trained model. The pruning unit performs pruning on the trained model and calculates the number of remaining parameters of each of the pathways. The reconstruction unit generates a candidate model for reconstruction, the candidate model for reconstruction being generated by deleting a pathway in which the number of parameters is equal to or less than a threshold. The determination unit determines whether deletion of a further pathway included in the candidate model for reconstruction is possible or not. If it is determined that deletion of the further pathway is possible, the learning unit, the pruning unit, and the reconstruction unit execute processes on the candidate model for reconstruction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Embodiments of the present invention relate to learning devices, methods, and programs. [Background technology]

[0002] One technique for reducing the model size of a neural network is pruning. By pruning, parameters (weight coefficients or channels) of the neural network are removed, allowing the model size to be reduced while maintaining inference accuracy. However, while pruning can efficiently optimize paths and reduce model size when stable processing results are obtained under the same learning conditions, it is difficult to efficiently reduce model size when learning is unstable. For example, in neural networks with multiple paths connected, including hidden layers, learning tends to be unstable, and pruning also exhibits unstable behavior. Therefore, it is difficult to perform pruning efficiently. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 6951295 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] This disclosure is made to solve the aforementioned problems and aims to provide a learning device, method, and program that can efficiently optimize paths. [Means for solving the problem]

[0005] The learning device according to this embodiment includes a learning unit, a pruning unit, a reconstruction unit, and a determination unit. The learning unit trains a neural network model having a plurality of paths and generates a learned model. The pruning unit performs pruning on the learned model and calculates the remaining number of parameters for each of the plurality of paths. The reconstruction unit generates a reconstruction candidate model obtained by deleting paths whose number of parameters is less than or equal to a threshold. The determination unit determines whether the paths included in the reconstruction candidate model can be further deleted. When it is determined by the determination unit that the paths can be further deleted, each process of the learning unit, the pruning unit, and the reconstruction unit is executed on the reconstruction candidate model.

Brief Description of the Drawings

[0006] [Figure 1] Block diagram showing the learning device according to this embodiment. [Figure 2] Flowchart showing an operation example of the learning device according to this embodiment. [Figure 3] Diagram showing the first stage in an example of the pruning process of the reconstruction model. [Figure 4] Diagram showing the second stage in an example of the pruning process of the reconstruction model. [Figure 5] Diagram showing the third stage in an example of the pruning process of the reconstruction model. [Figure 6] Flowchart showing an operation example of the learning device for a plurality of reconstruction candidate models. [Figure 7] Graph showing an example of negative correlation between paths. [Figure 8] Diagram showing an example of display of pruning results. [Figure 9] Diagram showing an example of the hardware configuration of the learning device.

Embodiments for Carrying Out the Invention

[0007] Note: There was an asterisk added to ID=4 as it seemed to be a placeholder in the original text without any content to translate. If this is incorrect, please provide more context or clarify the requirements.The learning device, method, and program according to this embodiment will be described in detail below with reference to the drawings. In the following embodiments, parts with the same reference numerals perform similar operations, and redundant explanations will be omitted as appropriate.

[0008] (First Embodiment) A learning device according to the first embodiment will be described with reference to the block diagram in Figure 1. The learning device 10 according to the first embodiment includes a learning unit 101, a pruning unit 102, a reconstruction unit 103, a determination unit 104, a display control unit 105, and a storage unit 106.

[0009] The learning unit 101 trains a neural network model containing multiple paths based on the training data and learning conditions, and generates a trained model. A neural network model containing multiple paths is a neural network model in which there are multiple paths between the input layer and the output layer. For example, it is a neural network model in which multiple modules are combined and the outputs from each module are aggregated along the way. Furthermore, the neural network model can be any network with an architecture that has multiple paths, such as convolutional neural networks (CNNs) such as VGG16, ResNet, and DenseNet, recurrent neural networks (RNNs), transformers, and graph neural networks.

[0010] The pruning unit 102 performs pruning on the trained model and calculates the remaining number of parameters for each of the multiple paths. The number of parameters is at least one of the number of weight coefficients and the number of channels. The reconstruction unit 103 generates a reconstruction candidate model, which is a trained model from which paths with a number of parameters below a threshold have been removed. The reconstruction unit 103 also selects the reconstruction candidate model as the reconstructed model after pruning is complete if there are no paths that can be removed from the paths included in the reconstruction candidate model or if predetermined termination conditions are met.

[0011] The determination unit 104 determines whether further paths included in the reconstruction candidate model can be deleted. The determination unit 104 further determines whether retraining is necessary based on the pruning history. The pruning history is data that associates the history of pruning results when pruning is performed multiple times by the pruning unit 102 with the history of the corresponding reconstruction candidate model. The display control unit 105 controls the display to show the pruning history on an external display device such as a display. The memory unit 106 stores training data, neural network models, pruning results, candidate reconstruction models, and reconstruction models.

[0012] Next, an example of the operation of the learning device 10 according to this embodiment will be explained with reference to the flowchart in Figure 2. In this embodiment, we will explain using an example of training a neural network that performs a two-class image classification task, classifying images as either dogs or cats. Note that the training data is not limited to images; it may also be video, text, audio, sensing data, or other time-series data. Furthermore, the task inferred by the neural network is not limited to classification tasks; it can be similarly applied to other tasks such as object detection, semantic segmentation, regression, and prediction. In the following embodiment, we will explain the pruning target parameters assuming the neural network model and the reconstruction candidate model are at the channel level, but pruning may also be performed at the weight coefficient level of the neural network model and the reconstruction candidate model, or at the module level that constitutes multiple paths, or any other unit and method of pruning may be used.

[0013] In step SA1, the learning unit 101 trains the neural network model and generates a trained model. Training the neural network model can be done by performing supervised learning using training data that includes correct answers, for example. Specifically, input image x → iLet (i = 1, ···, N). N is a natural number greater than or equal to 2. The upward arrow indicates that it is a vector set. Here, i represents the sequential number of the training data and the number of training data. The input image x → i is a set of pixels with a horizontal width W and a vertical height T, and is regarded as a vector of W×T dimensions.

[0014] Also, the teaching label t → i is a two-dimensional vector in which the element corresponding to the teaching label is 1 and the other elements are 0. Specifically, for the input image x → i if it is a dog, it can be expressed as (1, 0) T if it is a cat, it can be expressed as (0, 1) T Note that ( ) T indicates a column vector.

[0015] In the training of the neural network model, when the input image x → i is input, the output y → i of the neural network model can be expressed as in equation (1). Here, the output y → i is an estimated probability value. Here, for the input image y → i = f(Θ → , x → i ) ··· (1) Here, f(Θ → , ) is a function of the neural network model that holds the parameter set Θ → and outputs a two-dimensional vector.

[0016] Regarding the learning error L i it is assumed that the calculation formula of equation (2) is used. Here, ln is the natural logarithm. L i = -t → i T ln(y → i ) ··· (2) The teaching label t→ i And the output y of the neural network model → i The calculation is performed using the cross-entropy of the learning error L. i The parameter set Θ of the neural network model is minimized using backpropagation and stochastic gradient descent so as to minimize the loss function L based on the weighted average of the parameters. → This is learned through iterative training using mini-batch learning.

[0017] In this embodiment, the learning conditions are set as follows: for example, the optimizer is set to "Adam (learning rate: 0.01)", the L2 regularization intensity λ is set to "0.001", the number of epochs is set to "100", and the mini-batch size is set to "64", and the learning unit 101 trains the neural network model. This training method using these learning conditions causes group sparsity to occur in the channels of the hidden layers included in the neural network model and the reconstruction candidate model. When training the neural network model and the reconstruction candidate model again, the learning unit 101 may train under different learning conditions, such as different learning rates and different regularization intensities, or it may train under multiple learning conditions and adopt the one with the best performance.

[0018] The termination condition for iterative learning is, for example, the learning error L. i Alternatively, it is sufficient to determine whether a criterion such as the absolute value or decrease value of the output of the loss function L is below a threshold. If the criterion is below the threshold, the learning unit 101 (or the determination unit 104) can determine that the conditions for terminating iterative learning have been met. Alternatively, the learning unit 101 can determine whether the number of iterations has reached a predetermined number, and if the number of iterations has reached the predetermined number, it can determine that the conditions for terminating iterative learning have been met. Upon completion of iterative learning, the training of the neural network model is completed, and a trained model is generated. Note that the neural network can be trained using any method, not limited to the examples above. The training error L is also a factor. iThis may be calculated using binary cross-entropy. Furthermore, while the above binary classification of dog or cat assumes that the output layer of function f includes a sigmoid function, in the case of multi-class classification problems that classify into three or more classes, a softmax function may be used in the output layer of function f.

[0019] In step SA2, the pruning unit 102 performs pruning on the trained model. Pruning involves, for example, calculating the L2 norm of each channel in the hidden layer included in each path, and selecting a predetermined threshold (e.g., 10) from among the paths that have the L2 norm. -6 The pruning results are calculated to show the number of channels for each path that has an L2 norm greater than ). Note that other general methods may be used for pruning.

[0020] In step SA3, the reconstruction unit 103 reconstructs the trained model based on the pruning results obtained in step SA2 and generates a candidate model for reconstruction. For example, if there is a path for which the pruning result shows that the number of remaining channels in the hidden layer is below a threshold (it may be 0), that path can be deleted, and the trained model can be reconstructed using only the remaining paths to generate a candidate model for reconstruction. If there is no path for which the number of channels in the hidden layer is below the threshold, the model structure will be the same as the trained model before pruning.

[0021] In step SA4, the memory unit 106 stores the pruning results for each calculated path and the corresponding reconstruction candidate models as a pruning history. Alternatively, the memory unit 106 may also store the pruning history along with the execution count, which indicates which pruning it is.

[0022] In step SA5, the determination unit 104 determines whether or not to retrain the reconstruction candidate model. For example, if the reconstruction candidate model has only one path, it determines that retraining is unnecessary. On the other hand, if there are multiple paths, it determines that retraining is necessary. Furthermore, even if there are multiple paths, the determination unit 104 may determine that retraining is unnecessary if the pruning results for the reconstruction candidate model are stable. For example, by comparing the pruning results obtained up to the previous step with the newly acquired pruning results, if the change in the number of remaining channels is within a threshold, for example, if the difference in the number of channels is within a threshold, the pruning results can be said to be stable, and the determination unit 104 can determine that retraining is unnecessary. If it is determined that retraining is necessary, the process returns to step SA1 and the same process is repeated. That is, retraining is performed on the candidate model for reconstruction, and pruning is performed on the retrained candidate model for reconstruction. On the other hand, if it is determined that retraining is not necessary, the process proceeds to step SA6. In other words, retraining is performed until the candidate model for reconstruction contains only one path, or until the variation in the pruning results of one or more paths falls below a threshold.

[0023] In step SA6, the determination unit 104 outputs a reconstruction candidate model (hereinafter referred to as the reconstruction model) in which the optimal pruning has been performed. The reconstruction model may be stored in the storage unit 106. This completes the operation of the learning device 10.

[0024] Furthermore, for one neural network model or one reconstruction candidate model, pruning may be repeated multiple times, and if, based on the obtained pruning results, there exists a path where the remaining number of channels is below a threshold after a predetermined number of prunings, the reconstruction unit 103 may be configured to delete that path during reconstruction. In other words, steps SA1 and SA2 are repeatedly executed for one neural network model or one reconstruction candidate model. This improves the accuracy of identifying paths that should be deleted.

[0025] Next, an example of the pruning process up to generating the reconstruction model according to this embodiment will be described with reference to Figures 3 to 5. Figure 3 shows the neural network model 30 before pruning. The neural network model 30 includes path 1 (input layer 31 and hidden layer 32), path 2 (input layer 33 and hidden layer 34), path 3 (input layer 35 and hidden layer 36), concatenate layer 37, concatenate layer 38, hidden layer 39, and output layer 40. Here, we assume that paths 1, 2, and 3 have the same layer structure, but that the parameters of their hidden layers are different.

[0026] Each input layer 31, 33, and 35 receives the input image x → i The following is input. The outputs of path 1 and path 2 are aggregated in Concatenate layer 37. The output from Concatenate layer 37 and the output from path 3 are aggregated in Concatenate layer 38. The output from Concatenate layer 38 is input to hidden layer 39, and the output of hidden layer 39 is input to output layer 40, and the estimated probability value y is input. → i The output is as follows. Note that activation functions such as ReLU are placed after each hidden layer 32, 34, 36, and 39. Note that the input image x is input to each input layer 31, 33, and 35. → i The inputs may be the same image, or different images may be input to each input layer, such as so-called data-enhanced images that have been rotated, cropped, or have their colors changed. Here, we assume that, as a result of pruning in step SA2 of Figure 2, the number of remaining channels in the hidden layer 32 of path 1 becomes less than or equal to the threshold, while the number of remaining channels in the hidden layer 34 of path 2 and the hidden layer 36 of path 3 is greater than the threshold. In this case, path 1 can be deleted.

[0027] Figure 4 shows the reconstruction candidate model 50 after removing path 1. By removing path 1, there is no longer a need to aggregate the outputs of path 1 and path 2, so the Concatenate layer 37 becomes unnecessary and is also removed. Here, it is determined that the reconstruction candidate model 50 shown in Figure 4 requires retraining because multiple paths exist. Therefore, the processes from step SA1 to step SA4 in Figure 2 are executed again. Here, we assume that the number of remaining channels in the hidden layer 36 of path 3 is below the threshold, and the number of remaining channels in the hidden layer 34 of path 2 is greater than the threshold. In this case, path 3 can be deleted.

[0028] Figure 5 shows reconstruction candidate model 70, obtained by removing path 3 from reconstruction candidate model 50. Since the reconstruction candidate model shown in Figure 5 has only one path, it is output as the final reconstruction model. In this way, efficient pruning can be achieved by repeatedly applying the processes of learning, pruning, and reconstruction to the neural network model and reconstruction candidate model, and gradually removing paths.

[0029] In the examples in Figures 3 to 5, we assumed that the neural network model has three paths, but it is not limited to this; it may also have two paths, or four or more paths. Furthermore, the structure of each path, such as the number and type of hidden layers, may differ. Also, in the example in Figure 3, we assume that there are two concatenate layers, but the neural network model architecture may also be one in which the outputs from each path are aggregated in a single concatenate layer. In addition, although we have shown an example in which the outputs from each path are aggregated in a concatenate layer, the outputs from each path may also be added together. Furthermore, Figures 3 to 5 show a case where multiple paths are arranged from the input layer stage, multiple inputs are fed into different paths, and the outputs of multiple paths are aggregated at a certain layer. However, the architecture is not limited to this, and a neural network model may have only one input layer, with multiple paths existing due to branching along the way.

[0030] Furthermore, pruning may be performed multiple times on the same neural network model and the candidate model for reconstruction. This allows us to determine that there is a high probability that the same path can be removed if the same path is identified as a target for deletion after multiple pruning attempts.

[0031] Furthermore, if the results of the first pruning differ from those of the second pruning, multiple reconstruction candidate models may be generated. For example, as illustrated in Figure 3, suppose the first pruning result for the neural network model 30 indicates that path 1 can be removed, and a first reconstruction candidate model including paths 2 and 3 is generated. Then, the second pruning result for the neural network model indicates that path 2 can be removed, and a second reconstruction candidate model including paths 1 and 3 is obtained. In this case, the learning, pruning, and reconstruction processes may be performed on each of the obtained reconstruction candidate models, i.e., the first and second reconstruction candidate models.

[0032] An example of the operation of the learning device 10 for multiple reconstruction candidate models will be explained with reference to the flowchart in Figure 6. In step SB1, similar to step SA1 in Figure 2, the learning unit 101 learns a neural network model and generates a trained model. In step SB2, similar to step SA2 in Figure 2, the pruning unit 102 performs pruning on the trained model.

[0033] In step SB3, similar to step SA3 in Figure 2, the reconstruction unit 103 reconstructs the trained model based on the pruning results and generates a candidate model for reconstruction. In step SB4, the memory unit 106 stores the pruning history.

[0034] In step SB5, the determination unit 104 determines whether the reconstruction candidate model generated in step SB4 is different from the reconstruction candidate models generated in previous steps, or in other words, whether a different reconstruction candidate model has been generated. Specifically, for example, if a reconstruction candidate model with a different path than the one generated in the current process is stored in the storage unit 106, the determination unit 104 can determine that a different reconstruction candidate model has been generated. If a different reconstruction candidate model has been generated, the process proceeds to step SB7; if the same reconstruction candidate model as before has been generated, the process proceeds to step SB6.

[0035] In step SB6, the determination unit 104 determines whether or not the reconstruction candidate model needs to be retrained, similar to the case of step SA5 in Figure 2, since the reconstruction candidate model is the same as the previous time. If retraining is necessary, the process returns to step SB1 and the same process is repeated. If retraining is not necessary, the process proceeds to step SB9.

[0036] In step SB7, the determination unit 104 determines whether or not retraining is necessary for multiple reconstruction candidate models. For example, if a new reconstruction candidate model with a different path is generated, all reconstruction candidate models, including those generated up to that point, may be subject to retraining. Alternatively, from among multiple reconstruction candidate models, only those that satisfy predetermined conditions may be subject to retraining, and reconstruction candidate models that do not satisfy these conditions may not be retrained. For example, the determination unit 104 may determine that the reconstruction candidate model with the highest performance, specifically the highest recognition rate or accuracy rate, is subject to retraining. Or, the determination unit 104 may determine that the reconstruction candidate model with the smallest model size, specifically the one with the fewest parameters or computational complexity, is subject to retraining. Alternatively, the target of retraining is not limited to a single reconstruction candidate model; one or more reconstruction candidate models whose performance is above a threshold, or one or more reconstruction candidate models whose model size is below a threshold, may be subject to retraining. If it is determined that relearning is necessary, proceed to step SB8; otherwise, proceed to step SB9.

[0037] In step SB8, the learning unit 101 retrains the target reconstruction candidate model. The learning method should be the same as in step SB1. After that, the process returns to step SB2 and the same process is repeated. In step SB9, the determination unit 104 outputs the retrained candidate model as the final reconstructed model. The memory unit 106 may store the reconstructed model. Alternatively, as in the case of Figure 2, the pruning results may be calculated multiple times for one neural network model or one candidate model, and the reconstruction unit 103 may delete paths where the number of remaining channels falls below a threshold for a predetermined number of times or more.

[0038] If there are budget and time constraints before generating a reconstructed model after pruning using the learning device according to this embodiment, the process may be terminated early after a predetermined time has elapsed without performing further retraining. In this case, the determination unit 104 may select the reconstructed model from among the multiple reconstructed candidate models stored in the storage unit 106 that has the highest performance or the smallest model size.

[0039] Furthermore, if budget and time are limited, the conditions for generating candidate reconstructed models based on pruning results may be relaxed in the processing of step SB4. For example, the reconstruction unit 103 may reconstruct the model by deleting paths with fewer than a threshold number of remaining channels and leaving only paths with more than or equal to the threshold number of remaining channels, depending on the remaining time. Specifically, when the remaining time for pre-set training time becomes half, a quarter, etc., the reconstruction unit 103 may reconstruct the model by leaving only paths with more than or equal to P% of the total number of remaining channels (P is an arbitrary integer, e.g., 50). Note that the value of P may not be fixed, but may be increased according to the passage of remaining time. For example, the value of P may be increased by 1, 2, etc., each time the remaining time becomes half, a quarter, or an eighth of the total time.

[0040] Furthermore, paths may be forcibly deleted depending on the remaining time. For example, the reconstruction unit 103 may delete the path with the fewest remaining channels when the remaining time is half of the total time, and delete the path with the fewest remaining channels at that point when the remaining time is one-quarter of the total time. In this way, by performing pruning and reconstruction while changing the reconstruction conditions considering the budget and time, pruning can be advanced within the expected learning time.

[0041] Furthermore, even if a certain path cannot be removed during pruning, if there is a negative correlation between that path and other paths, multiple candidate reconstructed models may be generated by removing each of those paths. The case where there is a negative correlation between paths will be explained with reference to Figure 7. Figure 7 is a plot showing the relationship between the number of channels in each path for each pruning result, with the number of channels in path 1 on the horizontal axis and the number of channels in path 2 on the vertical axis. As shown in this plot, the more channels in path 1 there are, the fewer channels in path 2 there are, and conversely, the more channels in path 2 there are, the fewer channels in path 1 there are. In addition, for example, the display control unit 105 may display a straight line 71 that approximates the point cloud using linear regression. In this way, if the determination unit 104 can determine that there is such a negative correlation (trade-off) between paths by performing learning and pruning multiple times, the reconstruction unit 103 can generate a reconstruction candidate model in which path 1 is deleted and path 2 is kept, and a reconstruction candidate model in which path 2 is deleted and path 1 is kept, respectively, and then execute the learning process shown in Figure 6. The display control unit 105 may plot the pruning results on a graph each time they are obtained and display the graph shown in Figure 7 on an external display device such as a display.

[0042] Next, an example of how pruning results are displayed will be explained with reference to Figure 8. Figure 8 is a partial table of the pruning history showing the pruning results for each route after multiple pruning cycles. Specifically, it associates the number of remaining channels for each route from Route 1 to Route 3 with each pruning cycle. The "-" notation indicates that there is no corresponding value because the route was deleted based on the previous pruning result. The display control unit 105 displays this table on the display so that the user can easily understand which routes can be deleted or not.

[0043] Furthermore, in the example in Figure 8, routes for which pruning results in zero remaining channels are highlighted by hatching the cells of routes where the number of remaining channels falls below the threshold. However, this is not the only way to distinguish routes with a remaining number of channels that can be deleted from others; this could be done by changing the text color, making them bold, or other methods. This allows users to easily identify which routes can be deleted. Conversely, cells of routes with more remaining channels than the threshold could also be highlighted. This makes it easy to identify stable routes.

[0044] According to the embodiment described above, a neural network composed of multiple paths is trained, and the pruning results of the trained neural network are output based on a threshold. Next, a candidate model for reconstruction is reconstructed from the trained model based on the pruning results. If it is determined that retraining is necessary from the remaining paths of the candidate model for reconstruction, the process of retraining and pruning the candidate model for reconstruction is repeated. If it is determined that retraining is not necessary, the candidate model for reconstruction is output as the final reconstructed model. This allows pruning to be performed stepwise regardless of the stability of the pruning results, enabling pruning with a smaller model size and at a faster speed. As a result, paths can be optimized efficiently.

[0045] Next, an example of the hardware configuration of the learning device 10 according to the above embodiment is shown in the block diagram of Figure 9. The learning device 10 includes a CPU (Central Processing Unit) 91, RAM (Random Access Memory) 92, ROM (Read Only Memory) 93, storage 94, display device 95, input device 96, and communication device 97, all of which are connected by a bus.

[0046] The CPU 91 is a processor that performs arithmetic and control processing according to a program. The CPU 91 uses a predetermined area of ​​the RAM 92 as a working area and, in cooperation with programs stored in the ROM 93 and storage 94, performs the processing of each part of the learning device 10 described above.

[0047] RAM92 is a type of memory such as SDRAM (Synchronous Dynamic Random Access Memory). RAM92 functions as a workspace for the CPU91. ROM93 is a memory that stores programs and various information in a non-rewritable format.

[0048] The storage device 94 is a device that writes and reads data from magnetic recording media such as HDDs (Hard Disk Drives), semiconductor storage media such as flash memory, or magnetically recordable storage media such as HDDs, or optically recordable storage media. The storage device 94 writes and reads data from the storage media in accordance with the control from the CPU 91.

[0049] The display device 95 is a display device such as an LCD (Liquid Crystal Display). The display device 95 displays various information based on display signals from the CPU 91.

[0050] The input device 96 is an input device such as a mouse or keyboard. The input device 96 receives information input by the user as an instruction signal and outputs the instruction signal to the CPU 91.

[0051] The communication device 97 communicates with external devices via a network in response to control from the CPU 91.

[0052] The instructions shown in the processing procedure described in the above-described embodiment can be executed based on a software program. A general-purpose computer system can store this program in advance and, by reading this program, can obtain effects similar to those of the control operation of the learning device described above. The instructions described in the above-described embodiment are recorded as a program that can be executed by a computer on a magnetic disk (flexible disk, hard disk, etc.), optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, Blu-ray® Disc, etc.), semiconductor memory, or similar recording medium. Any storage format is acceptable as long as it is a recording medium that can be read by a computer or embedded system. The computer can read the program from this recording medium and, based on this program, have the CPU execute the instructions described in the program, thereby achieving operation similar to the control of the learning device in the above-described embodiment. Of course, when the computer acquires or reads the program, it may do so via a network. Furthermore, an operating system (OS) running on a computer, a database management software, a network, or other middleware (MW) operating on a computer, based on instructions from a program installed on a computer or embedded system from a recording medium, may execute some of the processes necessary to realize this embodiment. Furthermore, the recording medium in this embodiment is not limited to a medium independent of the computer or embedded system, but also includes a recording medium that stores or temporarily stores a program that has been downloaded via a LAN, the Internet, or the like. Furthermore, the recording medium is not limited to one; even when the processing in this embodiment is performed from multiple media, these are also included as recording media in this embodiment, and the configuration of the media may be any configuration.

[0053] In this embodiment, the computer or embedded system is used to execute each process in this embodiment based on a program stored on a recording medium, and may be configured as any of the following: a single device such as a personal computer or microcontroller, or a system in which multiple devices are connected via a network. Furthermore, the term "computer" in this embodiment is not limited to personal computers, but also includes arithmetic processing units, microcontrollers, and the like included in information processing equipment, and refers collectively to any equipment or device capable of realizing the functions of this embodiment through a program.

[0054] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0055] 10...Learning device, 30...Neural network model, 31, 33, 35...Input layer, 32, 34, 36, 39...Hidden layer, 37, 38...Concatenate layer, 40...Output layer, 50, 70...Reconstruction candidate model, 71...Linear, 91...CPU, 92...RAM, 93...ROM, 94...Storage, 95...Display device, 96...Input device, 97...Communication device, 101...Learning unit, 102...Pruning unit, 103...Reconstruction unit, 104...Decision unit, 105...Display control unit, 106...Memory unit

Claims

1. A learning unit that trains a neural network model with multiple paths and generates a trained model, A pruning unit performs pruning on the trained model and calculates the number of remaining parameters for each of the multiple paths. A reconstruction unit that generates a reconstruction candidate model by deleting paths in which the number of parameters is below a threshold, The system comprises a determination unit that determines whether or not the paths included in the candidate reconstruction model can be further deleted, If the determination unit determines that the path can be further deleted, the learning device executes the processes of the learning unit, the pruning unit, and the reconstruction unit on the candidate model for reconstruction.

2. The learning device according to claim 1, which repeatedly executes the processes of the learning unit, the pruning unit, and the reconstruction unit on the reconstruction candidate model until there are no more paths that can be deleted.

3. The learning device according to claim 2, wherein the reconstruction unit selects the reconstruction candidate model as the reconstructed model after pruning has been completed if there are no paths that can be deleted among the paths included in the reconstruction candidate model or if a predetermined termination condition is met.

4. The pruning unit performs the pruning multiple times on the neural network model or the candidate model for reconstruction. The learning device is The learning device according to claim 1, further comprising a storage unit that stores a pruning history which associates the history of multiple pruning results with the history of the reconstruction candidate model for each pruning result.

5. The learning device according to claim 4, wherein the reconstruction unit reconstructs a candidate model for reconstruction based on the pruning history.

6. The learning device according to claim 4, wherein the determination unit determines whether or not retraining is necessary based on the pruning history.

7. The number of parameters is the number of channels, The pruning unit performs the pruning multiple times on the neural network model or the candidate model for reconstruction. The learning device according to claim 1, wherein the reconstruction unit deletes a path if, in the pruning results performed multiple times, there exists a path where the number of remaining channels is less than or equal to the threshold for a predetermined number of times or more.

8. The learning device according to claim 4, further comprising a display control unit that displays the number of remaining parameters for each path on a display device as the pruning history.

9. The learning device according to claim 8, wherein the display control unit highlights paths in which the number of remaining parameters is less than or equal to a first threshold, or paths in which the number of remaining parameters is greater than or equal to a second threshold, which is greater than the first threshold.

10. The pruning unit performs the pruning multiple times on the neural network model or the candidate model for reconstruction. The learning device according to claim 1, wherein, if there are multiple reconstruction candidate models in which different paths have been deleted, the determination unit determines whether or not the paths can be further deleted for each of the multiple reconstruction candidate models.

11. The pruning unit performs the pruning multiple times on the neural network model or the candidate model for reconstruction. The determination unit determines whether there is a negative correlation between multiple paths included in the neural network model or between multiple paths included in the reconstruction candidate model. The learning device according to claim 1, wherein the reconstruction unit generates a plurality of reconstruction candidate models in which different paths are deleted from the plurality of paths when there is a negative correlation.

12. The learning device according to claim 1, wherein the number of parameters is at least one of the number of weight coefficients and the number of channels.

13. The learning device according to claim 1, wherein the learning unit trains the neural network model and the reconstruction candidate model using a training method that causes sparsification on a channel-by-channel basis.

14. The learning method trains a neural network model with multiple paths and generates a trained model. The pruning means performs pruning on the trained model and calculates the number of remaining parameters for each of the multiple paths. The reconstruction means generates a reconstruction candidate model by deleting paths in which the number of parameters is below a threshold, The determination means determines whether the paths included in the reconstruction candidate model can be further deleted. A learning method comprising: if the determination means determines that the path can be further deleted, executing the processes of the learning means, the pruning means, and the reconstruction means on the candidate model for reconstruction.

15. Computers, A learning method for training a neural network model with multiple paths and generating a trained model, A pruning means that performs pruning on the trained model and calculates the number of remaining parameters for each of the multiple paths, Reconstruction means for generating a reconstruction candidate model by deleting paths in which the number of parameters is below a threshold, This is configured to function as a determination means for determining whether or not the paths included in the aforementioned reconstruction candidate model can be further deleted. If the determination means determines that the path can be further deleted, a learning program is provided to cause the reconstruction candidate model to execute the processes of the learning means, the pruning means, and the reconstruction means.

Images