Learning processing device and method
Patent Information
- Authority / Receiving Office
- TW · TW
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-01-12
- Publication Date
- 2021-08-16
Smart Images
Figure TWG2TA000822400_001 
Figure TWG2TA000822400_002 
Figure TWG2TA000822400_003
Abstract
Description
[Technical Field]
[0001] The present invention relates to a learning processing apparatus and method utilizing neural networks. [Previous Technology]
[0002] In the field of signal processing, such as image processing, natural language processing, and voice recognition, learning processing devices utilizing multi-layer neural networks are used for recognition processing. In recent years, neural networks have acquired a large number of parameters and require enormous computational resources to improve recognition performance. However, in the recognition processing performed after learning, it is ideal to process large amounts of data such as images with fewer parameters and less computational resources. In this invention, a neural network capable of processing with fewer parameters and less computational resources is called a lightweight model.
[0003] As a technique for reducing the parameters and computational load of multi-layer neural networks after learning, there is a technique called "pruning". Pruning is a technique that identifies unnecessary parameters or computations in the learned neural network and reduces the parameters and computational load required for the recognition and processing of the neural network by deleting them.
[0004] For example, Patent Document 1 discusses a mechanism in which units of a learned neural network are randomly deleted, and after relearning, a cost function is evaluated, and the structure with the best cost function is output as the best structure of the neural network. [Prior Art Documents] [Patent Documents]
[0005] [Patent Document 1] Japanese Patent Application Publication No. 2015-11510 [Summary of the Invention]
[0006] [The problem the invention aims to solve]
[0007] Generally speaking, lightweighting of neural networks due to pruning requires a relatively long learning period. This is because, during neural network lightweighting, in order to minimize the adverse impact on recognition accuracy, only a small number of parameters can be deleted each time. Since deleting a large number of parameters at once will lead to a significant decrease in recognition accuracy, it is usually done by repeating the deletion of a small number of parameters several times and relearning to learn a lightweight model with less degradation in recognition accuracy.
[0008] Therefore, the learning of lightweight neural networks requires several times more learning cycles than that of conventional neural networks, resulting in a longer learning period. Especially when there are many types of recognition processing performed or when the neural network model used is switched in the execution environment, the long learning period is the main reason for the deployment and update delay of recognition processing functions because multiple lightweight models must be prepared.
[0009] Therefore, although it is necessary to enable the lightweight model to complete the required configuration in a short period of time, Patent Document 1 does not discuss the short-term learning of the lightweight model. [Means for solving the problem]
[0010] In view of the above, the present invention is designed as follows: "A learning processing apparatus for obtaining a new second learning model from an existing first learning model, characterized in that it comprises: an input unit, which acquires a first learning model generated in advance by learning a first learning data set and an unpruned neural network; an important parameter specifying unit, which initializes the neural network of the learning object using the first learning model and the neural network, and specifies the importance of the parameters of the initialized neural network in the recognition process using the second learning data set and the initialized neural network; a new model generation unit, which performs pruning processing by deleting unimportant parameters from the initialized neural network using the importance of the parameters and generates a second neural network; a learning unit, which learns the second neural network using the second learning data set; and an output unit, which outputs the learned second neural network as a second learning model."
[0011] Furthermore, in this invention, it is designed as: "A learning processing method for obtaining a new second learning model from an existing first learning model, characterized by comprising the following steps: obtaining a first learning model generated in advance by learning a first learning data set and an unpruned neural network; initializing the neural network of the learning object using the first learning model and the neural network; specifying the importance of the parameters of the initialized neural network in the identification process using the second learning data set and the initialized neural network; using the importance of the parameters to perform pruning processing to delete unimportant parameters from the initialized neural network and generate a second neural network; learning the second neural network using the second learning data set; and using the learned second neural network as the second learning model."
[0012] Furthermore, this invention is designed as follows: "A learning processing method for obtaining a new second learning model from an existing first learning model, characterized in that: an unpruned neural network of the learning object is initialized using an unpruned neural network and a first learning model; and an importance of the parameters of the initialized neural network in the recognition process is determined using a second learning dataset and the initialized neural network; pruning is performed according to the importance; and a second learning model is obtained by learning the pruned neural network using the second learning dataset." [Effects of the Invention]
[0013] According to the present invention, even if the recognition processing is performed in multiple ways, and the model of the neural network used is switched according to the execution environment, the learning time of the lightweight model can still be shortened, and a rapid recognition processing function can be provided.
Implementation Method
[0014] The following uses drawings to illustrate embodiments of the present invention. [Embodiment 1]
[0015] Figures 1 and 2 are illustrations of one example of the utilization mode of the neural network of the learning processing device of the present invention. Although these examples are illustrated for image processing performed by the learning processing device 100, they are applicable even if the object of processing is natural language processing, voice recognition, or other signal processing.
[0016] In the case of FIG1, the learning processing device 100 inputs the input image 101 and then sends it to a plurality of image processing units 102 (102-1, 102-2, 102-3), and the image processing unit 102 performs recognition processing on the input image 101.
[0017] The processing in the image processing unit 102 includes image processing such as image classification, object detection, semantic segmentation, and automatic captioning. Image processing units 102-1, 102-2, and 102-3 each perform different recognition processing. For example, in the case of product image inspection, for the given image, image processing unit 102-1 performs image processing 1 to classify the product into good or defective products, image processing unit 102-2 performs image processing 2 to detect foreign objects mixed into the product, and image processing unit 102-3 performs image processing 3 to segment the product shape. As a result, the processing result unit 103 (103-1, 103-2, 103-3) obtains the processing results for each part.
[0018] The image processing units 102-1, 102-2, and 102-3 are learning models containing neural networks that are specialized for each recognition process. In the case of applying multiple image processing methods to an input image 101, and using different neural networks in each image processing, a lightweight model must be learned for the multiple neural networks.
[0019] In the learning processing apparatus 100 configured in FIG1 of the present invention, it is intended to add an image processing unit 102-4 (lightweight model) and a processing result unit 103-4 for obtaining the added image processing result 4 when a new inspection item occurs.
[0020] Furthermore, even with the same recognition processing, there may still be cases where multiple lightweight models need to be learned. Figure 2 illustrates a case where each learning model in the image processing units 102-1, 102-2, and 102-3 of Figure 1 becomes an optimized learning model in response to changes in the environment, such as in image measurement. The image processing units 102' (102-1, 102'-2, 102'-3) are configured to use learning models (environmental models) that have been optimized in response to changes in the environment. In this invention, it is envisioned that the configuration in Figure 1 is changed to the configuration in Figure 2, which adds an environmental model.
[0021] For example, when images are obtained through lenses of different magnifications, the learning model required for processing is different from the learning model under normal magnification. Therefore, it is necessary to construct a new neural network that specializes the images obtained through lenses of different magnifications, and a new configuration as shown in Figure 2 must be obtained.
[0022] As shown in Figure 2, the environment models 102' (102-1, 102'-2, 102'-3) perform the same image processing as the learning models 102 (102-1, 102-2, 102-3) in Figure 1, but the model executed changes depending on the relevant environment of the input image 101. The environment referred to here is the location or condition in which the input image 101 is acquired, or the location or condition in which the recognition processing is performed, etc. For example, if it is outdoor image data, the model executed will change depending on conditions such as daytime or nighttime; if it is an image inspection system, the model will change depending on the type of object being inspected, etc.
[0023] In addition, the processing results 103' (103'-1, 103'-2, 103'-3) are the outputs of the environment model 102' (102-1, 102'-2, 102'-3). The processing results 1', 2', and 3' are different from the results in Figure 1 because the neural networks executed are different.
[0024] Even when using the environment-specific models 102' (102-1, 102'-2, 102'-3), it is still necessary to learn multiple lightweight models. Especially in the case of image inspection systems, there are infinitely many combinations of object types or inspection process types, so the combination of environment-specific models and image processing becomes enormous. Therefore, it is necessary to efficiently obtain lightweight models.
[0025] In addition, although this embodiment shows an example where there are 3 image processing and 3 environment models, it is acceptable as long as there is more than 1.
[0026] As a method for learning and creating lightweight models, pruning is known. Figure 3 is a schematic diagram of the pruning caused by the previous method. By performing pruning, the learning model can be made lightweight.
[0027] Typically, pruning begins with learning the unpruned neural network 204-0 using a learning dataset 301 via learning process 201-0. Then, pruning process 202-1 identifies and removes unwanted parameters from the previously unpruned neural network 204-0, outputting the pruned neural network 204-1. The unwanted parameters identified during pruning refer to parameters with smaller absolute values.
[0028] The neural network 204-1, whose parameters have been deleted through pruning, has a lower recognition accuracy compared to the undressed neural network 204-0. Therefore, the undressed parameters of the pruned neural network 204-1 are updated through relearning process 203-1, so that the reduced recognition accuracy can be brought closer to the value of the undressed neural network 204-0. Subsequently, if the pruning amount is insufficient or if multiple pruning processes are set, a pruned neural network 204-2 is obtained through pruning process 202-2 and relearning process 203-2. This pruning process and relearning process can be repeated more than twice. In addition, in relearning processes 203-1 and 203-2, the same learning dataset 301 used in the initial learning process 201-0 is used for learning.
[0029] Thus, in the pruning process of the previous method, unnecessary parameters are identified after learning, and after deleting the unnecessary parameters, relearning is performed, and this process is repeated. This is because, in order not to adversely affect the recognition performance during lightweighting caused by pruning, parameters must be gradually deleted. In the processing caused by this procedure, since the repeated learning process occurs multiple times, the time required for learning the lightweight model will increase. Therefore, especially in cases like those shown in Figures 1 and 2 where multiple lightweight models need to be prepared, the preparation period becomes long, which is the main reason for the delay in providing the recognition function.
[0030] Figure 4 is an illustration of the processing overview of the learning processing apparatus according to the present invention. In this processing, a first model (network) prepared in advance is used to generate a second model (network) suitable for another processing. This relationship is, for example, in Figure 1, any model (network) of the image processing unit 102 (102-1, 102-2, 102-3) is used as the first model, and a new image processing unit 102-4 is used as the second model to generate the network; or any model (network) of the image processing unit 102 (102-1, 102-2, 102-3) is used as the first model, and a new environment model 102' in Figure 2 is used as the second model to generate the network.
[0031] Therefore, the first model, which forms the basis for the creation of the new model, is a pre-learning model 302 prepared in advance. The pre-learning model 302 is generated using a learning object that has been specialized for the pre-learning model 302, namely the first learning data set 301A. In this figure, 303 is an unpruned neural network prompted by the operator.
[0032] In the important parameter specification section 304, for the first model, namely the pre-learning model 302, a second learning dataset 301B, which is a learning object specialized for the newly created second model, is applied. Here, important parameters in the second learning dataset 301B are specified from the pre-learning model 302, extracting only the important parameters. Important parameters here mean specifying the parameters of the second model that are important in the identification of the second learning dataset 301B.
[0033] Subsequently, in the new model generation unit 306, an unpruned neural network 303 is generated, and a neural network 305 is generated that extracts only the parameters of the second model that are important for identification from the second learning dataset 301B. This neural network 305 is learned through the learning process 307, and a lightweight model can be obtained in a short period of time. The second learning dataset 301B is used in the learning process 307.
[0034] The method of learning by using a model learned from a different dataset (learning dataset 301A) as an initial value without applying pruning is called transfer learning. In this invention, pruning is applied during transfer learning so that the previously necessary learning process 201-0 can be made unnecessary, thus reducing the number of learning iterations and achieving short-term learning of a lightweight model.
[0035] Although the present invention is basically performed as described above, there are several aspects that require ingenious design in its implementation.
[0036] The first point that needs to be cleverly designed and improved is that pruning is difficult to apply during transfer learning in the previous method. Generally speaking, when performing pruning processing 202 in the learning processing 201-0 of Figure 3, a normalization suitable for pruning processing 202 is applied in order to minimize the reduction in accuracy. For example, if the parameter with the smaller absolute value is deleted in pruning processing 202, a normalization that reduces the absolute value of the parameters of a neural network called weight decay is applied in learning processing 201-0. The loss function of the neural network containing weight decay is shown in equation (1).
[0037] In equation (1), the first term on the right-hand side of the loss function L constituting the neural network is the loss function Lr defined according to each learning process of the identification process, and the second term is the weight decay term. λ is the coefficient of weight decay, and θ is the parameter of the neural network. In order to remove more parameters, the coefficient λ of weight decay must be set to a large value.
[0038] However, the larger the weight decay coefficient λ is, the more many parameters will become small values. Therefore, it can only perform recognition performance on the already learned dataset and is not suitable for transfer learning. Therefore, the pre-learning model 302 is not a model with strong weight decay applied to a specific learning dataset 302, but a neural network with many parameters having non-zero values, which is more ideal.
[0039] In such a case, the important parameter specifying section 304 cannot specify the important parameter by the absolute value of the parameter.
[0040] Regarding this improvement, in the embodiments of the present invention, the following corresponding measures are also taken.
[0041] In Figure 4, generally speaking, in the pre-learning model 302 learned from the learning dataset 301A, which is different from the learning dataset 301B, there are parameters that are identified as valid and parameters that are not needed in the learning dataset 301B. The important parameter specific part 304 extracts the parameters that are identified as valid in the learning dataset 301B by analyzing the Hessian matrix. The Hessian matrix is the second derivative of the parameters with respect to the loss function L. Therefore, the impact of deleting each parameter on the loss function can be calculated, for example, by multiplying the square of the value of each parameter with the diagonal component of the Hessian matrix.
[0042] (2) represents the calculated formula. The subscript q indicates that this is the value of the q-th parameter. H is the Hessian matrix, and the subscript qq indicates that this is the value of the q-th row and q-th column. By removing parameters from the smaller value to the desired pruning amount, the parameters that are effective in the identification of the learning dataset 301B can be extracted from the pre-learning model 302.
[0043] Thus, in this invention, when a parameter is deleted, the impact on the behavior of the neural network is evaluated and used as the importance of each parameter. The greater the importance of a parameter, the more the neural network's recognition performance will decrease when it is deleted. Therefore, pruning is performed starting with parameters with lower importance, thereby suppressing the decrease in the neural network's recognition performance and deleting many parameters.
[0044] Furthermore, the second point related to this point, which requires clever design and improvement, is that, generally speaking, multi-layer neural networks have a large number of parameters, making it difficult to directly calculate the Hessian matrix.
[0045] In such a case, the impact of each parameter deletion can also be calculated using the second-order dynamics of the first derivative of the loss function L, i.e., the Fisher information matrix, or an approximation of the Fisher information matrix, i.e., the Kronecker-Factored Approximated Curvature. Alternatively, the impact can be calculated using the statistics of the first derivative of the loss function L or the statistics of the product of the first derivative and the parameters. For example, the absolute value of the product of the expected value of the gradient and the parameter values can be used as the impact of each parameter deletion on the loss function.
[0046] Furthermore, in the case of convolutional neural networks, the importance calculated in parameter units can also be integrated according to each dimension called a channel or filter to serve as the importance of each channel or filter.
[0047] Alternatively, the importance of each parameter can be evaluated using the relaxed Bernoulli distribution shown in Example 4.
[0048] Figure 5 is a diagram comparing the effects of the present invention and the prior method. The learning curve 401 caused by the present invention and the learning curve 402 caused by the prior method are both graphs, with the horizontal axis representing the number of parameter updates during learning and the vertical axis representing the recognition performance.
[0049] According to the present invention shown in Figure 5 on the left, the pre-learning model 302 identifies only the important parameters in the learning dataset 301B and directly learns the lightweight model 306. Therefore, as with the learning curve 401, only one learning curve is drawn.
[0050] In contrast, in the previous method, because learning processing 201-0 or more relearning processing 203 is required, more than two learning curves are presented, such as learning curve 402. Here, the dashed line near the center of learning curve 402 represents the effect of reduced accuracy caused by pruning. Thus, according to the present invention, a lightweight model can be obtained in one round of learning, thereby significantly shortening the time required to obtain a lightweight model.
[0051] Figure 6 illustrates a functional configuration example of the learning processing apparatus of the present invention, and Figure 7 illustrates a processing flow example of the learning processing method of the present invention. These figures illustrate the configuration and process of learning a lightweight model using the learning dataset 301 and the pre-learned model 302.
[0052] First, the functional configuration example diagram of the learning processing device of the present invention shown in FIG6 represents the processing content in the calculation unit when the processing of the present invention is implemented by a computer as the main processing function. Here, the pre-learning model 302 that has been learned from the learning dataset 301A and the unpruned neural network 303 are taken as objects.
[0053] In the important parameter specification section 304, the neural network 303 of the learning object is first initialized using the pre-learning model 302 and the unpruned neural network 303. Through this initialization, the parameters of the pre-learning model 302 are reflected in the unpruned form of the neural network 303, so the initialized neural network 303 can be said to be equivalent to the pre-learning model 302. In the important parameter specification section 304, the initialized neural network 303 is then used with the learning object, namely the second learning data set 301B, which is specialized for the newly created second model 305, to specify the parameters of the initialized neural network 303 that are important in the recognition process.
[0054] The pruning part 306 is a neural network 306 that has never been pruned, which is a neural network 305 that has been pruned and only extracts the parameters of the second model that are important for identification from the second learning data set 301B.
[0055] In the learning processing unit 307, the neural network 305 is learned using the second learning data set 301B.
[0056] In the model evaluation unit 503, the computational accuracy of the neural network 305 (second model) is evaluated using the evaluation dataset 504. The neural network 305 that can perform according to the benchmark of the evaluation dataset 504 will ultimately be used as the second model (metric model).
[0057] In the example of the processing flow of the learning processing method of the present invention shown in FIG7, the learning process is first started in step S601. This starts when the operator sets the necessary items for learning the lightweight model and then performs the learning by the operator.
[0058] Step S602 corresponds to a part of the important parameter specific section 304 in Figure 6. Here, the neural network of the learning object is initialized using the unpruned network structure 303 input by the operator and the pre-learning model 302. The unpruned network structure 303 here is the network structure given by the operator for image processing of the object to perform lightweight model learning. Here, the initialization of the neural network of the learning object is to determine the initial values of the parameters of the given network structure 303. Part or all of the aforementioned neural network of the learning object is initialized by copying the parameters of the pre-learning model 302. In the aforementioned neural network of the learning object, the parameters of the pre-learning model 302 that are not copied are determined by random numbers or constants set by the operator.
[0059] The initialization of the neural network using the parameters of the pre-learning model 302 is generally referred to as transfer learning. Furthermore, the initialized network structure 303 is an unpruned neural network structure that reflects the parameters of the pre-learning model 302. Therefore, the initialized network structure 303 can be said to be formed by reflecting the lightweight form of the pre-learning model 302 onto the original unpruned network structure.
[0060] Here, the operator determines which parameters to copy from the prior learning model 302 within the neural network of the prior learning object. The operation of copying parameters from the prior learning model 302 only for a portion of the prior learning object's neural network occurs when the image processing function of the learning object differs from the image processing function of the prior learning model. For example, if the learning object is an object detection function and the prior learning model 302 is an image classification function, then only the portion of the neural network called the feature extractor, which extracts image features from the input image, will have its parameters copied to the neural network used for object detection. Furthermore, even if both are the same image classification function, sometimes only a portion of the neural network called the feature extractor will have its parameters copied, while the subsequent layers predicting the classification results are initialized using other methods.
[0061] Step S603 corresponds to part 304 of the important parameter specificity section in Figure 6. Here, the importance of each parameter of the neural network initialized in step S602 is evaluated using the neural network and learning dataset 301B initialized in step S602. The evaluation of importance here is as described in the explanation of Figure 4.
[0062] Step S604 corresponds to the pruning section (new model generation section in Figure 4) 306 in Figure 6. Here, based on the importance of each parameter evaluated in step S603, parameters with low importance are deleted from the neural network initialized in step S602. The method for determining the parameters to be deleted is selected by the operator, and may include deleting parameters other than those with a high importance ratio, or deleting parameters with an importance below a certain threshold, etc.
[0063] Step S605 corresponds to the learning unit 307 in Figure 6. Here, the learning dataset 301B is used to learn the parameters of the lightweight neural network obtained in step S604. This is done using methods such as probability gradient descent, which are commonly used in neural network learning.
[0064] Step S606 corresponds to the model evaluation unit 503 in Figure 6. Here, the performance of the neural network learned in step S605 is evaluated using the evaluation dataset 504. The performance evaluation of the neural network is to evaluate the level of performance that the obtained lightweight model 505 can achieve in the image processing of object recognition.
[0065] In step S607, the neural network learned in step S605 is output as a lightweight model 505. At this time, the recognition performance or the execution time of recognition processing evaluated in step S606 can also be output at the same time.
[0066] In step S608, the learning process ends.
[0067] Thus, in this invention, the learning data set 301B and the evaluation data set 504 are identified from the pre-learning model 302 as important parameters for identification. By transferring only the important parameters, a lightweight model can be constructed during initialization. Therefore, a lightweight model 505 with good identification accuracy can be obtained through one round of learning. [Example 2]
[0068] In Example 2, when there are multiple learning models, the desired lightweight model is generated first. As for which learning model is appropriate, the judgment criteria are made in advance for reference in actual application.
[0069] According to Embodiment 1 of the present invention, it is not necessary to perform long-term learning to evaluate the extent to which the pre-learning model 302 maintains parameters that are important to the learning dataset 301B and the evaluation dataset 504. In the case of preparing multiple learning models, there are also multiple candidates for the model used as the pre-learning model 302. Generally speaking, the performance of the neural network used for learning changes with the model used as the pre-learning model 302. Therefore, it is important to select an appropriate pre-learning model 302 in order to learn a lightweight model in a short period of time.
[0070] However, in the previous method, if the neural network is not actually learned using the learning data set 301B and the learning unit 306, and the recognition performance is evaluated using the evaluation data set 504 and the model evaluation unit 503, it is impossible to determine the quality of each pre-learned model.
[0071] Therefore, given a plurality of candidate pre-learning models 302, if the best model is to be selected as the pre-learning model 302, all candidates must be learned and evaluated. This would result in a large amount of time required to learn the lightweight model 305. Therefore, selecting a suitable model from the plurality of candidate pre-learning models 302 without learning is important for learning a high-precision lightweight model in a short time.
[0072] Here, since the unpruned non-lightweight model becomes an indicator of the recognition performance that the neural network of the object can achieve, it is necessary to learn and maintain the aforementioned non-lightweight model for model management.
[0073] In Embodiment 2 of the present invention, a portion of the function of the learning processing device 100 shown in FIG. 6, namely the important parameter specifying unit 304, is used to pre-create the importance evaluation table 700 of FIG. 8. The pre-created importance evaluation table 700 is used as a reference for selecting an appropriate pre-learning model for practical application. The device configuration and processing flow required to pre-create the importance evaluation table 700 will be described below with reference to FIGS. 9 and 10.
[0074] Figure 8 is an illustration of an example of an importance evaluation table for multiple candidate pre-learning models 302. This table is provided to the operator as information needed to extract pre-learning candidates suitable for transfer learning, when the learning dataset 301B, network structure 303, and pruning-related settings have been selected. Specifically, the information includes the sum of the importance of each pre-learning model prepared for each dataset used in the learning process, and the sum of the importance after pruning, for the network structure of the multiple pre-learning models.
[0075] In the importance evaluation table 700 of Figure 8, the pre-learning model category 701 represents the network category of each pre-learning model. For example, when the object recognition processing is object detection, as a pre-learning model 302, when using an image classifier, only a portion of the neural network called the feature extractor is transferred; when the pre-learning model 302 also uses an object detector, in addition to the feature extractor, the portion of the neural network used for object detection is also transferred. Therefore, even if the network structure 303 is given, there can still be multiple categories of pre-learning models. Furthermore, even if only the feature extractor is transferred, the number of categories of pre-learning models may still be multiple depending on the number of layers or the number of parameters in each layer. Therefore, the category 701 of the pre-learning model is explicitly set here.
[0076] The pre-learning dataset 702 represents the dataset used in the pre-learning model (corresponding to the learning dataset 301A in Figure 4). When performing transfer learning, it is important for the high accuracy of the neural network that the features learned by the transfer source are consistent with the features held by the dataset 301A of the learning object.
[0077] The total importance score 703 is the sum of the importance scores of each of the pre-learning models 302. This is the sum of the importance scores of each parameter evaluated using the learning dataset 301A and the importance parameter specific unit 304. Furthermore, since the number of parameters varies depending on the pre-learning model 302, the model with more parameters tends to have a larger total importance score. Therefore, it is also possible to use the average importance score obtained by dividing the total importance score of the pre-learning models by the number of parameters of each pre-learning model.
[0078] The total importance after pruning, 704, is the sum of the importance of the parameters remaining after pruning using the pruning-related settings given by the operator. This value can also be the average importance, just like the total importance, 703.
[0079] Importance is the evaluation value of the adverse effects on recognition performance when parameters are deleted. Therefore, the larger the total importance of 703, the more parameters that are important to the recognition and processing of the learning object are contained.
[0080] Therefore, if the model with the larger total importance 703 is selected as the pre-learning model 302, a high-precision model can be easily learned. Furthermore, during the learning of the lightweight model, several parameters are removed through pruning. Therefore, by evaluating the total importance 704 after pruning, the importance of the remaining parameters after pruning for the recognition of the learning dataset 301B can be assessed. Therefore, a model with a larger total importance 704 after pruning can learn a lightweight and high-precision model when learning the lightweight model 505. By checking such a table and selecting a model where both the total importance 703 and the total importance 704 after pruning are larger, a lightweight and high-precision model can be learned.
[0081] Figures 9 and 10 illustrate the device configuration and processing flow example for generating an importance evaluation table 700 for the multiple candidates of the prior learning models 302 shown in Figure 8.
[0082] First, the functional configuration example diagram of the learning processing device described in Embodiment 2 of the present invention shown in FIG. 9 is formed by adding the functions of a learned model memory unit 801, a pre-learning model candidate extraction unit 802, a pre-learning model selection unit 803, and an importance evaluation result memory unit 804 to the learning processing device 100 of FIG. 6. The learned model memory unit 801 and the importance evaluation result memory unit 804 are memory units such as computer databases. The learned model memory unit 801 stores information such as models (networks) 102-1, 102-2, and 102-3 of FIG. 1 as pre-learning models. The importance evaluation result memory unit 804 stores the evaluation results of important parameters extracted from the important parameter specific unit (pruning unit) 304 of FIG. 6. Furthermore, the importance evaluation data of FIG. 8 is ultimately stored and remembered in the importance evaluation result memory unit 804. In addition, the pre-learning model candidate extraction unit 802 and the pre-learning model selection unit 803 represent the processing and judgment content performed by the operator.
[0083] The flowchart in Figure 10 uses the hardware resources and software resources shown in Figure 9 to perform the following processes one by one. First, in the initial step S901 of Figure 10, the importance evaluation process is started for the plurality of candidates for pre-learning models 302. This starts according to the execution sequence instructed by the operator.
[0084] In step S902 (corresponding to the processing of the pre-learning model candidate extraction unit 802), the network structure 303 given by the operator is used to extract the candidates for the pre-learning model 302 from the learning model memory unit 801. The learning model memory unit 801 stores the model ID, model name, model network structure, learning dataset 301A, and recognition performance achieved according to each model. In step S902 (corresponding to the processing of the pre-learning model candidate extraction unit 802), the models containing network structures that can be transferred to the given network structure 303 are extracted from the learned model memory unit 801 as pre-learning model candidates and output to the pre-learning model selection unit 803.
[0085] In step S903 (corresponding to the processing of the pre-learning model selection unit 803), a model that has not yet undergone importance evaluation is selected as the pre-learning model 302 from the pre-learning model candidate extraction unit 802.
[0086] In step S904 (processing corresponding to the important parameter specific part 304), the neural network of the learning object is initialized using the network structure 303 and the pre-learning model 302 given by the operator. This is performed in the same way as step S602 shown in FIG7.
[0087] In step S905 (processing corresponding to the important parameter specific part 304), the importance of each parameter of the neural network initialized in step S904 is evaluated using the learning dataset 301B. This is performed using the same method as step S603 shown in FIG7.
[0088] In step S906 (corresponding to the processing of the important parameter specific unit 304), it is confirmed whether the evaluation performed in steps S904 and S905 has reached the number of times set by the operator. If the set number has been reached, proceed to step S907; otherwise, return to step S904 and repeat until the set number is reached. The set number here only needs to be one or more times and can be any number. This is because the neural network initialization performed in step S904 uses random numbers, so multiple evaluations are performed to suppress the influence of random numbers.
[0089] For example, when learning a neural network for object detection, if a model that has been previously learned through image classification is being transferred, generally only the parameters of the feature extractor are copied, while the other parts are initialized with random numbers. The first or second order gradient information of the feature extractor used for importance evaluation is affected by the initialization of other neural networks located after the feature extractor. Therefore, in this case, in order to correctly evaluate the importance of each parameter contained in the feature extractor, multiple initializations with random numbers can be performed, and multiple importance evaluations can be performed. In fact, the importance calculated using statistical information in step S907 from the importance evaluated multiple times can be used as the importance of each prior model.
[0090] In step S907 (the processing corresponding to the important parameter specific section 304), the importance of each parameter is calculated from the importance evaluated in steps S904, S905, and S906. This can be done using the result of the number of times the sum of importance is the largest, or by using the average of the evaluations. Using the number of times the sum of importance is the same as using the evaluation value of the number of times the optimal initial value is obtained. In this case, the initial values of each parameter can also be memorized and then used during actual learning. Furthermore, using the average of each time as the importance is equivalent to emphasizing and utilizing parameters that are less susceptible to randomness.
[0091] In step S908 (processing corresponding to the importance parameter specific unit 304), the evaluation conditions used are linked to the importance evaluated in step S907 and stored in the importance evaluation result memory unit 804. The evaluation conditions used here are information about the pre-learning model 302, the number of times importance evaluation has been performed, or the method used for importance evaluation in step S907. The importance evaluation result memory unit 804 stores: information about the learning dataset 301B used in each evaluation, information about the evaluated importance, the model ID, name, network structure of the models used as pre-learning models, the dataset used in pre-learning, and the recognition performance during pre-learning. The importance information here can be all the importance of each parameter, or it can be statistically analyzed information. Through the processing in step S908, in the importance evaluation table 700 of Figure 8, data on the total importance 703 and the total importance 704 after pruning will be recorded for the candidate pre-learning model.
[0092] In step S909 (corresponding to the processing of the important parameter specific unit 304), it is confirmed whether the evaluation of all the pre-learning model candidates extracted in step S902 has been completed. If it has been completed, proceed to step S910. If it has not been completed, repeat the process from step S903 until it is completed.
[0093] In step S910, the importance evaluation process ends. After all necessary pre-preparation model processing is completed, the data in Figure 8 will be prepared in the importance evaluation result memory unit 804.
[0094] As described above, according to the present invention, during transfer learning, only the important parameters are identified and learned from the pre-learning model 302, thereby a lightweight model 306 can be obtained in just one round of learning, and the lightweight model 306 can be learned in a short period of time.
[0095] Furthermore, as shown in Figures 8, 9, and 10, the effectiveness of each pre-learning model can be evaluated simply by assessing its importance without actually learning it, thus shortening the learning time of the lightweight model 305. [Example 3]
[0096] In Example 3, a corresponding action is taken in case that the prior learning model 302 does not learn the important features in the identification and processing of the learning object during transfer learning.
[0097] In Embodiment 1 of the present invention, during transfer learning, the learning time required for the lightweight model 505 is shortened by transferring only the parameters that are important in the identification process of the learning object. However, during transfer learning, there may be a situation where the pre-learning model 302 fails to learn the features that are important in the identification process of the learning object. In such cases, it is difficult to learn a high-precision lightweight model 505 by transferring only the important parameters from the pre-learning model 302.
[0098] Figure 11 is a diagram of the partial re-initialization process introduced to solve such a problem. Figure 11 illustrates the process corresponding to Figure 4, in which partial re-initialization process 1001 and pruning process 1003 are added to the program in Figure 4.
[0099] In this series of processes, as previously explained, the important parameter specification unit 304 will specify the important parameters, and the network 305 will be defined at this time. However, during transfer learning, the pre-learning model 302 did not learn the important features in the identification process of the learning object. Therefore, the network 305 is insufficient, based on this premise. The so-called insufficiency here means that the network is too simple, or has not yet formed the necessary paths, etc.
[0100] Therefore, in the partial reinitialization process 1001, after the important parameter designation unit 304 has designated the important parameters, only the pruned parameters are reinitialized. Thus, if the partial reinitialization process 1001 is applied to the neural network 305, which retains only the parameters with high importance, then, as shown in neural network 1002, the neurons will be reinitialized as indicated by the dashed lines. This reinitialization is implemented in a pruning-friendly manner, allowing for pruning with minimal impact on accuracy after learning by the model generation processing unit 306 through pruning process 1003. This pruning-friendly reinitialization includes, for example, initializing each parameter to zero. Alternatively, random numbers with small absolute values can be used.
[0101] Figures 12 and 13 are illustrations of the functional configuration example and the processing flow example of the learning processing apparatus of the present invention with partial reinitialization as shown in Figure 11. Since these are configurations and flows formed by adding partial reinitialization processing to the functional configuration example and processing flow example of the present invention shown in Figures 6 and 7, only the differences are described.
[0102] In the flowchart of FIG13, in step S1201, the neural network initialized in step S604 by the pruning unit 502 according to the importance evaluated in step S603 has its low importance parameters deleted, and is then reinitialized by the partial reinitialization unit 1101. As mentioned above, this is done using random numbers such as zero or small absolute values, thereby implementing reinitialization that is easy to prune.
[0103] In step S1202, the pruning unit 502 removes unnecessary parameters from the neural network learned in step S605. The reinitialization process performed in step S1201 is implemented in a way that facilitates pruning, thus minimizing the impact on accuracy.
[0104] By combining partial reinitialization processing, if the pre-learning model 302 holds parameters that are important for identification in the learning dataset 301B, the transferred parameters will be learned; if it does not hold such parameters, the partially reinitialized neurons will be learned. Furthermore, if all the features important for identification in the learning dataset 301B have already been learned by the pre-learning model 302, the partially reinitialized neurons can be easily pruned. Also, after pruning in step S1202, the neural network can be learned again in step S605. [Example 4]
[0105] Although Embodiment 3 describes the use of a smaller value to reinitialize the parameters in the partial reinitialization process 1001, the same method as normal learning can also be used for initialization, and the pruning area can be determined in a different way. Embodiment 4 of the present invention describes pruning caused by a masking layer.
[0106] The concept of pruning caused by the masking layer is shown in Figure 14. Figure 14 illustrates the configuration when a masking layer 1304 is applied to a convolutional neural network commonly used in image processing. Typically, multiple convolutional layers 1302 are applied in a convolutional neural network, but Figure 14 shows the configuration focusing on a single convolutional layer 1302. In a typical convolutional neural network, a convolutional layer 1302, a normalization layer 1303, and an activation layer 1305 are applied to the input 1301 to obtain the output 1306.
[0107] Let the processing and parameters of the convolutional layer 1302 be f and w, let the processing and parameters of the normalization layer 1303 be g and θ, let the processing of the activation layer 1305 be a, let the input be x, and let the output be y. Then the series of processing of the convolutional layer 1302 can be represented by equation (3).
[0108] The x and y axes here are 3rd-order tensors, representing the width, height, and feature channel dimensions of the image features. Generally, when speeding up the convolutional layer 1302, deleting channels in terms of feature channels is effective. For example, if the original convolutional layer 1302 has 128 channels, 10 channels that do not affect the recognition performance will be deleted, and 118 channels will be output.
[0109] In Embodiment 4 of the present invention, the deletion of the channel is performed by the mask layer 1304. If the processing parameters of the mask layer 1304 are m and ν, then the series of processing of one of the convolutional layers 1302 shown in equation (3) becomes as shown in equation (4).
[0110] In order to delete the feature channel units of the convolutional layer 1302, the masking layer 1304 learns the parameter ν. The parameter ν is a vector with the same dimension as the feature channel, which takes the value of 0 or 1. The processing of the masking layer 1304 is to directly output the input feature map when the parameter ν is 1, and to output all width and height values as 0 when the parameter ν is 0. If such a parameter ν can be learned, it will not affect the recognition performance, and the calculation of the part where the parameter ν is 0 will be deleted after learning. That is, if the hidden variable obtained by applying the convolutional layer 1302 and the normalization layer 1303 to the input is denoted as h, and the hidden variable obtained by applying the masking layer to the hidden variable h is denoted as h', then the elements of h' are given by equation (5).
[0111] In equation (5), the hidden variable h' of the masking layer 1304 can be represented by the hidden variable h of the input to which the convolutional layer processing 1302 and the normalization layer 1303 are applied. Therefore, it can be represented by equation (6) which sets the width i, height j and feature channel position k of the hidden variable belonging to the 3rd order tensor to be variable.
[0112] Here, since the parameter ν takes the value of 0 or 1, it can be expressed by equation (7).
[0113] However, the discrete parameter ν, which takes values of 0 or 1 as shown in equation (7), cannot be learned using the probability gradient method commonly used in neural network learning. This is because the gradient of the discrete parameter ν does not propagate. Therefore, it is difficult to learn which feature channel of parameter ν should be set to be effective.
[0114] To solve this problem, in Embodiment 4 of the present invention, the parameter ν is further sampled from the relaxed Bernoulli distribution during learning. The relaxed Bernoulli distribution is formed by relaxing the discrete distribution, i.e., the Bernoulli distribution, into a continuous distribution, and can be learned using the probabilistic gradient descent method. It is also generally known as Gumbel-softmax. When using the relaxed Bernoulli distribution, the parameter ν is sampled as a continuous value from 0 to 1, such as 0.1 or 0.5. The masking layer 1304 calculates and outputs the product of the sampled parameter ν with all channels corresponding to the input feature map. Therefore, during learning, the parameter ν of channels that are not needed for identification will gradually become smaller values through the probabilistic gradient descent method, while the parameter ν of channels that are important for identification will gradually become larger values. The masking layer 1304 learns the logit of the relaxed Bernoulli distribution in order to sample the parameter ν. The logit parameter is the same as that of the Bernoulli distribution. A larger logit value increases the probability of generating a value of 1 in the Bernoulli distribution, but increases the probability of generating a value close to 1 in the relaxed Bernoulli distribution. During inference after learning, by using only feature channels with logits greater than a predetermined value in the relaxed Bernoulli distribution, recognition processing can be performed with less computation and without reducing recognition accuracy. That is, the output of the masking layer 1304 becomes as shown in equation (8).
[0115] Here, ν' is as shown in equation (9), a continuous variable between 0 and 1, sampled from a relaxed Bernoulli distribution RB with logit l and temperature t as parameters. In this way, logit l can be learned by probability gradient descent, and the mask can be learned according to the feature channel units that are pruned.
[0116] In the relaxed Bernoulli distribution, by learning in a way that reduces the effective number of channels to a predetermined value, it is possible to learn that only the logit values of important feature channels will increase, while the logit values of parameters that do not contribute to recognition will decrease. Thus, at the end of the learning process, the system will be learning with fewer feature channels, resulting in a situation similar to the relearning after pruning shown in Figure 2. Therefore, pruning can be performed without reducing recognition accuracy by utilizing only feature channels with larger logit values, without the need for relearning. The method of learning in a way that reduces the effective number of channels to a predetermined value refers to a method where a certain percentage of the logit values in the relaxed Bernoulli distribution held by the entire neural network increase the logit value, while the remaining channels decrease it. For example, in the feature channels corresponding to a certain percentage of the logit values, the Kullback-Leibler information between the Bernoulli distribution and the distribution where the probability of generating 1 is set to a higher value is minimized. That is, as in equation (10), a term related to the Logit is added to the loss function during learning.
[0117] The third term in equation (10) here refers to the Kullback-Leibler information between the Bernoulli distribution B(l) with the same logit as the relaxed Bernoulli distribution RB(l, t) and B(yl) which takes the logit corresponding to the ratio of the elements above the logit as the larger value.
[0118] Furthermore, the pruning treatment caused by the masking layer 1304 can also be applied independently without being combined with the important parameter specific parts.
[0119] Alternatively, the Rogert value of the relaxed Bernoulli distribution obtained by adding a masking layer can be used as the importance value evaluated in the importance parameter specific section 304, and the learning processing caused by the configuration shown in FIG6 can be performed. This can also be seen as the case where the larger the Rogert value of a feature channel, the greater the impact on the recognition performance of the neural network when the channel is deleted on a channel-by-channel basis.
[0120] Furthermore, when applying a masking layer to the configuration shown in Embodiment 3, during the reinitialization of the partial reinitialization unit 1101, the weights of the neural network are initialized using the same initialization method as usual, and the mounts of the masking layer are initialized with a smaller value than usual. Therefore, the reinitialized parameters can be initialized in a way that is easier to prune than other networks. [Embodiment 5]
[0121] Example 5 describes a method for applying a masking layer to a neural network with a complex network structure.
[0122] Figure 15 is a diagram illustrating a method for applying a masking layer to a neural network with a complex network structure.
[0123] In a neural network with a structure called Residual Shortcut or Residual Connection, as shown in Figure 15, multiple convolutional layers share the same feature map. For example, input 1401 is applied to convolutional layers 1402-a and 1402-b and transformed into a 3rd-order tensor of the same dimension, which is then summed. The sum of the outputs of these two convolutional layers is applied to convolutional layer 1402-c and then summed with itself. It is then applied to convolutional layer 1402-d and summed with itself, becoming output 1403. For simplicity, the normalization layer, masking layer, and activation layer other than the convolutional layers are omitted here, but they can also be applied to each convolutional layer. When multiple convolutional layers 1402-a~d share the same feature map, pruning may not be performed efficiently.
[0124] Figure 16 illustrates the relationship between the effectiveness of the four shared channels in the four layers of convolutional layers 1402-a to d in the neural network with the complex network structure shown in Figure 15 and the actual number of channels that can be pruned. The channel number 1404 in the first row of the table represents the number of the four feature channels of each convolutional layer 1402-a to d. Effective feature channels 1405-a to d represent the feature channels set as effective in each convolutional layer 1402-a to d. Here, a feature channel set as effective is a feature channel whose logit value held by the masking layer 1304 is above a threshold. Effective feature channels 1405-a to d are recorded as 1 in the cell corresponding to an effective feature channel when the logit value of that channel number is above a threshold, and as 0 when it is not an effective feature channel. For example, the effective feature channel 1405-a is such that the first and fourth elements of the Logit of the mask layer 1304 corresponding to channel numbers 1 and 4 are above the threshold, which are important feature channels for recognition processing.
[0125] The logical sum of 1406 represents the logical sum of the valid feature channels 1405-a~d of the shared convolutional layers of the feature map. Only channel number 3 is a 0, which is invalid in all convolutional layers 1402-a~d; other channel numbers are valid feature channels in some of the convolutional layers 1402-a~d and are therefore recorded as 1. In such cases, although there may be a situation where a certain feature channel is invalid in some convolutional layers, it can only be deleted if, like channel number 3, it is invalid in all of the shared convolutional layers 1402-a~d. Therefore, it is difficult to delete a large number of calculations.
[0126] Previous pruning methods evaluated the parameters to be pruned based on the values of the parameters of the convolutional layers or the importance of each parameter. Therefore, it is difficult to make the feature channels to be pruned consistent in multiple convolutional layers that share a feature map.
[0127] The masking layer 1403 can determine the feature channels that can be pruned efficiently even for layers with complex networks. This is done by making the logit values of the relaxed Bernoulli distribution of the masking layer 1304 attached to the convolutional layer that shares the feature map the same value. This is done, for example, by using the same parameters among the logit values of the masking layers attached to the convolutional layers 1402-a~d, or by learning in a way that reduces the amount of Kullback-Leibler information between each logit value. In addition, it can also be done in a way that reduces the amount of Kullback-Leibler information between statistics such as the average or maximum value of multiple logit values, or in a way that reduces the amount of Kullback-Leibler information between the logical sums representing the effective feature channels of multiple logit values as Bernoulli distributions with a probability that parameter ν will take the value 1, as in logical sum 1406. [Example 6]
[0128] Embodiment 6 illustrates a configuration example of a monitor screen suitable for the learning processing apparatus of the present invention. FIG17 is an illustration of the configuration of a screen required for implementing lightweight model learning. In the screen configuration example of FIG17, it is composed of various setting units, execution units, and result display units. The various setting units are composed of setting factors 1501 to 1507.
[0129] Within the setting section, the learning dataset setting section 1501 is the area for setting up the learning dataset 301B. Here, data is retrieved by specifying a memory area stored in a predetermined format. The evaluation dataset setting section 1502 is the area for setting up the evaluation dataset 504. Here, the same settings as the learning dataset setting section 1501 are performed. The learning condition setting section 1503 is the area used to set the conditions when the learning section (new model generation section) 306 learns from the neural network. These include, for example, the number of parameter updates during learning, learning rate scheduling, weight decay coefficients, and coefficients of various loss functions.
[0130] The network structure setting unit 1504 is the area for setting the network structure 303 to be learned. Here, the operator selects an appropriate network structure 303 according to the identification and processing of the learning object. The pre-learning model setting unit 1505 is the area for setting the pre-learning model 302A. Here, the operator selects or specifies a pre-learning model.
[0131] The pruning parameter setting unit 1506 is a field used to set pruning-related parameters. When pruning is used with the masking layer 1304, it sets the initial value of the relaxed Bernoulli distribution's logit or the initial value used to determine the masking probability of the logit, as well as the coefficients of the loss function for the masking layer 1304. Generally, in neural network learning, the initial learning focuses on raw image features such as edges or curves, and then gradually learns important and complex features to improve recognition performance. Therefore, setting the loss function for the masking layer too strong from the beginning of learning, or setting it so that many feature channels do not become effective, can lead to difficulties in learning. Therefore, in the early stages of learning, the coefficients of the masking loss function can be set to a very small value, or the percentage of the logit used to determine the masking layer 1304 that is allowed to become effective masking target values can be gradually relaxed as the learning progresses. For example, the learning process could be structured as follows: in the initial 30% of the learning, the coefficient of the masking loss function is set to zero; in the initial 50% of the learning, only 10% of the feature channels in the entire network are made effective; when the learning progress reaches 75%, up to 20% of the feature channels are made effective; and when the learning ends, 30% of the feature channels are made effective.
[0132] The important parameter specific condition setting unit 1507 is the area for setting the processing conditions related to the important parameter specific unit 304. Here, for example, the following are set as the importance evaluation function: the matrix product of the Hessian matrix and the parameters, the element product of the diagonal component of the Hessian matrix and the parameter vector, the absolute value of the element product of the gradient information and the parameter vector, the use of K-FAC as an alternative to the Hessian matrix, and the use of a masking layer, etc. In addition, the presence or absence of the ratio or partial reinitialization of the parameters to be transferred, and the execution method are set.
[0133] Next, the execution unit consists of an importance evaluation execution unit 1508 and a learning result display unit 1510. Here, once the importance evaluation execution unit 1508 is selected by the operator, the importance parameter specifying unit 304 will activate to perform an importance evaluation of each parameter of the pre-set pre-learning model 302. Once the lightweight model creation execution unit 1509 is selected by the operator, the learning unit 306 will activate to start the learning process from step S604 according to the pre-set conditions.
[0134] The learning result display unit 1510 is a field for displaying information related to the learning results. Here, it displays, for example, the achieved recognition performance or the time required for inference of the obtained lightweight model, the recognition performance relative to the learning time, and the change in the number of effective feature channels. Furthermore, at the point when the importance evaluation execution unit 1508 is selected and the evaluation is completed, a histogram of importance can also be displayed.
[0135] The operator uses this information to adjust the parameters until the desired recognition performance and inference speed are achieved.
[0136] Figure 18 is a diagram illustrating the structure of a screen required for evaluating the importance of multiple candidate pre-learning models. In this case, the screen is also composed of a setting section, an execution section, and a result display section.
[0137] The operator uses this screen to determine the pre-learning model to be used when learning the lightweight model 505 shown in Figure 17. The learning dataset setting unit 1501, network structure setting unit 1504, and important parameter specific condition setting unit 1507 are the same as those in Figure 17.
[0138] The evaluation pre-learning model setting unit 1601 sets the pre-learning model used to evaluate its importance. Although Figures 8 and 9 show an example where all the candidates for pre-learning models extracted by the pre-learning model candidate extraction unit 802 are evaluated, the operator can also use the evaluation pre-learning model setting unit 1601 to select the model to be evaluated for importance in order to shorten the evaluation time. Furthermore, the pre-learning models extracted by the pre-learning model candidate extraction unit 802 can also be displayed to the user here.
[0139] Once the importance evaluation batch execution unit 1602 is selected by the operator, the processing begins from step S903 of the importance evaluation process shown in Figure 10.
[0140] The pre-learning model evaluation result display unit 1603 displays the evaluation results of each pre-learning model. Here is an example of the table shown in Figure 8 and the histogram of the importance of the selected pre-learning models.
[0141] If the present invention is constructed using the above-described image, a lightweight model 505 can be learned efficiently.
[0142] This invention is not limited to the embodiments described above and may include various modifications. For example, the embodiments described above are detailed descriptions provided to illustrate the invention in an easily understandable manner, and are not limited to having all the described components. Furthermore, a part of the component of a certain embodiment may be replaced by the component of another embodiment, and the component of another embodiment may be added to the component of a certain embodiment. Moreover, for a part of the component of each embodiment, other components may be added, deleted, or replaced. [Simplified Explanation of the Diagram]
[0144] [Fig. 1] An illustration of an example of the utilization form of the neural network of the learning processing device of the present invention. [Fig. 2] An illustration of an example of the utilization form of the neural network of the learning processing device of the present invention. [Fig. 3] An illustration of the outline of pruning caused by the prior method. [Fig. 4] An illustration of the processing outline of the learning processing device of the present invention. [Fig. 5] An illustration comparing the effects caused by the present invention and the prior method. [Fig. 6] An illustration of an example of the functional configuration of the learning processing device of the present invention. [Fig. 7] An illustration of an example of the processing flow of the learning processing method of the present invention. [Fig. 8] A diagram showing the judgment basis information of the operator when evaluating the importance of a plurality of candidates of the prior learning model 302 in tabular form. [Fig. 9] An illustration of an example of the functional configuration of the learning processing device of Embodiment 2 of the present invention. [Fig. 10] An illustration of an example of the processing flow of the learning processing method of Embodiment 2 of the present invention. [Fig. 11] An illustration of the outline of partial re-initialization processing. [Fig. 12] An illustration of an example of the functional configuration of the learning processing device of Embodiment 3 of the present invention. [Figure 13] Illustration of a processing flow example of the learning processing method described in Embodiment 2 of the present invention. [Figure 14] Illustration of the outline of pruning caused by the masking layer. [Figure 15] Illustration of a method for applying a masking layer to a neural network with a complex network structure. [Figure 16] Diagram showing in tabular form the situation where convolutional layers 1402-a~d share 4 channels. [Figure 17] Illustration of a configuration example of a monitor screen suitable for the learning processing device of the present invention. [Figure 18] Illustration of a configuration example of a screen required for performing importance evaluation on a plurality of candidate pre-learning models.
Claims
1. A learning processing apparatus for obtaining a new second learning model from an existing first learning model, characterized by comprising: an input unit that acquires a first learning model generated in advance from a first learning dataset and an unpruned neural network; an important parameter specifying unit that initializes a prior neural network of a learning object using the prior first learning model and the prior neural network, and specifies the importance of parameters of the initialized prior neural network in the recognition process using a second learning dataset and the initialized prior neural network; a new model generation unit that performs pruning processing by deleting unimportant parameters from the initialized prior neural network using the importance of the prior parameters and generates a second neural network; a learning unit that learns the prior second neural network using the prior second learning dataset; and an output unit that outputs the learned prior second neural network as a second learning model.
2. The learning processing apparatus as described in claim 1, wherein, It includes: a display unit that calculates the importance of parameters obtained in the specific unit for each of the multiple prior first learning models, memorizes the types of the prior first learning models, and provides them as importance information.
3. The learning processing apparatus as described in claim 2, wherein, The importance system mentioned above includes the sum of the importance scores and the sum of the importance scores after pruning.
4. The learning processing apparatus as described in claim 1, wherein, It has the following features: a partial re-initialization unit, which, based on the importance assigned to the previous important parameters, partially re-initializes the pruned parameters when the previous first learning model has not learned the features that are important in the identification and processing of the learning object; after partial re-initialization, the previous new model generation unit is performed.
5. The learning processing apparatus as described in claim 1, wherein, The neural network is a convolutional neural network; it learns by using a relaxed Bernoulli distribution whether the characteristic channels of the output of the convolutional layers in some or all of the convolutional layers in the convolutional neural network are used for identification.
6. The learning processing apparatus as described in claim 5, wherein, During the learning of the relaxed Bernoulli distribution mentioned earlier, the amount of feature channels used for identification is gradually increased during the learning process.
7. The learning processing apparatus as described in claim 6, wherein, The importance of the predecessor is evaluated by the first or second derivative information of the loss function with respect to the parameters of the predecessor neural network, and the quantities obtained from the parameters of the predecessor neural network or the parameters of the predecessor relaxed Bernoulli distribution.
8. The learning processing apparatus as described in claim 7, wherein, When initializing the parts of the network structure that have not been initialized with the parameters of the first learning model mentioned above, the initialization is performed in a way that makes them easy to prune.
9. A learning processing method for obtaining a new second learning model from an existing first learning model, characterized by comprising the following steps: obtaining a first learning model generated in advance from a first learning dataset and an unpruned neural network; initializing the prior neural network of the learning object using the prior first learning model and the prior neural network; specifying the importance of the parameters of the initialized prior neural network in the recognition process using the second learning dataset and the initialized prior neural network; performing pruning processing by deleting unimportant parameters from the initialized prior neural network using the importance of the prior parameters and generating a second neural network; learning the prior second neural network using the prior second learning dataset; and using the learned prior second neural network as the second learning model.
10. A learning processing method for obtaining a new second learning model from an existing first learning model, characterized by: initializing the prior unpruned neural network of the learning object using an unpruned neural network and the first learning model; using a second learning data set and the initialized prior neural network, determining the importance of the parameters of the initialized prior neural network in the recognition processing; performing pruning processing corresponding to the importance; and obtaining the second learning model by learning the pruned prior neural network using the prior second learning data set.