Machine learning device and pruning method

Abstract

To provide a technique capable of achieving both improvement in calculation efficiency in a neural network obtained after pruning and suppression of deterioration in processing accuracy.SOLUTION: An exemplary machine learning device includes a processing unit that prunes a neural network. A pruning includes a first pruning process and a second pruning process different from the first pruning process.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a technique for pruning a neural network. [Background technology]

[0002] Conventionally, pruning has been known as a method for running neural networks trained by deep learning on edge devices with limited computational resources. Pruning involves removing redundant parameters from the neural network. Pruning can compress the neural network and improve the computational efficiency of the neural network.

[0003] Patent Document 1 discloses a technique for deleting layers in a multilayer neural network. The learning device in Patent Document 1 calculates, for each layer in the multilayer neural network, a contribution indicating the degree of contribution of the layer to the estimation result of the multilayer neural network, and selects a layer to be deleted based on the contribution of each layer. The learning device then deletes the selected layer to be deleted from the multilayer neural network and trains the multilayer neural network from which the layer to be deleted has been deleted. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-185275 Summary of the Invention [Problem to be solved by the invention]

[0005] Although removing layers from a neural network can improve the computational efficiency of the resulting neural network, there is a concern that the processing accuracy of the neural network may decrease. On the other hand, if redundant parameters cannot be sufficiently removed by pruning, the computational efficiency of the resulting neural network may not be sufficiently improved.

[0006] In view of the above, an object of the present invention is to provide a technology that can achieve both improved calculation efficiency in a neural network obtained after pruning and suppression of a decrease in processing accuracy. [Means for solving the problem]

[0007] An exemplary machine learning device of the present invention includes a processing unit that performs pruning on a neural network, the pruning including a first pruning process and a second pruning process different from the first pruning process. [Effects of the Invention]

[0008] According to the exemplary embodiment of the present invention, it is possible to improve the calculation efficiency of the neural network obtained after pruning while suppressing a decrease in processing accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] Block diagram showing the general configuration of the machine learning system [Figure 2] Schematic diagram showing an overview of pruning [Figure 3] Schematic diagram for explaining an example of the structure of a neural network [Figure 4] Schematic diagram showing an example of the structure of a stage in a neural network [Figure 5] FIG. 10 is a diagram for explaining an example of a first pruning process; [Figure 6]FIG. 10 is a diagram for explaining another example of the first pruning process; [Figure 7] 1 is a flowchart illustrating an example of a machine learning method including pruning. [Figure 8] Flowchart showing the pruning process [Figure 9] 1 is a flowchart illustrating a variation of a machine learning method that includes pruning. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0011] <1. Machine Learning System> 1 is a block diagram showing a schematic configuration of a machine learning system 100 according to an embodiment of the present invention. As shown in FIG. 1, the machine learning system 100 includes a machine learning device 1 and an edge device 2.

[0012] The machine learning device 1 performs processing related to the generation of a trained neural network. The processing performed by the machine learning device 1 includes at least processing for pruning the trained neural network. In this embodiment, the processing performed by the machine learning device 1 includes not only pruning but also re-learning processing after pruning.

[0013] Note that the process performed by the machine learning device 1 may be pruning only. In this case, the machine learning device 1 may be a pruning device. As another example, the process performed by the machine learning device 1 may include a process of training a neural network (pre-training process) that is performed before pruning.

[0014] In this embodiment, the machine learning device 1 is connected to an edge device 2 via a communication network such as the Internet. The machine learning device 1 is a so-called server device, and distributes a neural network that has been pruned and subsequently re-trained to the edge device 2. Note that the server device referred to here may be a physical server or a cloud server. The edge device 2 may be, for example, a smartphone, a personal computer, an in-vehicle device, an IoT (Internet of Things) home appliance, or the like.

[0015] The neural network pruned and subsequently re-learned by the machine learning device 1 may be configured to be recorded on a recording medium such as an optical recording medium or a magnetic recording medium and provided to another device (such as an in-vehicle device) via the recording medium. The neural network pruned and subsequently re-learned by the machine learning device 1 may be configured to be provided to another device (such as an in-vehicle device) via a wired connection.

[0016] <2. Machine Learning Device> (2-1. Overview of the device) As shown in FIG. 1, the machine learning device 1 includes a processing unit 11 and a storage unit 12.

[0017] The processing unit 11 is configured, for example, by one or more processors. The processor may be configured to include, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Alternatively, the processor may be configured to include, for example, a CPU and a GPU.

[0018] The storage unit 12 non-temporarily stores or memorizes computer-readable programs, data, etc. The storage unit 12 has a non-volatile storage medium. The non-volatile storage medium of the storage unit 12 may be composed of at least one of a semiconductor memory, a magnetic medium, an optical medium, etc.

[0019] 1, the processing unit 11 has, as its functional configuration, a pruning unit 111 and a learning unit 112. The pruning unit 111 and the learning unit 112 included in the processing unit 11 are realized, for example, by a processor executing arithmetic processing in accordance with a program stored in the storage unit 12. The scope of this embodiment includes a computer program that realizes the machine learning device 1 by a computer. The scope of this embodiment also includes a computer-readable non-volatile recording medium on which such a program is recorded.

[0020] As described above, the above-mentioned units 111 and 112 may be realized by causing a processor to execute a program, i.e., by software, but may also be realized by other methods. The above-mentioned units 111 and 112 may be realized using, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). That is, the above-mentioned units 111 and 112 may be realized by hardware using a dedicated IC or the like. The above-mentioned units 111 and 112 may also be realized by a combination of software and hardware. Furthermore, the above-mentioned units 111 and 112 are conceptual components. The function performed by one component may be distributed among multiple components, or the functions of multiple components may be integrated into one component.

[0021] The pruning unit 111 performs pruning on the neural network. That is, the machine learning device 1 includes a processing unit 11 that performs pruning on the neural network. In detail, the neural network to be pruned is a trained neural network that has been trained in advance. This training in advance may be performed by the machine learning device 1 or by another device. Note that the trained neural network to be pruned is different from the trained neural network that the machine learning device 1 distributes to the edge device 2.

[0022] [2-1-1. Overview of pruning] Figure 2 is a schematic diagram showing an overview of pruning. The neural network NN1 shown on the left side of Figure 2 is the neural network before pruning is performed. The neural network NN2 shown on the right side of Figure 2 is the neural network after pruning has been performed on the neural network NN1.

[0023] As shown in FIG. 2, neural networks NN1 and NN2 have multiple layers L, each of which is made up of multiple neurons N. In the example shown in FIG. 2, neural networks NN1 and NN2 have four layers L. Neurons N are equivalent to so-called nodes. Between layers L, there are weights W, which indicate the strength of the connections between neurons N. As shown in FIG. 2, the neural network NN2 after pruning has fewer neurons N and weights W than the neural network NN1 before pruning. In other words, when pruning is performed, at least one of the weights W and neurons N of the neural network is removed. The number of neurons N and weights W included in the neural network NN2 after pruning shown in FIG. 2 is reduced compared to the neural network NN1 before pruning. In other words, the neural network NN2 after pruning is smaller (compressed) in size than the neural network NN1 before pruning. Since the neural network NN2 after pruning is compressed, when the neural network NN2 after pruning is used, not only the amount of calculation but also the memory occupation amount and memory access are reduced, thereby improving the calculation efficiency. Details of pruning in this embodiment will be described later.

[0024] 1, the learning unit 112 re-learns the pruned neural network using a dataset (not shown). That is, the processing unit 11 trains the pruned neural network. With this configuration, it is possible to optimize the weights of the pruned neural network and suppress a decrease in the processing accuracy of the neural network.

[0025] The learning unit 112 does not have to be included in the machine learning device 1. A known method may be used for learning using a dataset. The dataset may be a group of training data including training data and correct labels. In this embodiment, the learning unit 112 outputs the trained neural network obtained by re-learning to the edge device 2.

[0026] (2-2. Neural Networks) 3 is a schematic diagram illustrating an example of the structure of a neural network 30 on which pruning is performed by the processing unit 11. The neural network 30 is, for example, a neural network for image recognition that performs image classification, object detection, etc. However, the neural network 30 is not limited to a neural network for image recognition, and may also be, for example, a neural network for voice recognition.

[0027] In the example shown in Figure 3, the neural network 30 to be pruned has multiple stages 31, each including at least one block 32. More specifically, the neural network 30 has a structure in which multiple stages 31 are connected together. The number of blocks 32 included in the multiple stages 31 may be the same or different. Among the multiple stages 31, there is at least one stage 31 that includes multiple blocks 32.

[0028] FIG. 4 is a schematic diagram showing an example of the structure of a stage 31 included in a neural network 30. In the example shown in FIG. 4, the stage 31 includes a plurality of blocks 32, specifically three blocks 32. As shown in FIG. 4, each block 32 includes a plurality of layers 33. In the example shown in FIG. 4, each block 32 includes three convolutional layers 33.

[0029] In the example shown in FIG. 4 , the last convolutional layers 33 of the blocks 32 included in the stage 31 are connected by a residual connection 34. The residual connection is synonymous with a residual connection or a skip connection. Specifically, the stage 31 includes two residual connections 34. The first residual connection 34a connects the output of the last convolutional layer 33aL of the first block 32a to the output of the last convolutional layer 33bL of the second block 32b. The second residual connection 34b connects the output of the last convolutional layer 33bL of the second block 32b to the output of the last convolutional layer 33cL of the third block 32c. The number of residual connections 34 included in the stage 31 may be changed as appropriate depending on the number of blocks 32 included in the stage 31.

[0030] Convolutional layers 33 whose outputs are connected by residual connections 34 have the same output channel configuration because elements are added together. Therefore, the last convolutional layer 33 of each block 32 connected by the residual connections 34 has the same number of channels. In other words, each block 32 in the stage 31 has the same number of output channels. In the example shown in FIG. 4, the last convolutional layer 33aL of the first block 32a, the last convolutional layer 33bL of the second block 32b, and the last convolutional layer 33cL of the third block 32c all have the same number of output channels, C1. In other words, the three blocks 32a, 32b, and 32c all have the same number of output channels, C1.

[0031] In this embodiment, the number of output channels of the blocks 32 differs for each stage 31. In other words, the stage 31 is a collection of blocks 32 that have the same number of output channels. For example, if the neural network 30 has five stages 31, the numbers of output channels of the blocks 32 in the first, second, third, fourth, and fifth stages are 24, 40, 80, 120, 160, etc., respectively. Note that no residual connections 34 are established across the stages 31.

[0032] The neural network 30 configured as described above may be configured using, for example, the well-known ResNet or MobileNetV2. In ResNet, each block 32 includes two layers. In MobileNetV2, each block 32 includes three layers.

[0033] In the above, the residual connections 34 are configured to connect the last layers 33 of each block 32. However, this is merely an example. The neural network to be pruned may include residual connections. For example, the residual connections may be configured to connect the middle layers 33 of each block 32 (e.g., layer 33aM and layer 33bM).

[0034] (2-3. Details of pruning) In this embodiment, pruning includes a first pruning process and a second pruning process different from the first pruning process. With this configuration, it is possible to selectively use different pruning methods depending on the configuration of the neural network 30. By selectively using multiple pruning methods, it is possible to appropriately remove weights and nodes from the neural network 30, thereby achieving both improved computational efficiency and suppressed degradation of processing accuracy in the neural network 30 obtained after pruning.

[0035] More specifically, the first pruning process and the second pruning process prune different units of components of the neural network 30. With this configuration, different pruning methods can be used depending on differences in the local structure of the neural network 30, and weights and nodes of the neural network 30 can be appropriately removed.

[0036] More specifically, the first pruning process performs pruning in units of channels of the convolutional layer. The second pruning process performs pruning in units of weight parameters. The channel pruning process performs pruning in units of larger components of the neural network 30 than the weight parameter pruning process. That is, the first pruning process performs pruning in units of larger components of the neural network 30 than the second pruning process.

[0037] As described above, the neural network 30 includes residual connections 34. In such a configuration, the output channels of the convolutional layers 33 connected by the residual connections 34 must have the same configuration. For this reason, it is not possible to perform channel-by-channel pruning on the layers 33 connected by the residual connections 34. In other words, when performing channel-by-channel pruning on the layers 33 connected by the residual connections 34, it is necessary to perform pruning by comprehensively determining the importance of each channel in each connected layer 33. In other words, the pruning process tends to be complicated.

[0038] Furthermore, if there is a bias in the number of unnecessary channels among the layers 33 connected by the residual connections 34, it is necessary to perform pruning according to the layer 33 with the fewest number of unnecessary channels. As a result, in the layer 33 with the most unnecessary channels, excess channels may remain, reducing the effectiveness of pruning per channel.

[0039] Furthermore, in pruning, a method is generally used in which the ratio of weight parameters to be pruned is determined for the entire neural network 30. In this case, because of the above-mentioned constraints on channel-by-channel pruning in layers 33 connected by residual connections 34, pruning tends to be biased toward layers 33 other than those connected by residual connections 34. As a result, the processing accuracy of the neural network 30 obtained after pruning may be reduced.

[0040] In this regard, in this embodiment, since it is possible to selectively use pruning processing on a channel basis and pruning processing on a weight parameter basis, it is possible to perform appropriate pruning while avoiding the complexity of the pruning processing and bias in the pruning targets. As a result, it is possible to achieve both improved calculation efficiency and suppression of deterioration in processing accuracy in the neural network 30 obtained after pruning.

[0041] Specifically, the processing unit 11 excludes from the first pruning process (pruning process in units of channels) at least some of the layers 33 whose outputs are connected by residual connections 34 among the multiple layers 33 constituting the neural network 30. The second pruning process (pruning process in units of weight parameters) is performed on the layers 33 that are excluded from the first pruning process. With this configuration, it is possible to perform appropriate pruning on the layers 33 whose outputs are connected by residual connections 34 while avoiding the process from becoming complicated.

[0042] In the example shown in Figure 4, the last convolutional layers 33aL, 33bL, and 33cL of each block 32a, 32b, and 32c are excluded from the first pruning process (channel-based pruning). These convolutional layers 33aL, 33bL, and 33cL are then subjected to the second pruning process (weight parameter-based pruning). The first convolutional layers 33aF, 33bF, and 33cF of each block 32 and the intermediate convolutional layers 33aM, 33bM, and 33cM are subjected to the first pruning process.

[0043] Note that only the first pruning process may be performed on the first convolutional layers 33aF, 33bF, and 33cF and the intermediate convolutional layers 33aM, 33bM, and 33cM of each block 32. However, it is preferable that the first convolutional layers 33aF, 33bF, and 33cF and the intermediate convolutional layers 33aM, 33bM, and 33cM of each block 32 are subjected to the second pruning process in addition to the first pruning process.

[0044] FIG. 5 is a diagram illustrating an example of the first pruning process. In FIG. 5, i represents the number of the convolutional layer 33, C represents the number of channels, and w and h represent the width and height of the feature map 40, respectively. A smaller value of i represents a closer position to the input layer, and a larger value of i represents a closer position to the output layer. The weights of the convolutional layer 33 are configured by arranging M×N convolutional filters (kernels) of size k×k. The number of rows M corresponds to the number of channels (Ci+1) of the output feature map 40. The number of columns N corresponds to the number of channels (Ci) of the input feature map 40.

[0045] In the example shown in FIG. 5, the absolute sum of the weight parameters is calculated for each convolution filter. For example, if the size of the convolution filter is 3×3, the absolute sums of nine weight parameters are calculated. For each input channel (each column), the sum of the calculated absolute sums (the sum of M absolute sums) is calculated as an evaluation value. For an input channel with a small evaluation value, the convolution filter of the input channel and the input feature map 40 corresponding to that input channel are deleted. The input channels to be deleted are, for example, a predetermined number in ascending order of evaluation value.

[0046] 5, one input channel is targeted for deletion, and all convolution filters included in the target input channel and the hatched input feature map 40 corresponding to the target input channel are deleted. Also, instead of the above-described sum of absolute values, for example, a configuration in which a sum of squares or a maximum absolute value is used may be adopted.

[0047] FIG. 6 is a diagram illustrating another example of the first pruning process. In FIG. 6, as in FIG. 5, i represents the number of the convolutional layer 33, C represents the number of channels, and w and h represent the width and height of the feature map 40, respectively. A smaller value of i represents closer to the input layer, and a larger value of i represents closer to the output layer. The weights of the convolutional layer 33 are configured by arranging M×N convolutional filters of size k×k. The number of rows M corresponds to the number of channels (Ci+1) of the output feature map 40. The number of columns N corresponds to the number of channels (Ci) of the input feature map 40.

[0048] In the example shown in Fig. 6, the absolute sum of the weight parameters is also calculated for each convolution filter. However, in the example shown in Fig. 6, the sum of the calculated absolute sums (the sum of N absolute sums) is calculated as the evaluation value for each output channel (each row). For an output channel with a small evaluation value, the convolution filter of the output channel and the output feature map 40 corresponding to that output channel are deleted. The output channels to be deleted are, for example, a predetermined number in ascending order of evaluation value.

[0049] 6, one output channel is targeted for deletion, and all convolution filters included in the output channel targeted for deletion and the hatched output feature map 40 corresponding to the output channel are deleted. Also, similar to the example shown in FIG. 5, a configuration may be adopted in which, instead of the sum of absolute values ​​described above, for example, a sum of squares or a maximum absolute value is used.

[0050] The pruning process shown in Fig. 5 is a pruning process that reduces the number of input channels, which is known as channel pruning. The pruning process shown in Fig. 6 is a pruning process that reduces the number of output channels, which is known as filter pruning. As can be seen from the above explanation, the pruning processes shown in Fig. 5 and Fig. 6 are both performed on a channel-by-channel basis.

[0051] As described above, the second pruning process is performed in weight parameter units, which are units smaller than channel units. For example, if the absolute value of a weight parameter is smaller than a predetermined threshold, the weight parameter is set to zero. Instead of the absolute value, for example, a square value may be used.

[0052] Note that a predetermined number of weight parameters may be set to 0 in ascending order of absolute value of the weight parameters. Alternatively, for example, a method of exploratory finding weight parameters that do not affect the performance (e.g., recognition rate, etc.) of the neural network 30 may be used.

[0053] The second pruning process, which is performed in weight parameter units, prunes components in the neural network 30 in units of smaller components than the first pruning process, which is performed in units of channels. For this reason, the second pruning process can suppress accuracy degradation to a smaller extent than the first pruning process.

[0054] <3. Machine Learning Methods> Next, a machine learning method using a pruning technique according to an embodiment of the present invention will be described. Fig. 7 is a flowchart showing an example of a machine learning method including pruning according to an embodiment of the present invention.

[0055] In step S1, the neural network 30 is trained. The training is performed using a dataset (not shown). The dataset may be a group of training data including training data and correct answer labels. This training is the pre-training described above. The pre-training may be performed by the machine learning device 1 or another device. When the training is completed, the process proceeds to the next step S2.

[0056] In step S2, the processing unit 11 of the machine learning device 1 performs pruning on the trained neural network 30. When pruning is completed, the process proceeds to the next step S3. Before describing the process of step S3, the flow of pruning will be described with reference to FIG. 8. FIG. 8 is a flowchart showing the flow of pruning. More specifically, FIG. 8 is a flowchart showing the detailed flow of pruning in step S2 of FIG. 7.

[0057] In step S21, a weight parameter reduction rate is set. The reduction rate is the ratio by which weight parameters are reduced relative to all weight parameters possessed by the trained neural network 30. For example, the reduction rate is set by a person such as a developer who develops the neural network 30. The reduction rate is manually set for the machine learning device 1 using an input device (not shown). Note that the reduction rate may be automatically set by the machine learning device 1.

[0058] Although the reduction rate may be configured to have only one value set, in this embodiment, multiple reduction rates are set so that the numerical value increases in stages. This is because the reduction rate increases with each pruning count from the first pruning. For example, the reduction rate is set in the form of 20%, 25%, 30%, etc. Once the reduction rate is set, the process proceeds to the next step S22.

[0059] In step S22, the processing unit 11 identifies unnecessary channels in the trained neural network 30. The process of identifying unnecessary channels is included in the first pruning process performed on a channel-by-channel basis as described above. Of the multiple layers 33 included in the neural network 30, layers 33 whose outputs are connected by residual connections 34 are excluded from the targets for identifying unnecessary channels. For example, a predetermined number of channels are identified as unnecessary channels in ascending order of the evaluation value described above. The predetermined number may be configured to be automatically determined according to, for example, the reduction rate. Once the unnecessary channels have been identified, the process proceeds to the next step S23.

[0060] In step S23, the processing unit 11 identifies unnecessary weight parameters in the trained neural network 30. The process of identifying unnecessary weight parameters is included in the second pruning process performed on a weight parameter basis as described above. In this embodiment, the process of identifying unnecessary weight parameters is performed on all layers 33 included in the neural network 30. Note that weight parameters included in channels that have already been identified as unnecessary channels may be excluded from the process. Unnecessary weight parameters are identified, for example, in ascending order of absolute value of the weight parameters according to the previous identification result of unnecessary channels and the reduction rate. Once unnecessary weight parameters have been identified, the process proceeds to the next step S24.

[0061] The order of the processing in step S22 and the processing in step S23 may be reversed.

[0062] In step S24, the processing unit 11 executes a process of pruning the channels and weight parameters identified as unnecessary in the processes of steps S22 and S23. The pruning process in step S24 includes a part of a first pruning process and a part of a second pruning process. After the pruning process is executed, the process proceeds to step S3 shown in FIG. 7.

[0063] As can be seen from the above, the processing unit 11 identifies layers 33 that are not to be subject to the first pruning process from among the multiple layers 33 constituting the neural network 30, and performs the first pruning process on layers excluding the identified layers 33. In other words, the processing unit 11 identifies layers 33 that are to be subject to the first pruning process from among the multiple layers constituting the neural network 30, and performs the first pruning process on the identified layers 33. With this configuration, the first pruning process can be avoided for layers 33 that are not suitable for the first pruning process, which performs pruning on a channel-by-channel basis. For example, at least some of the layers 33 whose outputs are connected by residual connections 34 can be excluded from the first pruning process. On the other hand, for layers 33 suitable for channel-by-channel pruning, pruning is performed on a channel-by-channel basis, which prevents the pruning effect from being reduced.

[0064] Furthermore, the processing unit 11 performs a second pruning process on layers 33 among the multiple layers 33 that are identified as not being targeted for the first pruning process. In other words, the processing unit 11 performs a second pruning process on layers 33 among the multiple layers 33 that are different from the layers 33 identified as being targeted for the first pruning process. With this configuration, layers 33 not targeted for the first pruning process are targeted for the second pruning process, which prevents the effect of pruning from becoming extremely small. For example, pruning can be performed on weight parameter units for layers 33 whose outputs are connected by residual connections 34. Note that when the second pruning process is performed on layers 33, the weight parameters between layers are targeted for pruning.

[0065] Furthermore, the processing unit 11 performs a second pruning process on layers 33 other than the layers 33 identified as not being the target of the first pruning process. In other words, the processing unit 11 performs a second pruning process on layers 33 identified as the target of the first pruning process. With this configuration, the layers 33 that were the target of the first pruning process can also be the target of the second pruning process. This makes it possible to avoid insufficient pruning, and can be expected to improve the calculation efficiency of the neural network 30 after processing. Note that the above-described embodiment has such a configuration. However, the processing unit 11 may also be configured not to perform the second pruning process on the layers 33 that were the target of the first pruning process.

[0066] 7, in step S3, the processing unit 11 uses the data set to re-learn the pruned neural network 30. When the re-learning is completed, the process proceeds to the next step S4.

[0067] In step S4, the processing unit 11 determines whether or not to terminate pruning. For example, the processing unit 11 determines to terminate pruning when the processing accuracy of the neural network 30 after pruning becomes equal to or less than a preset threshold. As another example, the processing unit 11 determines to terminate pruning when the execution time of a task (image recognition, etc.) of the neural network 30 after pruning becomes equal to or less than a preset target value. If it is determined to terminate pruning (Yes in step S4), the processing shown in FIG. 7 ends. If it is determined not to terminate pruning (No in step S4), the process returns to step S2, and the processing from step S2 onwards is performed.

[0068] When the process returns to step S2, the reduction rate is changed to a higher rate than the previous rate, and pruning is performed. The neural network 30 obtained by the process shown in FIG. 7 is distributed to, for example, the edge device 2.

[0069] In the above, the first pruning process and the second pruning process are executed together, but the first pruning process and the second pruning process may be executed in order as shown in Fig. 9. Fig. 9 is a flowchart showing a modified example of the machine learning method including pruning.

[0070] First, the trained neural network trained in step N1 is subjected to a first pruning process (step N2). A reduction rate is set, and pruning is performed on a channel-by-channel basis. Note that layer 33, whose output is connected by residual connections 34, is excluded from the pruning process.

[0071] After the first pruning process, re-learning is performed (step N3), and then it is determined whether or not to terminate the first pruning process (step N4). For example, if the processing accuracy of the neural network after the first pruning process falls below a preset threshold, it is determined that the first pruning process should be terminated. If the first pruning process is not to be terminated (No in step N4), the process returns to step N2 and the processes from step N2 onwards are repeated.

[0072] If the first pruning process is to be terminated (Yes in step N4), a second pruning process is performed on the neural network that has undergone the first pruning process (step N5). A reduction rate is set, and pruning is performed for each weight parameter. In this example, all layers, including layers whose outputs are connected by residual connections, are pruned.

[0073] After the second pruning process, re-learning is performed (step N6), and then it is determined whether or not to terminate the second pruning process (step N7). For example, if the processing accuracy of the neural network after the second pruning process is equal to or lower than a preset threshold, it is determined that the second pruning process is to be terminated. If the second pruning process is not to be terminated (No in step N7), the process returns to step N5 and the processes from step N5 onwards are repeated. If the second pruning process is to be terminated (Yes in step N7), the process shown in FIG. 9 is terminated. The neural network obtained by the process shown in FIG. 9 is distributed to, for example, edge device 2.

[0074] In this modified example, the second pruning process is performed after the first pruning process, but the first pruning process may be performed after the second pruning process.

[0075] <4. Things to keep in mind> In addition to the above-described embodiments, the various technical features disclosed in this specification can be modified in various ways without departing from the spirit of the technical creation. In other words, the above-described embodiments are illustrative in all respects and should be considered not to be limiting. The technical scope of the present invention is defined by the claims, not by the description of the above-described embodiments, and should be understood to include all modifications that fall within the meaning and scope of the claims. Furthermore, the multiple embodiments and modifications described in this specification may be combined as appropriate to the extent possible. [Explanation of symbols]

[0076] 1. Machine learning device 11 Processing section 30. Neural Networks 33··· layers, convolutional layers 34···Residual Connection

Claims

1. A pruning device for pruning a neural network, comprising: a processing unit; The pruning includes: a first pruning process for pruning in units of channels of a convolutional layer; a second pruning process for performing the pruning in units of weight parameters; Contains, The processing unit determining whether to perform the first pruning process on the convolutional layer according to a connection state of a residual connection at the output of the convolutional layer; determining whether or not to perform the second pruning process regardless of the coupling state of the residual connections; Pruning device.

2. A pruning device for pruning a neural network, comprising: a processing unit; The pruning includes: a first pruning process for pruning in units of channels of a convolutional layer; a second pruning process for performing the pruning in units of weight parameters; Contains, The processing unit determining whether to perform the first pruning process on the convolutional layer according to a connection state of a residual connection at the output of the convolutional layer; The convolutional layer that was not subjected to the first pruning process is subjected to the second pruning process. Pruning device.

3. The processing unit excludes the convolutional layers connected by the residual connections from the first pruning process.

3. A pruning device according to claim 1 or 2.

4. A machine learning device including the pruning device according to any one of claims 1 to 3, performing one of the first pruning process and the second pruning process; After the one of the processes, a learning process is performed using the data set; After the learning process, the other process is performed, which is not the one process; After the other processing, a learning process is performed using the data set. Machine learning device.

5. A method for pruning a neural network by a processor, comprising: The pruning includes: a first pruning process for pruning in units of channels of a convolutional layer; a second pruning process for performing the pruning in units of weight parameters; Contains, determining whether to perform the first pruning process on the convolutional layer according to a connection state of a residual connection at the output of the convolutional layer; determining whether or not to perform the second pruning process regardless of the coupling state of the residual connections; Pruning methods.

Images