Support method, support device, and program

Abstract

To provide a support method which can compress a learning model with less accuracy degradation when pruning the learning model.SOLUTION: The support method supports pruning of a learning model using a neural network having a Depthwise convolutional layer and a convolutional layer in a following stage of the Depthwise convolutional layer. The learning model includes an object neuron of pruning in the Depthwise convolutional layer, and the support method determines whether or not the Depthwise convolutional layer has a bias term (S14) and, if the Depthwise convolutional layer has a bias term (Yes in S14), acquires a bias value based on the bias term (S15) and uses a value based on the bias value to calculate a correction value for correcting the bias term which the convolutional layer in the following stage has (S20).SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an assistance method, an assistance device, and a program for assisting pruning of a learning model that uses a neural network having a depthwise convolutional layer and a convolutional layer subsequent to the depthwise convolutional layer. [Background technology]

[0002] Known methods for reducing the weight of a learning model include pruning, quantization, distillation, etc. Patent Document 1 discloses a technology for reducing the weight of a learning model by quantization. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2022-49997 Summary of the Invention [Problem to be solved by the invention]

[0004] Pruning is sometimes used as a lightweighting technique. In pruning, increasing the amount of pruning is necessary to increase the compression rate of a training model, but increasing the amount of pruning can result in a deterioration in the accuracy of the training model. Thus, in the past, there was a trade-off between compression rate and accuracy degradation, making it difficult to compress a training model while suppressing accuracy degradation.

[0005] Therefore, the present disclosure provides an assistance method, assistance device, and program that can compress a learning model while suppressing accuracy degradation when pruning the learning model. [Means for solving the problem]

[0006] A support method according to one embodiment of the present disclosure is a support method for supporting pruning of a learning model that uses a neural network having a first depthwise convolutional layer and a convolutional layer subsequent to the first depthwise convolutional layer, wherein the learning model includes a target neuron to be pruned in the first depthwise convolutional layer, and the support method determines whether the first depthwise convolutional layer has a bias term, and if the first depthwise convolutional layer has a bias term, obtains a first bias value based on the bias term, and calculates a correction value for correcting the bias term in the subsequent convolutional layer using a value based on the first bias value.

[0007] An assistance device according to one embodiment of the present disclosure is an assistance device that assists in pruning of a learning model that uses a neural network having a depthwise convolutional layer and a convolutional layer subsequent to the depthwise convolutional layer, wherein the learning model includes target neurons to be pruned in the depthwise convolutional layer, and the assistance device includes: a determination unit that determines whether the depthwise convolutional layer has a bias term; an acquisition unit that, if the depthwise convolutional layer has a bias term, acquires a bias value based on the bias term; and a calculation unit that uses a value based on the bias value to calculate a correction value for correcting the bias term in the subsequent convolutional layer.

[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-described assistance method. [Effects of the Invention]

[0009] According to one aspect of the present disclosure, it is possible to realize an assistance method or the like that can compress a learning model while suppressing deterioration in accuracy when pruning the learning model. [Brief explanation of the drawings]

[0010] [Figure 1]FIG. 1 is a block diagram showing a functional configuration of the support device according to the first embodiment. [Figure 2] FIG. 2 is a flowchart showing the operation of the support device according to the first embodiment. [Figure 3] FIG. 3 is a diagram for explaining pruning and bias term correction in the support device according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining the effect of the support device according to the first embodiment. [Figure 5] FIG. 5 is a flowchart showing the operation of the support device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0012] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims are described as optional components.

[0013] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.

[0014] Furthermore, in this specification, terms indicating relationships between elements such as "same," as well as numerical values ​​and numerical ranges, are not expressions that express only the strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about several percent (or about 10%).

[0015] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0016] (Embodiment 1) The assistance device and the like according to this embodiment will be described below with reference to FIGS.

[0017] [1. Configuration of the support device] First, the configuration of the support device according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of the support device 1 according to this embodiment.

[0018] The assistance device 1 is an information processing device for assisting in pruning a learning model (deep learning model) that uses a convolutional neural network (Convolutional Neural Network (CNN)) having a depthwise convolutional layer and a convolutional layer subsequent to the depthwise convolutional layer. In this embodiment, the subsequent convolutional layer is the layer immediately following the depthwise convolutional layer. The depthwise convolutional layer and the subsequent convolutional layer each have a plurality of neurons, and each of the plurality of neurons included in the depthwise convolutional layer is connected (coupled) to each neuron in the subsequent convolutional layer. Note that each layer in a neural network generally has an activation function such as a ReLU function, an identity function, or a sigmoid function, and each layer in the neural network according to this embodiment will be described as also having an activation function. However, in this embodiment, the activation function will basically be omitted and will be specified as needed. In addition, in this embodiment, an example will be described in which the subsequent convolutional layer is a pointwise convolutional layer, but it may be a convolutional layer different from the depthwise convolutional layer and the pointwise convolutional layer (for example, a normal convolutional layer). Note that each layer in a neural network is not limited to having an activation function. Each layer in a neural network may have, for example, an identity function instead of an activation function.

[0019] Depthwise convolutional layers and pointwise convolutional layers are layers included in convolutional neural networks such as MobileNetV1. While a normal convolutional layer simultaneously performs convolutions in the spatial and channel directions, a depthwise convolutional layer performs only convolutions in the spatial direction, and a pointwise convolutional layer performs only convolutions in the channel direction. Hereinafter, a depthwise convolutional layer will also be referred to as a DW convolutional layer or DW, and a pointwise convolutional layer will also be referred to as a PW convolutional layer or PW.

[0020] The learning model is, for example, a machine learning model for performing image recognition, speech recognition, etc., but its applications are not limited to these. Note that, hereinafter, the convolutional neural network will also be simply referred to as a neural network.

[0021] 1, the support device 1 has, as its functional configuration, a learning unit 10, a pruning unit 20, and a storage unit 30. The support device 1 is realized by a non-volatile memory in which a program is stored, a volatile memory that is a temporary storage area for executing the program, an input / output port, a communication interface, a processor that executes the program, etc. The support device 1 may be realized by a mobile terminal such as a stationary personal computer (PC), a portable PC, a smartphone, or a tablet, a dedicated computer, etc., or may be realized by a server (for example, a cloud server).

[0022] The learning unit 10 is a processing unit that performs a learning process to create a desired learning model. The learning unit 10 performs the learning process to determine each parameter of the machine learning model (for example, bias, weight, etc., which will be described later). The bias (bias term) is one of the important parameters in a neural network, and is a constant that is added to the output of each neuron and is a constant value regardless of the input to that neuron. The weight is a constant that indicates the strength of the connection between neurons. The learning unit 10 includes a model creation unit 11 and a model evaluation unit 12.

[0023] The model creation unit 11 creates a learning model by executing a learning process for the machine learning model using a learning dataset including learning image data and ground truth data. The learning model is a machine learning model that uses a neural network having at least a DW convolution layer, and in this embodiment, it is a machine learning model that uses a neural network having a PW convolution layer subsequent to the DW convolution layer. The model creation unit 11 learns optimal parameters for each layer of the learning model using a known method such as backpropagation. Note that the method for creating the learning model by the model creation unit 11 is not limited to backpropagation, and any known method may be used.

[0024] The model evaluation unit 12 evaluates the learning model created by the model creation unit 11 using an evaluation dataset including evaluation image data and supervised data. The model evaluation unit 12 inputs the evaluation image data into the learning model to obtain a label corresponding to the image data as an output of the learning model, and evaluates the learning model based on the label and supervised data. The evaluation dataset may include at least a portion of the data in the learning dataset, or may include data different from the learning dataset.

[0025] This makes it possible to determine whether the learning model created by the model creation unit 11 has performance equal to or greater than a predetermined level. If the performance of the learning model is lower than a predetermined level, the model creation unit 11 may re-learn the created learning model.

[0026] The pruning unit 20 is a processing unit that performs pruning to reduce the weight of the learning model created by the learning unit 10. Neurons are connected to neurons in the next layer, and pruning includes deleting (cutting) weakly connected paths (weights) between neurons. For example, pruning may be a process for stopping one-way data exchange between neurons (a process for cutting connections). Furthermore, pruning may include deleting neurons whose output (channel) is zero or close to zero. Hereinafter, the phrase "deleting a neuron" refers to deleting not only the neuron but also the paths and weights connected to the neuron. Pruning reduces the amount of calculation and memory usage. The pruning unit 20 includes a selection unit 21, a correction unit 22, and a pruning processing unit 23.

[0027] The selection unit 21 selects neurons to be pruned from among multiple neurons included in the learning model created by the learning unit 10. The method for selecting the target neurons in the selection unit 21 is not particularly limited, and an example thereof is a method using the APOZ (Average Percentage Of Zeros) index. The APOZ index is an index for determining which neurons to remove from a neural network. For example, the percentage of zero-activation outputs (outputs that are almost or completely zero) is calculated, and neurons to be pruned are selected so as to remove neurons with a high percentage of zero activation. The selection unit 21 may create a target neuron list indicating a list of neurons to be pruned, and store the list in the storage unit 30 in association with the learning model.

[0028] When the target neuron selected by the selection unit 21 is connected to a neuron in a DW convolutional layer having a bias term, the correction unit 22 corrects the bias term of the PW convolutional layer subsequent to the DW convolutional layer according to the bias term of the DW convolutional layer. The target neuron is a neuron in the convolutional layer preceding the DW convolutional layer.

[0029] The correction unit 22 calculates a correction value based on the term of the DW convolution layer connected to the target neuron to be pruned and the weight of the PW convolution layer subsequent to the DW convolution layer connected to the target neuron, and corrects the bias term of the subsequent PW convolution layer (for example, the bias term of the neuron connected to the target neuron among the multiple neurons in the subsequent PW convolution layer) based on the calculated correction value.

[0030] The pruning unit 23 deletes the target neuron selected by the selection unit 21 and stores the learning model from which the target neuron has been deleted in the storage unit 30. The pruning unit 23 performs pruning by, for example, deleting from the network neurons with a high rate of zero activation selected by the selection unit 21 and simultaneously removing connections between the neurons. The learning model stored in the storage unit 30 by the pruning unit 23 is a learning model from the learning model created by the learning unit 10, in which bias terms in subsequent convolutional layers have been corrected and the target neuron has been deleted.

[0031] The memory unit 30 is a storage device that stores various information used in the pruning process, the learning model after the pruning process, etc. The memory unit 30 stores, for example, the learning model created by the learning unit 10 (the learning model before the pruning process), the learning model after the pruning process by the pruning unit 20, and information about the learning model. The information about the learning model includes information indicating the configuration of the neural network in the learning model created by the learning unit 10, information used for the convolution process, etc. The information indicating the configuration of the neural network includes information indicating whether each layer is a DW convolution layer, a PW convolution layer, or another convolution layer, and whether a bias term is provided in each layer. The information indicating the configuration of the neural network may also include information about the activation function of each layer in the neural network. The information used for the convolution process includes the kernel size, etc. The memory unit 30 may also store various data sets. The memory unit 30 may be realized, for example, by a semiconductor memory, but is not limited to this.

[0032] The support device 1 does not necessarily have to include the learning unit 10. The support device 1 may acquire a learning model generated by an external device via communication or the like, and support pruning of the acquired learning model. The support device 1 is required to include at least the pruning unit 20.

[0033] [2. Operation of the support device] Next, the operation of the support device 1 configured as above will be described with reference to Figs. 2 to 4. Fig. 2 is a flowchart showing the operation (support method) of the support device 1 according to this embodiment. Each operation shown in Fig. 2 is executed by the correction unit 22. In the drawings, layers, neurons, biases, etc. are written in English.

[0034] 2, the correction unit 22 first reads a model (learning model) from the storage unit 30 (S11). The correction unit 22 acquires the learning model created by the learning unit 10 by reading it from the storage unit 30. Then, the correction unit 22 executes the processes from steps S12 to S21 for each layer (loop 1). The correction unit 22 functions as an acquisition unit.

[0035] Next, the correction unit 22 determines whether or not the layer is a DW convolutional layer (DW) based on the information about the learning model (S12). The correction unit 22 functions as a determination unit.

[0036] Next, when the correction unit 22 determines that the layer is a DW convolution layer (Yes in S12), the correction unit 22 acquires a target neuron list (S13). The correction unit 22 may read the target neuron list from the storage unit 30, for example.

[0037] Here, details of the problem of the present disclosure, pruning, etc. will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining pruning and bias term correction in the assistance device 1 according to the present embodiment. In Fig. 3, white circles indicate neurons, dashed circles indicate neurons to be pruned, circles with diagonal hatching indicate bias terms in the DW convolution layer or PW convolution layer, solid arrows indicate weights, and dashed arrows indicate weights connected to neurons to be pruned.

[0038] FIG. 3(a) illustrates an example in which neuron n1 in a normal convolutional layer (CONV) is selected as the pruning target. For example, neuron n1 has a zero output. In this case, the input to neuron n2 is zero, but neuron n2 has a bias term 101, so the bias term 101 is output from neuron n2 to each neuron in the PW convolutional layer. Note that neuron n1 does not have a bias term. The bias term 101 is the bias term of the DW convolutional layer connected to the pruning target neuron (here, neuron n1). Weights w2 and w3 are the weights of the PW convolutional layer subsequent to the DW convolutional layer connected to the target neuron (here, neuron n1).

[0039] As shown in (b) of FIG. 3, only the output from neuron n1 is input to neuron n2 in the DW convolutional layer (DW). Therefore, when neuron n1 is pruned, neuron n2 is also selected as a pruning target, assuming that the output of neuron n2 becomes zero. Also, (b) of FIG. 3 illustrates an example in which weight w1 connecting neurons n1 and n2, weights connecting neuron n2 to each neuron in the PW convolutional layer (e.g., weights w2 and w3), and weights on the input side of neuron n1 (e.g., weights w4 and w5) are selected as weights to be pruned. This selection is performed by the selection unit 21. Note that bias term 101 is an example of a bias term possessed by neuron n2.

[0040] Here, as described above, neuron n2 has bias term 101, so if neuron n2 is pruned, weights w2 and w3 are also pruned, and the weight, bias term 101, is no longer output from neuron n2 to neurons n11 and n12.

[0041] The following equation 1 shows the output of neuron n2, (DW) means the output of the neurons in the DW convolutional layer, and O (CONV) means the output of the neurons in the normal convolutional layer before the DW convolutional layer, and W (DW) is the weight connecting the DW convolution layer and the PW convolution layer, and b (DW) means the bias term of the neuron in the DW convolution layer. The subscript i is a number that identifies the neuron in the convolution layer, and pruned indicates that it is a target for pruning. (DW) denotes the activation function applied to the DW convolution layer. Also, "·" in the formula denotes multiplication (or convolution operation).

[0042]

number

[0043] As can be seen from Equation 1, O (CONV) Even if n is zero, neuron n2 in the DW convolution layer would normally output a value obtained by applying an activation function to the bias term (hereinafter referred to as the bias value). Applying an activation function means that the activation function is also taken into account in the output of a neuron in the DW convolution layer, for example, multiplying the bias term of a neuron in the DW convolution layer by the activation function.

[0044] However, if neuron n2 is the target of pruning, the output from neuron n2 becomes zero, which differs from Equation 1. In other words, the values ​​input to each neuron in the PW convolutional layer will be different before and after the pruning process. This leads to a decrease in the accuracy of the learning model. Note that the bias terms and bias values ​​are expressed as vectors.

[0045] The output of the neuron in the PW convolution layer is shown in the following equations 2 and 3. Equation 2 shows the value output from the neuron before pruning, and equation 3 shows the value output from the neuron after pruning. (PW) means the value of the neuron in the PW convolutional layer before the activation function is applied, and W (PW) means the weight between the neurons in the DW convolutional layer and the neurons in the PW convolutional layer, and b (PW) means the bias value of a neuron in the PW convolution layer. The subscript j is a number that identifies the neuron in the convolution layer, and pruned indicates that the neuron is to be pruned. ji means the weight between the i-th neuron in the DW convolutional layer and the j-th neuron in the PW convolutional layer. Note that the activation function applied to the PW convolutional layer is omitted for the sake of explanation, but in a normal PW convolutional layer, O (PW) An activation function is applied to

[0046]

number

[0047]

number

[0048] From equations 2 and 3, it can be seen that an error shown in equation 4 below occurs.

[0049]

number

[0050] This error corresponds to the multiplication (convolution operation) of the bias term 101 of neuron n2 and the weight (weight w2 or w3) of neurons n2 and n11 or n12. When neuron n2 is pruned, the multiplication (convolution operation) of the bias term 101 of neuron n2 and the weight of neurons n2 and n11 or n12 is skipped. Note that weight w1 is the weight between neurons n2 and n11. Weight w2 is the weight between neurons n2 and n12.

[0051] Therefore, if neuron n2 is the target neuron for pruning, the bias value of neuron n2 will no longer be input to each neuron in the PW convolutional layer (e.g., neurons n11 and n12), resulting in the above-mentioned output error, which may reduce the accuracy of the learning model.

[0052] Therefore, in the present disclosure, the bias term of each neuron in the subsequent PW convolutional layer is corrected based on the bias term corresponding to the neuron to be pruned.

[0053] 2 again, next, based on the information about the learning model, the correction unit 22 determines whether the DW convolution layer has a bias (bias term) (S14). Since there are DW convolution layers that have a bias and layers that do not have a bias, in step S14, it is determined whether the target DW convolution layer is a layer that has a bias.

[0054] Next, when the correction unit 22 determines that the DW convolutional layer has a bias (Yes in S14), it acquires a value based on the bias term corresponding to the target neuron based on the target neuron list (S15). The correction unit 22 acquires, for example, the bias value of the DW convolutional layer connected to the target neuron. Note that if the DW convolutional layer has an identity function, the bias term corresponding to the target neuron (i.e., the bias term of the DW convolutional layer) is acquired as the value based on the bias term of the DW convolutional layer connected to the target neuron in step S15.

[0055] Next, the correction unit 22 determines whether a PW convolutional layer is present after the DW convolutional layer based on the information about the learning model (S16). The correction unit 22 determines whether the DW convolutional layer and the PW convolutional layer are arranged consecutively.

[0056] Note that the subsequent convolutional layer is not limited to being a PW convolutional layer, and therefore the determination in step S16 may be omitted.

[0057] Next, when the correction unit 22 determines that a PW convolutional layer exists after the DW convolutional layer (Yes in S16), the correction unit 22 further determines whether the PW convolutional layer has a bias based on information about the learning model (S17).

[0058] Next, when the correction unit 22 determines that the PW convolutional layer has a bias (Yes in S17), it acquires the bias term of each of the multiple neurons in the PW convolutional layer (S19). The correction unit 22 acquires the bias term of the PW convolutional layer subsequent to the DW convolutional layer connected to the target neuron.

[0059] Furthermore, when the correction unit 22 determines that the PW convolutional layer does not have a bias (No in S17), it generates a bias for the PW convolutional layer (S18). The correction unit 22 generates a matrix with zero elements as a bias term for the PW convolutional layer. In other words, the correction unit 22 obtains the bias term for the PW convolutional layer by generating a matrix with zero elements (S19).

[0060] Next, the correction unit 22 calculates a correction value for each neuron in the PW convolution layer based on the value based on the bias term acquired in step S15 (S20). The correction unit 22 calculates a correction value for correcting the bias term of each neuron in the PW convolution layer. The correction unit 22 executes the process of step S20 for each neuron (loop 2). The correction unit 22 functions as a calculation unit.

[0061] The correction unit 22 calculates the value calculated by the calculation formula shown in Equation 4 above as the correction value of the neuron in the PW convolutional layer.

[0062] That is, the correction unit 22 calculates the above-described error as a correction value. ji Since σ can be different for each neuron in the PW convolutional layer, the value shown in Equation 4 (i.e., the correction value) is calculated individually for each neuron in the PW convolutional layer.

[0063] Referring back to FIG. 2, next, the correction unit 22 overwrites the bias value of the bias term of the learning model learned by the learning unit 10 with the correction value calculated in step S20 (S21). The correction target here is the bias term of the PW convolutional layer. The correction unit 22 individually corrects the bias term of each neuron of the PW convolutional layer based on the correction value calculated in step S20. In other words, the correction unit 22 replaces the bias value of each neuron of the PW convolutional layer with a bias value corrected by the correction value. The corrected bias value is calculated, for example, by the following Equation 5.

[0064]

number

[0065] Here, specific correction values ​​will be described with reference to FIG.

[0066] When neurons n1 and n2 are pruning target neurons, the first correction value for bias term 102 of neuron n11 is calculated by multiplying weight w2 by the bias value of bias term 101 (convolution operation), and the bias value of bias term 102 after correction is the value obtained by adding the first correction value to the bias value of bias term 102 before correction. Also, the second correction value for bias term 103 of neuron n12 is calculated by multiplying weight w3 by the bias value of bias term 101 (convolution operation), and the bias value of bias term 103 after correction is the value obtained by adding the second correction value to the bias value of bias term 103 before correction. In this way, the bias term of each neuron in the PW convolution layer is corrected.

[0067] For example, neuron n3 shown in (b) of Figure 3 is not a neuron to be pruned, so the bias term of the PW convolutional layer is not corrected using the bias term of neuron n3.

[0068] 2 again, next, the correction unit 22 saves the learning model in which the bias term of each neuron in the PW convolutional layer has been corrected (S22). The correction unit 22 stores the learning model in the storage unit 30.

[0069] Furthermore, if the correction unit 22 determines No in any of steps S12, S14, and S16, the process proceeds to after step S21 in the process of loop 1, and continues the process of loop 1.

[0070] In the above, an example has been described in which a bias for a subsequent convolutional layer (in this embodiment, the PW convolutional layer) is generated when the subsequent convolutional layer does not have a bias term (No in S17). However, this is not limiting, and for example, the process may proceed to step S22 and a learning model in which the bias term has not been corrected may be stored in the storage unit 30.

[0071] In step S20, an example in which all correction values ​​of each neuron are calculated has been described, but correction values ​​of only one or more of the neurons may be calculated. For example, correction values ​​of only neurons whose weights are equal to or greater than a predetermined value may be calculated.

[0072] The model evaluation unit 12 may evaluate the learning model that has been pruned as described above. Also, the model creation unit 11 may re-train the learning model that has been pruned and whose bias terms have been corrected.

[0073] [3. Operation of the support device] Next, the effects of the support device 1 that executes the support method described above will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining the effects of the support device 1 according to this embodiment.

[0074] 4 shows the comparison results of the accuracy of the learning model before pruning, after pruning by the conventional method, and after pruning by the proposed method proposed in this disclosure. Note that the number of pruned branches in the pruning process is the same.

[0075] As shown in Figure 4, when pruning is performed using the conventional method that does not correct the bias term, accuracy drops by 9.5%. However, when pruning is performed using the proposed method of the present disclosure that corrects the bias term, accuracy degradation caused by the bias term in the DW convolutional layer due to pruning can be suppressed, and accuracy almost equivalent to that before pruning can be maintained.

[0076] (Embodiment 2) The support device according to this embodiment will be described below with reference to FIG. 5. The following description will focus on differences from embodiment 1, and descriptions of the same or similar content as embodiment 1 will be omitted or simplified. The configuration of the support device according to this embodiment is the same as that of the support device 1 according to embodiment 1, and descriptions thereof will be omitted. The following description will also use the reference numerals of the support device 1 according to embodiment 1.

[0077] In this embodiment, the learning model has a batch normalization (BN) layer after each of the DW convolution layer and the PW convolution layer. The batch normalization layer is arranged to normalize the scale of the output result of each convolution layer. Hereinafter, the batch normalization layer will also be referred to as a BN layer or BN. The batch normalization layer is an example of a normalization layer.

[0078] The BN layer has, as parameters, a scale coefficient (γ) and an offset (beta) that are learned by a learning process by the learning unit 10. The scale coefficient is also called a scale parameter, and the offset is also called a shift parameter. The offset is a fixed value.

[0079] For example, a learning model has a BN layer between a DW convolutional layer and a PW convolutional layer, and a BN layer between the PW convolutional layer and a convolutional layer subsequent to the PW convolutional layer. The DW convolutional layer and the PW convolutional layer can also be said to be connected via the BN layer. For example, when the output from a previous neuron is zero, a beta value (beta value) is output from the BN layer. Depending on the learning model, an activation function may be applied. For example, in a learning model, an activation function may be applied to a BN layer subsequent to a DW convolutional layer, or a BN layer may be connected after an activation function is applied to a DW convolutional layer, or an activation function may be applied to a convolutional layer subsequent to a BN layer. However, in this embodiment, an example in which an activation function is applied to a BN layer subsequent to a DW convolutional layer will be described.

[0080] Although the present embodiment will be described using a batch normalization layer as an example, the normalization layer used in the proposed method of the present disclosure is not limited to the batch normalization layer. The normalization layer may be another normalization layer having an offset parameter, such as a layer normalization layer or an instance normalization layer.

[0081] FIG. 5 is a flowchart showing the operation (assistance method) of the assistance device 1 according to this embodiment.

[0082] 5, when it is determined that the DW convolutional layer does not have a bias (No in S14), and after step S15, the correction unit 22 determines whether the DW convolutional layer has a BN layer based on information about the learning model (S31, S32). Based on the information about the learning model, the correction unit 22 determines whether the subsequent stage of the DW convolutional layer is a BN layer.

[0083] Next, when the correction unit 22 determines that the subsequent stage of the DW convolutional layer is a BN layer (Yes in S31, Yes in S32), it acquires beta (beta value) of the BN layer (S33). The correction unit 22 may acquire beta by, for example, reading out the beta of the BN layer stored in the storage unit 30.

[0084] In this way, step S33 may be executed regardless of the determination result of step S14. Note that, for example, the correction unit 22 may acquire beta only when the DW convolution layer has a bias term and the DW convolution layer has a BN layer.

[0085] Next, the correction unit 22 performs beta_folding (S34). The correction unit 22 performs filtering on the BN layer, converting the beta of the BN layer into a value that takes into account other parameters of the BN layer. For example, the correction unit 22 performs filtering based on the following equation 6: β denotes a shift parameter (beta value), μ denotes the batch mean, γ denotes a scale parameter learned by the learning process by the learning unit 10, σ denotes the batch variance, and eps denotes a small numerical value for preventing division by zero. The numerical value of eps is stored in advance in the storage unit 30, for example.

[0086]

number

[0087] The converted beta is used to calculate the correction value in step S20.

[0088] Note that Equation 6 applies when the output of the preceding DW convolutional layer is zero. If the output of the DW convolutional layer preceding the BN layer is a bias value, the numerators of the two terms on the right side of Equation 6 are the bias value minus the batch mean multiplied by the scale parameter. In other words, the BN layer outputs a value based on the bias term of the preceding DW convolutional layer to the PW convolutional layer following the BN layer.

[0089] When a neuron in a DW convolutional layer or a connection relationship between a DW convolutional layer and a BN layer is pruned, in step S20, a correction value for the bias term is calculated using a value based on the bias term of the preceding DW convolutional layer output from the BN layer. That is, even when a learning model has a BN layer subsequent to a DW convolutional layer, the bias term of each neuron in the PW convolutional layer is corrected using a value based on the bias term of the DW convolutional layer connected to the neuron to be pruned in the DW convolutional layer (here, a value based on the bias term and the converted beta).

[0090] Furthermore, when the correction unit 22 determines that the PW convolutional layer does not have a bias (No in S17) and after step S19, the correction unit 22 determines whether the PW convolutional layer has a BN layer based on information about the learning model (S35, S36). Based on information about the learning model, the correction unit 22 determines whether the PW convolutional layer is followed by a BN layer.

[0091] Next, when the correction unit 22 determines that the layer subsequent to the PW convolutional layer is a BN layer (Yes in S35, Yes in S36), the correction unit 22 acquires beta (beta value) of the BN layer subsequent to the PW convolutional layer (S37). The correction unit 22 may acquire beta by, for example, reading out beta stored in the storage unit 30.

[0092] In this way, step S37 may be executed regardless of the determination result of step S17. Note that, for example, the correction unit 22 may acquire beta only when the PW convolutional layer has a bias term and the PW convolutional layer has a BN layer.

[0093] Next, the correction unit 22 executes weight_folding (S36). Weight_folding is a process of correcting the weight (Wbni shown in the following formula 7) between the PW convolutional layer and the BN layer subsequent to the PW convolutional layer, which has been learned by the learning process by the learning unit 10. i and the weights of the PW convolutional layer that are updated taking the BN layer into consideration are Wbni, the correction unit 22 corrects the weights based on the following Equation 7.

[0094]

number

[0095] The corrected weight Wbni is used to calculate the correction value in step S20. When the BN layer is taken into consideration, the correction destination is beta (beta value) of the BN layer subsequent to the PW convolutional layer, and is calculated, for example, by the following Equation 8.

[0096]

number

[0097] The corrected beta (beta with a hat) shown in Equation 8 (DW) ) indicates the beta of the BN layer after the DW convolution layer converted by Equation 6, Wbnjipruned indicates the weight of the PW convolution layer corrected by the BN layer after the PW convolution layer, and β (PW) indicates the beta of the BN layer after the PW convolutional layer, and the corrected beta indicates the beta of the BN layer after the PW convolutional layer.

[0098] In this way, in this embodiment, the correction value is calculated using the updated weight (Wbni shown in Equation 7) of the PW convolutional layer and the bias term of the DW convolutional layer. This correction value is used to correct beta of the BN layer.

[0099] If the determination in step S35 is No, in step S20, the bias term of the PW convolutional layer is corrected using the following equation 9.

[0100]

number

[0101] Next, the correction unit 22 overwrites the beta of the BN layer of the learning model learned by the learning unit 10 with the correction value calculated in step S20 (S21). The correction target here is the beta of the BN layer. The correction unit 22 corrects the beta of the BN layer based on the correction value calculated in step S20. It can also be said that the correction unit 22 replaces the beta of the BN layer with the beta corrected by the correction value.

[0102] Furthermore, if the correction unit 22 determines No in any of steps S12, S16, and S32, the correction unit 22 proceeds to the step after step S21 in the processing of loop 1, and continues the processing of loop 1.

[0103] (Other embodiments) The assistance device and the like according to one or more aspects have been described above based on Embodiments 1 and 2, but the present disclosure is not limited to Embodiments 1 and 2. As long as they do not deviate from the spirit of the present disclosure, various modifications that a person skilled in the art may make to the present embodiment and the present modifications, and forms constructed by combining components of different embodiments may also be included in the present disclosure.

[0104] For example, in the above-described first and second embodiments, the learning model has been described as having one DW convolutional layer, but the present invention is not limited thereto and may have multiple DW convolutional layers. For example, the learning model may have multiple DW convolutional layers arranged consecutively. In this way, when DW convolutional layers are consecutive, correction may be performed by performing a convolutional operation with a subsequent convolutional layer (e.g., a PW convolutional layer) after taking into consideration the calculation of the weights and biases of the consecutive DW convolutional layers in advance. For example, the bias term after correction of the subsequent convolutional layer is shown in the following Equation 10.

[0105]

number

[0106] In Equation 10, L denotes the number of consecutive DW convolutional layers, and W (DWm) denotes the weight of the mth DW convolutional layer, and b (DWl) indicates the bias term of the l-th DW convolution layer. As shown in Equation 10, the weight and bias term of the successive DW convolution layers are calculated in advance, and the value obtained by multiplying this value by the weight of the subsequent convolution layer (convolution calculation) is the correction value. The corrected bias term is calculated by adding this correction value to the bias term of the subsequent convolution layer. Weight W (PW) is the weight between the final DW convolutional layer and the subsequent convolutional layer in the series of DW convolutional layers.

[0107] For example, a first DW convolutional layer and a subsequent convolutional layer are connected via a second DW convolutional layer, and a neuron in the first DW convolutional layer and a neuron in the second DW convolutional layer are connected via a neuron to be pruned. In other words, correction for a case where two DW convolutional layers are arranged consecutively will be described. In this case, the correction value is calculated by multiplying (convolutionally) the second weight of a neuron in the second DW convolutional layer by the first bias value of the neuron in the first DW convolutional layer, and adding the resulting value to (convolutionally) the second bias value of the neuron in the second DW convolutional layer. Specifically, the correction value is calculated by multiplying (convolutionally) the sum of the first bias value and the second bias value by the weight between the neuron. The second bias value is a value based on the bias term of the neuron. The second bias value may be, for example, the bias term of the one neuron itself, or may be a value obtained by applying an activation function of the second DW convolutional layer to the bias term.

[0108] Note that the parameters of the BN layer (beta after beta_folding) and calculations corresponding to the bias terms can all be corrected using calculations similar to those in Equation 10.

[0109] In addition, in the above-described first and second embodiments, the correction value is calculated based on the bias term and the weight, but the present invention is not limited to this, and it is sufficient that the correction value is calculated based on at least the bias term. In other words, the weight does not need to be used in calculating the correction value.

[0110] Furthermore, in the above-described first and second embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0111] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.

[0112] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0113] Furthermore, the support devices according to the first and second embodiments may be realized as a single device or may be realized by multiple devices. When the support device is realized by multiple devices, the components of the support device may be distributed among the multiple devices in any manner. When the support device is realized by multiple devices, the communication method between the multiple devices is not particularly limited, and may be wireless communication or wired communication. Furthermore, wireless communication and wired communication may be combined between the devices.

[0114] Furthermore, each of the components described in the first and second embodiments above may be implemented as software or, typically, as an LSI, which is an integrated circuit. These components may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. While the term "LSI" is used here, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI depending on the level of integration. Furthermore, the integration method is not limited to LSI; it may also be implemented using a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. It is also possible to use a field programmable gate array (FPGA), which can be programmed after LSI fabrication, or a reconfigurable processor, which allows the connection or settings of circuit cells within an LSI to be reconfigured. Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or a derivative technology, that technology may naturally be used to integrate the components.

[0115] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip, and is specifically a computer system consisting of a microprocessor, ROM, RAM, etc. The ROM stores computer programs. The system LSI achieves its functions when the microprocessor operates in accordance with the computer programs.

[0116] Furthermore, one aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the support method shown in either FIG. 2 or FIG.

[0117] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes.

[0118] (Addendum) The above description of the first and second embodiments discloses the following techniques.

[0119] (Technology 1) A method for supporting pruning of a learning model that uses a neural network having a first depthwise convolutional layer and a convolutional layer subsequent to the first depthwise convolutional layer, wherein the learning model includes a target neuron to be pruned in the first depthwise convolutional layer, the method comprising: determining whether the first depthwise convolutional layer has a bias term; if the first depthwise convolutional layer has a bias term, obtaining a first bias value based on the bias term; and calculating a correction value for correcting the bias term in the subsequent convolutional layer using a value based on the first bias value.

[0120] This allows the bias terms of the subsequent convolutional layers to be corrected based on the bias terms to be pruned, thereby preventing the accuracy of the learning model from deteriorating due to the bias terms to be pruned. Therefore, when pruning the learning model, the learning model can be compressed (reduced in size) while preventing accuracy degradation.

[0121] (Technology 2) This is a supporting method of Technology 1, in which the correction value is calculated based on the first bias value and a weight between the target neuron and a neuron in the subsequent convolutional layer.

[0122] This allows the weight to be taken into consideration when calculating the correction value, thereby further preventing the deterioration of the accuracy of the learning model caused by the pruned bias term.

[0123] (Technology 3) The correction value is calculated by multiplying the first bias value by the weight.

[0124] This allows the value output to the subsequent convolutional layer when the input to the first depthwise convolutional layer is zero to be calculated as the correction value. Therefore, when the input to the first depthwise convolutional layer is zero or a value close to zero, a more appropriate correction value can be calculated. This leads to further suppression of accuracy degradation of the learning model.

[0125] (Technology 4) This is a support method according to any one of techniques 1 to 3, wherein the learning model further includes a normalization layer, the first depthwise convolutional layer and the subsequent convolutional layer are connected via the normalization layer, and the correction value is calculated based on the first bias value and a shift parameter of the normalization layer.

[0126] This makes it possible to suppress deterioration in accuracy of a learning model caused by the pruned bias terms, even in a learning model in which a normalization layer is arranged after the first depthwise convolutional layer.

[0127] (Technology 5) The support method according to any one of techniques 1 to 4, wherein the learning model further includes a second depthwise convolutional layer, the first depthwise convolutional layer and the subsequent convolutional layer are connected via the second depthwise convolutional layer, one neuron in the first depthwise convolutional layer is connected to one neuron in the second depthwise convolutional layer, and the correction value is calculated by multiplying the first bias value by a second weight of the one neuron in the second depthwise convolutional layer and adding the product to the second bias value of the one neuron in the second depthwise convolutional layer.

[0128] As a result, the bias terms of the subsequent convolutional layers are corrected using the bias terms of each of the successively arranged depthwise convolutional layers, so that even in a learning model having successive depthwise convolutional layers, deterioration in the accuracy of the learning model due to the pruned bias terms can be suppressed.

[0129] (Technology 6) Furthermore, the support method is any one of techniques 1 to 5, which corrects a bias term in the subsequent convolutional layer based on the calculated correction value, and saves the learning model having the corrected bias term.

[0130] This makes it possible to perform correction processing for the bias term.

[0131] (Technology 7) The subsequent convolution layer is a pointwise convolution layer, and is a support method according to any one of techniques 1 to 6.

[0132] This makes it possible to suppress deterioration in accuracy of a learning model that has a pointwise convolution layer as a subsequent convolution layer, which is caused by the bias terms that are pruned.

[0133] (Technology 8) The supporting method according to any one of techniques 1 to 7, wherein it is determined whether the subsequent convolutional layer has a bias term, and if the subsequent convolutional layer has a bias term, the correction value is calculated.

[0134] This reduces the amount of processing required by the assistance device that executes the assistance method.

[0135] (Technology 9) The supporting method according to any one of Techniques 1 to 8, wherein the first bias value based on the bias term is calculated based on a bias term included in the first depthwise convolution layer and an activation function included in the first depthwise convolution layer.

[0136] This allows the correction value to be calculated taking into account the activation function that each layer in a neural network typically has, thereby enabling more accurate calculation of the correction value and improving the versatility of the support method.

[0137] (Technology 10) An assistance device that assists in pruning of a learning model that uses a neural network having a depthwise convolutional layer and a convolutional layer subsequent to the depthwise convolutional layer, wherein the learning model includes a target neuron to be pruned in the depthwise convolutional layer, and the assistance device includes: a determination unit that determines whether the depthwise convolutional layer has a bias term; an acquisition unit that, if the depthwise convolutional layer has a bias term, acquires a bias value based on the bias term; and a calculation unit that uses a value based on the bias value to calculate a correction value for correcting the bias term in the subsequent convolutional layer.

[0138] This provides the same effect as the above-mentioned support method.

[0139] (Technology 11) This is a program for implementing any of the support methods from Techniques 1 to 9.

[0140] This provides the same effect as the above-mentioned support method.

[0141] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of the system, method, integrated circuit, computer program, or recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet. [Industrial Applicability]

[0142] The present disclosure is useful for an information processing device or the like that prunes a learning model. [Explanation of symbols]

[0143] 1 Support equipment 10 Learning Department 11 Model Creation Department 12 Model Evaluation Section 20 Pruning Department 21 Selection Department 22 Correction unit (judgment unit, acquisition unit, calculation unit) 23 Pruning unit 30 Storage section 101, 102, 103 Bias terms n1, n2, n3, n11, n12 neurons w1, w2, w3, w4, w5 weights

Claims

1. A method for supporting pruning of a learning model using a neural network having a first depthwise convolutional layer and a convolutional layer subsequent to the first depthwise convolutional layer, the method comprising: The learning model includes a target neuron to be pruned in the first depthwise convolutional layer, The support method includes: determining whether the first Depthwise convolutional layer has a bias term; If the first Depthwise convolutional layer has a bias term, obtain a first bias value based on the bias term; Calculating a correction value for correcting a bias term of the subsequent convolutional layer using a value based on the first bias value. How to help.

2. The correction value is calculated based on the first bias value and a weight between the target neuron and a neuron in the subsequent convolutional layer. The support method according to claim 1 .

3. The correction value is calculated by multiplying the first bias value by the weight. The support method according to claim 2.

4. The learning model further comprises a normalization layer; The first Depthwise convolutional layer and the subsequent convolutional layer are connected via the normalization layer, The correction value is calculated based on the first bias value and a shift parameter of the normalization layer. The support method according to claim 1 .

5. The training model further comprises a second depthwise convolutional layer; the first depthwise convolutional layer and the subsequent convolutional layer are connected via the second depthwise convolutional layer; One neuron in the first depthwise convolutional layer is connected to one neuron in the second depthwise convolutional layer; The correction value is calculated by multiplying the first bias value by a second weight of the one neuron in the second depthwise convolutional layer, and adding the multiplied value to the second bias value of the one neuron in the second depthwise convolutional layer. The support method according to claim 1 .

6. Further, based on the calculated correction value, a bias term of the subsequent convolutional layer is corrected; Save the trained model with the corrected bias term. The support method according to any one of claims 1 to 5.

7. The latter convolutional layer is a Pointwise convolutional layer. The support method according to any one of claims 1 to 5.

8. Determining whether the subsequent convolutional layer has a bias term; If the subsequent convolutional layer has a bias term, calculate the correction value. The support method according to any one of claims 1 to 5.

9. The first bias value based on the bias term is calculated based on a bias term included in the first depthwise convolutional layer and an activation function included in the first depthwise convolutional layer. The support method according to any one of claims 1 to 5.

10. An assistance device that assists in pruning a learning model using a neural network having a depthwise convolutional layer and a convolutional layer subsequent to the depthwise convolutional layer, the device comprising: The learning model includes a target neuron to be pruned in the Depthwise convolutional layer, The support device includes: a determination unit that determines whether the Depthwise convolutional layer has a bias term; When the Depthwise convolution layer has a bias term, an acquisition unit that acquires a bias value based on the bias term; a calculation unit that calculates a correction value for correcting a bias term included in the subsequent convolutional layer using a value based on the bias value. Support equipment.

11. A program for executing the support method according to any one of claims 1 to 5.

Images