An Improved Method for Lightweighting Convolutional Neural Networks

Abstract

This application provides an improved method for lightweighting convolutional neural networks. In this method, group convolution is composed of a set of normal convolutions. Each normal convolution has the same number of input and output channels, defined as the group channel. That is, by setting a group channel, the normal convolutions with non-zero kernels in the model are replaced with group convolutions and pointwise convolutions with a defined group channel value. This application, from an instruction-level optimization perspective, reduces parameters and computational load while maintaining model accuracy, and is highly compatible with instruction-level optimization for embedded devices.

Description

Technical Field

[0001] This invention relates to the field of convolutional neural network technology, and in particular to an improved method for lightweighting convolutional neural networks. Background Technology

[0002] With the development of computer technology, the computation of neural network models on embedded devices remains a significant challenge due to memory and power consumption limitations. The main idea behind manually designing lightweight models is to develop more efficient computational methods for convolution, reducing parameters without sacrificing network performance. Variations of standard convolution include group convolution, depthwise convolution, and pointwise convolution. Lightweight convolutional neural networks include MobileNet, ShuffleNet, and SqueezeNet.

[0003] However, existing methods have drawbacks, as follows:

[0004] Existing lightweighting efforts focus on algorithm optimization without considering instruction-level optimization. For example, channel shuffle in ShuffleNet consumes a lot of memory and pointer jumps. In actual operation of neural networks on embedded devices, instruction-level optimization has a significant impact.

[0005] In addition, the following are commonly used terms in the prior art:

[0006] 1. Convolutional Neural Network: A feedforward neural network that includes convolutional computation and has a deep structure. It is one of the representative algorithms of deep learning.

[0007] 2. Convolution: Ordinary convolution.

[0008] 3. Group convolution: Divide the channel into several groups, and perform a regular convolution on each group.

[0009] 4. Pointwise convolution: Pointwise convolution with a 1x1 kernel, a standard convolution. Summary of the Invention

[0010] To address the problems in the prior art, the purpose of this application is to improve the lightweighting of convolutional neural networks by setting a defined groupchannel value to transform ordinary convolutions with kernels other than 1 into group convolutions and pointwise convolutions with defined channels.

[0011] Specifically, this application provides an improved method for lightweighting convolutional neural networks. In this method, group convolution is composed of a set of normal convolutions, where the input channel and output channel values ​​of each normal convolution are equal, defined as a group channel. That is, a specific group channel value is set, and normal convolutions with kernel values ​​other than 1 are replaced with group convolutions and pointwise convolutions with defined channel values. The group convolution is a set of normal convolutions with equal channel values, i.e., equal input and output channels. The input / output channel values ​​are represented by "group channel". The number of normal convolutions in the group convolution is "groupnum". The product of "group channel" and "groupnum" equals the value of the input channel of the replaced normal convolution. Thus, the input feature map of the normal convolution can also be used as the input of the group convolution. The input channel value and output channel value of the group convolution are equal, both equal to the input channel value of the replaced normal convolution. Channel value; the pointwise convolution is a regular convolution with a 1x1 kernel.

[0012] The method includes the following steps:

[0013] S1, Adjustment: In addition to the input and output layers, the intermediate layers in the convolutional neural network select appropriate group channel values ​​according to the requirements. In order to make them meet the replacement conditions, the input channel of ordinary convolutional layers with kernels other than 1*1 is adjusted to an integer multiple of the group channel.

[0014] That is, select appropriate group channels based on the computing power and memory space of the platform according to the requirements, and set the number of channels for each convolution layer so that the number of channels for each layer is an integer multiple of the group channel.

[0015] S2, Replacement: Remove the input and output layers of the neural network. For ordinary convolutions with kernel sizes not equal to 1, replace them with group convolution and pointwise convolution. In the method, the number of input and output channels of the ordinary convolutions that make up the group convolution is equal. In this application, group convolution is composed of a group of ordinary convolutions with the same number of channels. A single ordinary convolution refers to one of these identical ordinary convolutions, which has universal representativeness.

[0016] S3, Result: The final result is that the computation in the network other than the input layer and the output layer is simplified to group convolution consisting of ordinary convolutions with fixed input channel and output channel values, and ordinary convolution with a 1x1 kernel. The fixed value is the appropriate group channel value selected in step S1.

[0017] The replacement step in S2 includes the following:

[0018] S2.1 Assume that the number of input channels in a normal convolution is in_num, the number of output channels is out_num, the number of fixed group channels is g_channel, and the number of groups is g_nums, satisfying in_num = g_channel * g_nums;

[0019] S2.2, at this point, the number of output channels of the group convolution is still in_num. Because after the replacement, the group convolution is calculated first and the pointwise convolution is calculated later, the output channels of the pointwise convolution are out_num. Set the number of input channels of the pointwise convolution to in_num and the number of output channels to out_num. In this way, the combination of group convolution and pointwise convolution maintains the same input and output channels as the original ordinary convolution, thus completing the replacement.

[0020] In this context, the kernel size of the group convolution is equal to the kernel size of the replaced ordinary convolution; group num is the number of groups into which the input feature map is divided, and the number of channels in the input feature map is the same as the number of input channels in the replaced ordinary convolution and also the number of input channels in the group convolution. This number divided by the number of group channels gives the number of groups, group num.

[0021] The method does not require a fixed number of group channels. Instead, the user selects an appropriate value based on the network as the fixed value for this replacement process. This value can be 8, 16, or 32. The fixed value here means that this value is determined during the user's replacement process, and there will be no situation where, for example, 32 is used to replace the first ordinary convolution and 16 is used to replace the second ordinary convolution. The value of the group channel is defined as being divisible by the input channel value of the ordinary convolution. If not, the original network model needs to be modified to ensure that the input channel value of the replaced group convolution is divisible by the group channel value. During the replacement process, the convolution kernel size is not changed, that is, the kernel size of the group convolution and the replaced convolution are equal. The kernel size of the pointwise convolution is fixed at 1 during the replacement process, and the kernel size of the group convolution is equal to the kernel size of the replaced ordinary convolution. This does not change the receptive field of the network. In a convolutional neural network, the receptive field is defined as the size of the region mapped on the input image by the pixels on the feature map output by each layer of the convolutional neural network. To explain further, a point on the feature map corresponds to a region on the input map. The receptive field is defined to explain why the kernel size of the group convolution is the same as the size of the convolution being replaced, that is, it does not change some inherent characteristics of the network. During the replacement process, the kernel size of the pointwise convolution is 1, which is fixed, and the kernel size of the group convolution is equal to the kernel size of the ordinary convolution being replaced. The number of channels in each layer is set according to the needs so that it can be divided by the group channel. This allows the intermediate layer convolutions outside the input and output layers to be replaced by group convolution and pointwise convolution, and the number of group channels in group convolution is a fixed value.

[0022] Configure the number of channels in each layer according to requirements so that it is divisible by the group channel. This allows the intermediate convolutional layers outside the input and output layers to be replaced with group convolution and pointwise convolution. The number of channels in the group convolution is a fixed value, which is set by the user according to their actual situation and serves as a fixed value in this replacement process. If a different network is used or the network's operating platform changes, using another suitable fixed value will yield better results.

[0023] In the method described, if the kernel k equals 3, the input channel equals 256, and the output channel equals 512, then the parameters of the ordinary convolution are 3*3*256*512=1179648. After the replacement, the group channel is set to 32, then the group number is 256 / 32=8, that is, 8 group convolutions with 32 group channels each, and the total number of channels is still 512. The total number of parameters used is the parameters of the group convolution plus the parameters of the pointwise convolution, 3*3*32*32*8+1*1*256*512=73728+131072=204800, and the total number of parameters is approximately 204800÷1179648≈0.174 of the ordinary convolution. Therefore, after step S2, the network calculation, except for the input and output layers, is simplified to group convolutions with a kernel of 3*3 and a group channel of 32 and pointwise convolutions.

[0024] You can also set the group channel to 16, which will result in 256 / 16 = 16 3x3 group convolutions with 16 group channels each.

[0025] In step S1, if the number of channels in one layer is not divisible by the set number of group channels, the result is rounded up and multiplied by the number of channels. The result is then set as the input channel number of that layer, which is divisible by the group channel number. For example, if the group channel number is set to 32, but there is a layer with 80 channels, the value of 80 / 32 is 2.5, rounded up to 3, and then multiplied by 32 to get 96. The input channel number of that layer is then set to 96.

[0026] In step S1, when the group channel size is equal to the input channel size of the original convolutional network, the model becomes a combination of normal convolution and point convolution, with the most parameters, the largest computational and memory requirements, and the strongest model capability. When the group channel value is 1, the model becomes depthwise separable convolution, with the fewest parameters, the smallest computational and memory requirements, and the weakest model capability. Therefore, it is necessary to select an appropriate group channel based on the platform's computational capabilities and memory space.

[0027] Therefore, the advantage of this application is that it reduces the number of parameters and computational load in the algorithm from the perspective of instruction-level optimization, ensuring the accuracy of the model, and is very friendly to instruction-level optimization of embedded devices. Attached Figure Description

[0028] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0029] Figure 1 This is a schematic diagram illustrating how ordinary convolutions with kernel sizes not equal to 1 are replaced by group convolution and pointwise convolution in the method of this invention.

[0030] Figure 2 This is a schematic diagram illustrating how the intermediate convolutional layers outside the beginning and end are replaced with group convolution and pointwise convolution in the method of this invention, and the number of channels in the group convolution is a fixed value.

[0031] Figure 3 This is a flowchart of the method of the present invention. Detailed Implementation

[0032] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0033] This application relates to an improved method for lightweighting convolutional neural networks. In this method, group convolution consists of a set of ordinary convolutions with the same number of channels. A single ordinary convolution refers to one of these identical ordinary convolutions, possessing universal representativeness. The input and output channels of a single ordinary convolution are the same; this value is defined as the group channel, which is equal to the aforementioned input and output channel values. That is, a group channel value is set, and ordinary convolutions with kernels other than 1 in the model are replaced with group convolutions and pointwise convolutions with determined channel values. The group convolution is a set of ordinary convolutions with the same number of channels (equal to the input and output channels). The group channel represents the input / output channel value. The number of ordinary convolutions in the group convolution is group_num. The product of group_channel and group_num is equal to the input channel value of the replaced ordinary convolution. Thus, the input feature map of the ordinary convolution... The map can also be used as input to the group convolution. The value of the output channel of the group convolution is equal to the value of the input channel of the replaced ordinary convolution. In this application, the number of input and output channels of the group convolution is equal to the number of input channels of the replaced ordinary convolution. The pointwise convolution is an ordinary convolution with a 1x1 kernel.

[0034] In this method, the input and output channel values ​​of the pointwise convolution are the same as those of the replaced ordinary convolution. The output feature map of the group convolution is used as the input. The output feature map of the group convolution is a tensor, and its number of channels is the same as the output channel of the group convolution. The output feature map of the group convolution is used as the input of the pointwise convolution. Its number of channels must be equal to the input channel of the convolution in order to complete the convolution calculation.

[0035] The method described does not fix the number of group channels, but this value can be determined based on the situation. This value is generally divisible by the number of input channels of a regular convolution. If not, the original network model needs to be modified so that the input channel value of the group convolution to be replaced is divisible by the number of group channels. For example, for a regular 3x3 convolution with 256 input channels, the number of group channels can be set to 32. This results in 256 / 32 = 8 3x3 group convolutions with 32 group channels each, maintaining a total of 512 channels. Alternatively, the number of group channels can be set to 16, resulting in 256 / 16 = 16 3x3 group convolutions with 16 group channels each. The method replaces regular convolutions with kernels other than 1 with group convolutions and pointwise convolutions with defined channels. Convolution; where group convolution divides the channel into n groups, as in the previous example, a regular convolution with 256 input channels is divided into 16 groups, each group having 16 input and output channels, and each group performs a regular convolution. The pointwise convolution is a regular convolution with a 1x1 kernel. During the replacement process, the kernel size of pointwise convolution is fixed at 1, and the kernel size of group convolution is equal to the kernel size of the regular convolution being replaced. During the replacement process, the kernel size does not change, that is, the kernel size of group convolution and the replaced convolution are equal, so the receptive field of the network is not changed. The receptive field is defined as the size of the region mapped on the input image by the pixels on the feature map output by each layer of the convolutional neural network. It means that a point on the feature map corresponds to a region on the input map. In simple terms, it means that the output pixel is calculated from a certain number of input pixels. The explanation here is that the kernel size of the replacement group convolution and the replaced ordinary convolution are the same, so that the receptive field and the original network are consistent.

[0036] The method can be divided into the following steps:

[0037] S1. Adjust the size of the group channel according to actual needs, balancing efficiency and model capability. For example, when the group channel size equals the original input channel size, the model becomes a combination of normal convolution and point convolution, with the most parameters, the largest computational and memory requirements, and the strongest model capability. When the group channel value is 1, the model becomes depthwise separable convolution, with the fewest parameters, the smallest computational and memory requirements, and the weakest model capability. Therefore, choosing an appropriate group channel based on the platform's computing power and memory space can balance efficiency and model capability. The appropriate group channel should be chosen according to requirements; here, "appropriate" is related to the platform's computing power and memory space. With ample resources, a larger value can be set, and vice versa. The number of channels in each convolutional layer should be set so that the number of channels in each layer is an integer multiple of the group channel count. Aside from the input and output layers of the neural network, the number of channels in each layer can be flexibly set according to requirements. For example, if the group channel count is set to 32, but one layer has 80 channels, then 80 / 32 (2.5) can be used, rounded up to 3, and multiplied by 32 to get 96. Setting the input channel count of this layer to 96 is divisible by the group channel count, thus satisfying the condition of replacing the intermediate convolutional layers (excluding the first and last layers) with group convolution and pointwise convolution.

[0038] S2. For ordinary convolutions with kernel sizes not equal to 1, replace them with group convolution and pointwise convolution. The replacement steps are as follows: Figure 1As shown: Assume that the number of input channels in normal convolution is in_num, the number of output channels is out_num, the fixed number of group channels is g_channel, and the number of groups is g_nums, satisfying in_num = g_channel * g_nums. In this case, the output channel of the group convolution remains in_num. The input channel of the pointwise convolution is set to in_num, and the output channel is set to out_num. This way, the combination of group convolution and pointwise convolution maintains the same input and output channels as the original normal convolution. The replacement is complete; the input channel and output channel are the input channel (number of input channels) and output channel (number of output channels) of the normal convolution, respectively, and the kernel size is k; group num is the number of groups into which the input feature map is divided.

[0039] S3. After step S2 is completed, most of the network calculations are simplified, and the input channel and output channel are ordinary convolutions with fixed values, and the convolution kernel is an ordinary convolution with a 1*1 kernel.

[0040] For ease of understanding, the method described in this application can be further described as follows:

[0041] 1. For ordinary convolutions with kernel sizes other than 1, use group convolution and pointwise convolution instead, such as... Figure 1 As shown.

[0042] This model can also consider depthwise separable convolution as a special case of this invention, because depthwise convolution can be seen as group convolution with a group channel value of 1. Therefore, depthwise separable convolution can be seen as a special case of the replacement method in this application. The input channel and output channel are the input and output channels of ordinary convolution, respectively, and the kernel size is k. The group num is the number of groups into which the input feature map is divided. Taking a convolution kernel of 3, an input channel of 256, and an output channel of 512 as an example, the parameters of ordinary convolution are 3*3*256*512=1179648. After the replacement, setting the group channel to 32, the group number becomes 256 / 32 = 8. The total number of parameters used is the sum of the parameters from group convolution and pointwise convolution: 3*3*32*32*8 + 1*1*256*512 = 73728 + 131072 = 204800. The total number of parameters is approximately 0.174 of that from ordinary convolution. It can be seen that reducing the number of parameters reduces both memory usage and computational cost. This data illustrates the advantage of this method in terms of computational cost and memory usage compared to the original method (204800 ÷ 1179648). The size of the group channel can be adjusted according to actual needs, balancing efficiency and model capability.

[0043] 2. Based on requirements, select an appropriate group channel and set the number of channels in each convolutional layer so that the number of channels in each layer is an integer multiple of the group channel count. Aside from the input and output layers of the neural network, the number of channels in each layer can be flexibly set. This allows you to replace the intermediate convolutional layers (excluding the beginning and end layers) with group convolution and pointwise convolution, such as... Figure 2 As shown.

[0044] 3. Optimizations are made for group channel convolution and pointwise convolution. The most common ordinary convolution kernel in current networks is 3*3. Therefore, after step 2, most of the network computation can be simplified to ordinary convolution with a 3*3 kernel and fixed input and output channels, and ordinary convolution with a 1*1 kernel. Group convolution itself is a combination of a group of ordinary convolutions. Another characteristic of group convolution in this application is that the input and output channels of each ordinary convolution in the group are the same. Pointwise convolution itself is an ordinary convolution with a 1*1 kernel, which is beneficial for instruction-level optimization and even hardware implementation in embedded devices.

[0045] In summary, the method can be simplified as follows: Figure 3 As shown, the method includes the following steps:

[0046] S1, Adjustment: In addition to the input and output layers, the intermediate layers in the convolutional neural network select appropriate group channel values ​​according to the requirements. In order to make them meet the replacement conditions, the input channel of ordinary convolutional layers with kernels other than 1*1 is adjusted to an integer multiple of the group channel.

[0047] That is, select appropriate group channels based on the computing power and memory space of the platform according to the requirements, and set the number of channels for each convolution layer so that the number of channels for each layer is an integer multiple of the group channel.

[0048] S2, Replacement: Remove the input and output layers of the neural network. For ordinary convolutions with kernel sizes not equal to 1, replace them with group convolution and pointwise convolution. In the method, the number of input and output channels of the ordinary convolutions that make up the group convolution is equal. In this application, group convolution is composed of a group of ordinary convolutions with the same number of channels. A single ordinary convolution refers to one of these identical ordinary convolutions, which has universal representativeness.

[0049] S3, Result: The final result is that the computation in the network other than the input layer and the output layer is simplified to group convolution consisting of ordinary convolutions with fixed input channel and output channel values, and ordinary convolution with a 1x1 kernel. The fixed value is the appropriate group channel value selected in step S1.

[0050] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An improved method for lightweighting a convolutional neural network, the method comprising: The neural network is operated in an embedded device, the method is used for reducing parameters and calculation amount on an algorithm, ensuring accuracy of the model, and being friendly to instruction level optimization of the embedded device, group convolution in the method is composed of a group of normal convolution, wherein an input channel and an output channel of each normal convolution are equal in channel value, and the group convolution is defined as a group channel, a certain group channel value is set, a normal convolution with a convolution kernel not equal to 1 in the model is replaced by group convolution with a certain channel value and pointwise convolution, wherein the group convolution is a set of normal convolution with equal input channel and output channel, group channel is used to represent the input / output channel value, the number of normal convolution in the group convolution is group num, the product of group channel and group num is equal to the value of the input channel of the replaced normal convolution, so that the input feature map of the normal convolution can also be used as the input of the group convolution, the input channel value and the output channel value of the group convolution are equal, and both are equal to the input channel value of the replaced normal convolution; the pointwise convolution is normal convolution with a 1*1 convolution kernel; The method Comprise the following steps: S1, adjustment: in the convolutional neural network, except for the input layer and the output layer, the intermediate layer selects a suitable group channel value according to the demand, so as to make it meet the replacement condition, and the input channel of the normal convolution layer with a convolution kernel not equal to 1*1 is adjusted to be an integer multiple of the group channel; that is, a suitable group channel is selected according to the demand, the calculation ability of the running platform and the memory space, the channel number of each layer is set, so that the channel number of each layer is an integer multiple of the group channel; S2, replacement: except for the input layer and the output layer of the neural network, for the normal convolution with a convolution kernel size not equal to 1, group convolution and pointwise convolution are used for replacement, and in the method, the input channel and the output channel of the normal convolution constituting the group convolution are equal. S3, result: the final result is that the part of the network except the input layer and the output layer is simplified to the group convolution composed of the ordinary convolution of the input channel and the output channel with the determined value, and the ordinary convolution kernel is 1x1, and the determined value is the appropriate group channel value selected in step S1.

2. The improved method for lightweighting convolutional neural networks of claim 1, wherein, The replacement step in S2 includes the following: S2.1, assuming that the channel number of the input channel of the ordinary convolution is in_num, the channel number of the output channel is out_num, the channel number of the fixed group channel is g_channel, the number of groups is g_nums, and in_num=g_channel*g_nums is satisfied; S2.2, at this time, the output channel number of the group convolution is still in_num, the channel number of the input channel of the pointwise convolution is set to in_num, and the channel number of the output channel is out_num, so that the combination of the group convolution and the pointwise convolution keeps the same input channel and output channel as the original ordinary convolution, and the replacement is completed; Wherein, the convolution kernel size of the group convolution is equal to the convolution kernel size of the replaced ordinary convolution; groupnum is the number of groups of input input feature map, and the channel number of input feature map is the input channel number of the replaced ordinary convolution, which is also the input channel number of the group convolution. This number is divided by the group channel number, that is, the number of groups groupnum.

3. The improved method for lightweighting convolutional neural networks of claim 1, wherein, The method does not require a fixed number of group channels, but a user selects a suitable value according to a network as a determined value in this replacement process, where the determined value refers to a value determined in the user's replacement process, the value defined as the group channel value can be divided by the input channel value of the ordinary convolution, if not, the original network model needs to be modified to make the input channel value of the replaced group convolution divisible by the group channel value; in the replacement process, the convolution kernel size will not be changed, that is, the convolution kernel size of the group convolution and the replaced convolution is equal, the channel number of each layer is set according to the requirement to be divisible by the group channel, so that the intermediate layer convolution outside the input layer and the output layer can be replaced by the group convolution and the pointwise convolution, and the group channel number of the group convolution is a determined value.

4. The improved method for lightweighting convolutional neural networks of claim 3, wherein, The determined value in this replacement process includes 8, or 16, or 32.

5. The improved method for lightweighting convolutional neural networks of claim 3, wherein, In the method, assuming that the replaced ordinary convolution kernel k is equal to 3, the input channel is equal to 256, and the output channel is equal to 512; after replacement, the group channel group channel is set to 32, then group num is 256 / 32=8, that is, 8 group channels are 32 3x3 group convolutions, and the total channel number is still 512; the total number of parameters used is the parameters of the group convolution plus the parameters of the pointwise convolution; therefore, after step S2 is completed, the ordinary convolution of the network except the input layer and the output layer is simplified to the group convolution and the pointwise convolution with a convolution kernel of 3*3 and a group channel of 32.

6. The improved method for lightweighting convolutional neural networks of claim 5, wherein, The group channel group channel can also be set to 16, so that 256 / 16=16 group channels are 16 3x3 group convolutions.

7. The improved method for lightweighting convolutional neural networks of claim 3, wherein, In step S1, assuming that the channel number of a layer cannot be divided by the set group channel number, the result is rounded up, then multiplied by the channel number, and the obtained result is set as the input channel number of the layer, that is, it can be divided by the group channel; assuming that the group channel number is set to 32, but the channel number of a layer is 80, at this time, the value of 80 / 32 is 2.5, which is rounded up to 3, then multiplied by 32 to get 96, and the input channel number of the layer is set to 96.

8. The improved method for lightweighting convolutional neural networks of claim 1, wherein, In the step S1, when the group channel size is equal to the input channel size of the original convolutional network, the model becomes a combination of normal convolution and point convolution, has the most parameters, requires the largest amount of calculation and memory space, and has the strongest model capability; when the value of the group channel is 1, the model becomes a depthwise separable convolution, has the least parameters, requires the least amount of calculation and memory space, and has the weakest model capability; therefore, a suitable group channel needs to be selected according to the platform calculation capability and memory space.

Images