A lightweight image convolution method

Abstract

The present invention discloses a lightweight image convolution method, which belongs to the field of image data processing. The lightweight image convolution method comprises the following steps: firstly, constructing a spatial processing unit and a channel processing unit, wherein the spatial input features input to the spatial processing unit are consistent in size with the spatial output features output by the spatial processing unit, and the channel input features input to the channel processing unit are consistent in size with the channel output features output by the channel processing unit; then, the spatial processing unit and the channel processing unit are combined to process the input features to obtain the output result of the image convolution. The present invention simultaneously introduces two processing methods, namely, spatial processing and channel processing, to eliminate redundant features from two aspects at the same time, thereby solving the problem of large amount of convolution calculation, and also ensuring the full extraction of information.

Description

Technical Field

[0001] The invention belongs to the field of image data processing, and in particular relates to an image convolution method. Background Art

[0002] The research and application of convolutional neural networks in the field of computer vision have proved their irreplaceable importance and have been widely used in many fields such as medicine, industry, and public security. The powerful representation ability of convolutional neural networks, especially their excellent performance in image feature extraction, makes them a key technology in the field of computer vision. However, with the expansion of model scale and the deepening of networks, the requirements for hardware computing power and storage space are also increasing. In particular, when it is necessary to deploy convolutional neural networks on small computers, it is often difficult to meet the needs of high computing due to factors such as cost. Therefore, the design of lightweight networks and modules becomes particularly important.

[0003] In order to solve this problem, technicians in this field have explored a variety of lightweight network technologies and proposed many independent models to reduce the computational complexity of the basic model. For example, the SE module in SENet is a channel attention mechanism that can be inserted into different networks to effectively improve network performance. However, the SE module cannot completely replace the traditional convolution structure. FasterNet is a faster and more efficient lightweight network. It believes that the feature maps of different channels in the same feature block have very similar information and contain a lot of redundancy. Therefore, the PConv convolution operation only uses part of the channels when extracting features, reducing redundant information and increasing the speed of the network. However, the PConv module is too simple and the information extraction is insufficient. The convolution triple attention module establishes dimensional dependencies between feature vectors through rotation operations of different dimensions. It can replace traditional convolution and be applied to a variety of networks, but it mainly focuses on spatial information and does not process channel information.

[0004] In summary, although existing research has made progress in dealing with spatial or channel redundancy, reducing the amount of computation and parameters and improving efficiency, it only focuses on the redundant features of a single aspect of the channel or space, resulting in redundancy in features on the other side, and has not solved the problem of large convolution computation. Summary of the invention

[0005] The present invention proposes a lightweight image convolution method, the purpose of which is to solve the problem of large amount of calculation in existing image convolution methods.

[0006] The technical solution of the present invention is as follows:

[0007] A lightweight image convolution method is provided. First, a spatial processing unit and a channel processing unit are constructed. The spatial input feature input to the spatial processing unit is consistent with the spatial output feature output by the spatial processing unit, and the channel input feature input to the channel processing unit is consistent with the channel output feature output by the channel processing unit.

[0008] Then the spatial processing unit and the channel processing unit are combined to process the input features to obtain the output result of the image convolution. The combination method is any of the following:

[0009] Method 1: Input the input features to be convolved into the spatial processing unit and the channel processing unit at the same time, and then add the spatial output features and the channel output features point by point to obtain the output result of the image convolution;

[0010] Method 2: Input the input features to be convolved as spatial input features into the spatial processing unit, then input the obtained spatial output features into the channel processing unit as channel input features, and then use the obtained channel output features as the output results of the image convolution;

[0011] Method 3: Input the input features to be convolved as channel input features into the channel processing unit, then input the obtained channel output features as spatial input features into the spatial processing unit, and then use the obtained spatial output features as the output result of image convolution.

[0012] As a further improvement of the lightweight image convolution method: in the spatial processing unit, the spatial input features are divided along the channel direction to obtain more than two groups of first sub-features, and then each first sub-feature is subjected to a DW convolution operation respectively, and then all the results of the DW convolution are reversely spliced ​​according to the original division method to obtain the spatial output features.

[0013] As a further improvement of the lightweight image convolution method: in the spatial processing unit, the convolution kernel sizes of each DW convolution operation are different.

[0014] As a further improvement of the lightweight image convolution method: in the spatial processing unit, the spatial input feature Along the channel direction by division coefficient Divided into two first sub-features: and , is the number of channels of the input feature, and are the height and width of the spatial input features, respectively;

[0015] Then, the two first sub-features and Input to convolutions with different receptive fields respectively and , and get the corresponding result and , The convolution kernel size is , the number of output channels is The convolution operation, The convolution kernel size is , the number of output channels is Convolution operation;

[0016] Finally and The spatial output features are obtained by splicing.

[0017] As a further improvement of the lightweight image convolution method: in the channel processing unit, the channel input feature is divided along the channel direction to obtain two second sub-features and group convolution and point convolution are performed respectively, and then the results of the group convolution and point convolution are added to obtain the intermediate feature, and then the intermediate feature is processed. The channel output features are obtained through pooling and attention mechanism.

[0018] As a further improvement of the lightweight image convolution method: in the channel processing unit, the channel input feature Segment along the channel direction to obtain two second sub-features and , is the division coefficient, is the number of channels of the channel input feature, and are the height and width of the channel input features respectively;

[0019] Then the second sub-feature Perform group convolution to obtain features , the second sub-feature Perform point convolution to obtain features ;

[0020] Then the features and Features Add to get the intermediate features .

[0021] As a further improvement of the lightweight image convolution method, the pooling process includes: The global average pooling operation obtains the features , for the intermediate features Perform global maximum pooling operation to obtain features , then the feature and Features Add to get the pooling feature .

[0022] As a further improvement of the lightweight image convolution method, the attention mechanism refers to: pooling features Input to the softmax layer, and then combine the output of the softmax layer with the intermediate features Multiply to get the channel output features.

[0023] As a further improvement of the lightweight image convolution method, the training process of the convolution module constructed based on the spatial processing unit and the channel processing unit is as follows:

[0024] Step T1, prepare data set;

[0025] Step T2, preprocessing the images in the data set;

[0026] Step T3: embed the constructed convolution module into the target network to replace the original convolution module in the target network;

[0027] Step T4: training based on the replaced target network.

[0028] As a further improvement of the lightweight image convolution method, the specific process of step T4 is as follows:

[0029] Step T4-1, extracting input data blocks of m images from the training set;

[0030] Step T4-2, inputting the input data block in step T4-1 into the target network;

[0031] Step T4-3, after the target network is forward propagated, the network loss is obtained, and according to the obtained loss, the parameters of the constructed convolution module are adjusted using the stochastic gradient descent optimization method in the backward propagation of the target network to reduce the network loss;

[0032] Repeat steps T4-1 to T4-3 until the training is completed.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention introduces two processing methods, spatial processing and channel processing, to eliminate redundant features from two aspects at the same time, thereby solving the problem of large convolution calculation amount. In addition, the two units use DW convolution, point convolution and group convolution through different architectures to extract features and fuse them, which reduces the amount of calculation while ensuring sufficient extraction of information.

[0035] 2. The spatial processing unit obtains feature blocks with different spatial information through the three-step processing of "separation-convolution-fusion", fully integrates different spatial information, and enhances the information representation capability.

[0036] 3. The channel processing unit consists of three parts: "separation-convolution-pooling". The results of convolution (group convolution and point convolution) and channel soft attention (maximum pooling and average pooling operations) are fused between channels to provide the network with stronger channel processing capabilities.

[0037] 4. The convolution module based on the present invention can easily replace the existing convolution module in the built network, which is plug-and-play, effectively reducing the number of parameters to achieve the goal of lightweight model and save hardware resources.

[0038] 5. The combination of the spatial processing unit and the channel processing unit is very flexible. They can be processed in parallel or connected front to back, and are highly versatile. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the overall flow chart of the present invention;

[0040] Figure 2 This is an architectural diagram of a spatial processing unit;

[0041] Figure 3 is an architectural diagram of a channel processing unit;

[0042] Figure 4 This is a comparison chart between the present invention and other convolution methods. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0044] A lightweight image convolution method first constructs a spatial processing unit and a channel processing unit. The spatial input features input to the spatial processing unit are consistent with the spatial output features output by the spatial processing unit, and the channel input features input to the channel processing unit are consistent with the channel output features output by the channel processing unit.

[0045] Then the spatial processing unit and the channel processing unit are combined to process the input features to obtain the output result of the image convolution. The combination method is any of the following:

[0046] Method 1: Figure 1 , the input features to be convolved are simultaneously input into the spatial processing unit and the channel processing unit, and then the spatial output features and the channel output features are added point by point to obtain the output result of the image convolution.

[0047] Method 2: Input the input features to be convolved as spatial input features into the spatial processing unit, then input the obtained spatial output features into the channel processing unit as channel input features, and then use the obtained channel output features as the output results of the image convolution.

[0048] Method 3: Input the input features to be convolved as channel input features into the channel processing unit, then input the obtained channel output features as spatial input features into the spatial processing unit, and then use the obtained spatial output features as the output result of image convolution.

[0049] like Figure 2 In the spatial processing unit, the spatial input feature is segmented along the channel direction to obtain more than two groups of first sub-features, and then each first sub-feature is subjected to a DW convolution operation, and then all the results of the DW convolution are reversely spliced ​​according to the original segmentation method to obtain the spatial output feature. The convolution kernel size of each DW convolution operation is different. The number of output channels of the DW convolution operation is the same as the number of channels of the corresponding first sub-feature.

[0050] Specifically, in this embodiment, the spatial input feature Along the channel direction by division coefficient Divided into two first sub-features: and , is the number of channels of the input feature, and are the height and width of the spatial input feature respectively. Generally speaking, the first sub-feature obtained by segmentation will not have much difference in spatial information. Before training, the segmentation coefficient needs to be debugged, and the optimal segmentation coefficient is selected before training.

[0051] Then, the two first sub-features and Input to convolutions with different receptive fields respectively and , and get the corresponding result and , The convolution kernel size is , the number of output channels is The convolution operation, Same reason. and Contains spatial information of different receptive fields, enhancing the representation ability of spatial information.

[0052] Finally and The spatial output features are obtained by splicing.

[0053] Compared with ordinary convolution, DW convolution has the advantage of being lightweight and can greatly reduce the number of model parameters. In addition, when the channel dimension is relatively large, the spatial features of the feature map are similar, that is, the two first sub-feature maps will contain some of the same spatial information. Therefore, the separation operation is used to split the channel dimension into two parts for DW convolution respectively. While reducing the amount of calculation, it will hardly reduce the network's ability to process information, and can ensure that sufficient information is extracted.

[0054] like Figure 3 In the channel processing unit, the channel input feature is divided along the channel direction to obtain two second sub-features and group convolution and point convolution are performed respectively, and then the results of group convolution and point convolution are added to obtain the intermediate feature, and then the intermediate feature is processed. The channel output features are obtained through pooling and attention mechanism.

[0055] Specifically, the channel is first input into the feature Segment along the channel direction to obtain two second sub-features and , is the division coefficient, is the number of channels of the channel input feature, and Input the height and width of the feature for each channel. Then, the second sub-feature Perform group convolution to obtain features , the second sub-feature Perform point convolution to obtain features In this embodiment, the convolution kernel size of the group convolution is , point convolution refers to PW depth convolution with a convolution kernel size of 1×1.

[0056] Then the features and Features Add to get the intermediate features The two branches of this method perform convolution operations on different aspects of the feature map, thereby extracting more detailed and diverse channel information. Fusion of the two convolution results can not only enrich the representation of the feature map and improve the model's efficiency in utilizing channel information, but also reduce the overall number of parameters due to the sharing and optimization of parameters between the two branches.

[0057] Furthermore, the pooling process includes: The global average pooling operation obtains the features , for the intermediate features Perform global maximum pooling operation to obtain features , then the feature and Features Add to get the pooling feature . A single pooling operation cannot fully extract channel information, so the present invention introduces a maximum pooling operation that is performed in parallel with average pooling. The role of maximum pooling is to enhance the channel information extracted by average pooling. In addition, for those key information that average pooling fails to capture and that can distinguish feature maps from other parts, maximum pooling can effectively extract and process them, thereby further improving the representation of features.

[0058] Furthermore, the attention mechanism refers to: pooling features Input to the softmax layer, and then combine the output of the softmax layer with the intermediate features Multiply to get the channel output features.

[0059] like Figure 4 The convolution module SCTConv obtained based on the above method 1 (the structure of parallel spatial processing module and channel processing module) is compared with other classic lightweight models GhostModule, PConv, SKConv, SCConv models and ordinary convolution Conv on the ResNet50 network. The experimental structure is the same as Table 1. Figure 4 After replacing the same number of ordinary convolutions, the accuracy of the convolution module designed by this method is the highest among all lightweight models, and compared with ordinary convolution Conv, the number of parameters of the convolution module designed by this method is also reduced to a considerable extent.

[0060] The convolution module designed based on this method can replace the convolution layer of part of the network that has been built. The output result of the convolution is consistent with the input feature size and can be plug-and-play. After the replacement, the parameter amount of the network will decrease, so as to achieve the purpose of lightweight in the neural network. The convolution module of the present invention can be applied to all known various network models and tasks. For example, in the field of image classification, the convolution in the Resnet network is replaced by this module, and its network parameters will decrease by about 30%, and the accuracy rate is almost unchanged. In some tasks, the model accuracy rate will also be improved to a certain extent. In addition to image classification, this module can also be applied to other fields such as target detection, such as the model of the yolo series that is now widely used. This module can be inserted into any part of the network model at will or replace a part of the convolution layer, without the need to redesign the network structure, and has a wide range of application scenarios.

[0061] The convolution module training and testing process based on the method of the present invention is as follows:

[0062] Step T1: Prepare the data set.

[0063] This embodiment uses two classic and open source datasets: the CIFAR-100 dataset is a classic dataset, which contains 100 categories, each category has 600 images, and the entire dataset has a total of 60,000 images, of which 50,000 are used as training sets and 10,000 are used as test sets. The pixel size of all images in the CIFAR-100 dataset is 32×32; TinyImagenet has 200 classes, each class has 500 training images, 50 verification images, and 50 test images.

[0064] The dataset is divided into training and testing sets.

[0065] Step T2: preprocess the images in the data set.

[0066] In order to fully train the network, the images in the dataset need to be preprocessed. First, the given image is randomly cropped into different sizes and aspect ratios, then the cropped image is scaled to the size of the original image, and then the image in the dataset is randomly rotated horizontally with a probability of 0.5, and finally the image is normalized.

[0067] Step T3: embed the designed convolution module into the target network to replace the original convolution module in the target network.

[0068] Taking Resnet34 and Resnet50 networks as examples, the convolutional module of the present invention is used to replace the 2nd to 5th convolutional layers in Resnet, and the obtained networks are named Resnet34_SCT and Resnet50_SCT. The replaced structures are shown in Table 1:

[0069] Table 1 - Network structure of Resnet34_SCT and Resnet50_SCT

[0070]

[0071] In Table 1, the position occupied by SCT is a 3×3 ordinary convolution before replacement.

[0072] Step T4: training based on the replaced target network.

[0073] In this example, SGD is used as the model optimizer, and the initial learning rate is set to 0.05. After the 50th, 100th, and 150th rounds of training, the learning rate is reduced by 0.1 and the momentum is 0.9. The specific process is:

[0074] Step T4-1: extract input data blocks of m images from the training set.

[0075] Step T4-2: input the input data block in step T4-1 into the target network.

[0076] Step T4-3: After the target network is forward propagated, the network loss is obtained. According to the obtained loss, the parameters of the constructed convolution module are adjusted using the stochastic gradient descent optimization method in the backward propagation of the target network to reduce the network loss.

[0077] Repeat steps T4-1 to T4-3 until the training is completed. In order to fully learn, the data set is learned for 200 rounds.

[0078] Step T4-5: ​​Model testing.

[0079] In this embodiment, the CIFAR-100 test set is used to verify the performance of the module. The test set images are divided according to step T1, and then the images are input into the target network replaced according to step T3, and the image is identified through calculation to which category of the 100 categories of images the image belongs.

[0080] All test set images are input into the network in batches, and the number of correctly classified images is calculated to obtain the accuracy of the network. The accuracy formula is as follows:

[0081] .

[0082] The following is a comparison of the parameters of the convolution module proposed in the present invention and the common convolution:

[0083] 1. The number of parameters for ordinary convolution is:

[0084] ;

[0085] represents the receptive field of the convolution, and are the number of convolution input channels and output channels respectively.

[0086] 2. The parameter quantity of the convolution module proposed in the present invention includes two parts:

[0087] Parameter quantity of spatial processing unit:

[0088] ;

[0089] The spatial processing unit consists of two DW convolutions with different receptive fields. is the division coefficient of the spatial processing unit (here it is assumed that it is divided into only two first sub-features), is 3, i.e. the convolution kernel size of 3×3DW convolution, is 5, i.e. the convolution kernel size of 5×5DW convolution, is the number of input channels for the 3×3DW convolution, is the number of input channels for the 5×5 DW convolution.

[0090] Parameter quantity of channel processing unit:

[0091] ;

[0092] Although the channel processing unit looks more complicated than the spatial processing unit, its parameters are concentrated in the first half: group convolution and point convolution. is the division coefficient of the channel processing unit, is 3, that is, the convolution kernel size of the 3×3 group convolution, is the total number of convolution input channels, is the number of input channels for each group in the group convolution, is the number of output channels in the point convolution, is the number of groups in the group convolution.

[0093] The parameter quantity of the convolution module proposed in this invention is ,Right now:

[0094] Taking the replacement of ordinary 3×3 convolution as an example: after calculation, the number of parameters of ordinary convolution is 36864, and the number of parameters of the convolution module of the present invention is 9280, which reduces the number of parameters by 27584, and the effect is very significant.

[0095] The replaced Resnet34_SCT and Resnet50_SCT are tested and compared with the original Resnet34 and Resnet50. The comparison results based on the Tiny ImageNet dataset are shown in Table 2, and the comparison results based on the CIFAR-100 dataset are shown in Table 3.

[0096] Table 2 - Comparison results based on the Tiny ImageNet dataset

[0097]

[0098] Table 3 - Comparison results based on the CIFAR-100 dataset

[0099]

[0100] It can be seen from Tables 2 and 3 that this method can effectively reduce the amount of floating-point operations and parameters while keeping the accuracy basically unchanged or even improving.

[0101] It should be noted that, for those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention.

Claims

1. A lightweight image convolution method, characterized in that: First, a spatial processing unit and a channel processing unit are constructed, the spatial input feature input to the spatial processing unit is consistent with the spatial output feature output by the spatial processing unit, and the channel input feature input to the channel processing unit is consistent with the channel output feature output by the channel processing unit; Then the spatial processing unit and the channel processing unit are combined to process the input features to obtain the output result of the image convolution. The combination method is any of the following: Method 1: Input the input features to be convolved into the spatial processing unit and the channel processing unit at the same time, and then add the spatial output features and the channel output features point by point to obtain the output result of the image convolution; Method 2: Input the input features to be convolved as spatial input features to the spatial processing unit, then input the obtained spatial output features as channel input features to the channel processing unit, and then use the obtained channel output features as the output results of image convolution; Method 3: Input the input features to be convolved as channel input features to the channel processing unit, then input the obtained channel output features as spatial input features to the spatial processing unit, and then use the obtained spatial output features as the output result of image convolution; In the spatial processing unit, the spatial input feature is segmented along the channel direction to obtain more than two groups of first sub-features, and then each first sub-feature is subjected to a DW convolution operation respectively, and then all the results of the DW convolution are reversely spliced ​​according to the original segmentation method to obtain the spatial output feature; In the spatial processing unit, the convolution kernel sizes of the DW convolution operations are different; In the spatial processing unit, the spatial input feature Along the channel direction by division coefficient Divided into two first sub-features: and , is the number of channels of the input feature, and are the height and width of the spatial input features, respectively; Then, the two first sub-features and Input to convolutions with different receptive fields respectively and , and get the corresponding result and , The convolution kernel size is , the number of output channels is The convolution operation, The convolution kernel size is , the number of output channels is Convolution operation; Finally and Splicing to obtain the spatial output features; In the channel processing unit, the channel input feature is divided along the channel direction to obtain two second sub-features and group convolution and point convolution are performed respectively, and then the results of group convolution and point convolution are added to obtain the intermediate feature, and then the intermediate feature is processed. The channel output features are obtained through pooling and attention mechanism; In the channel processing unit, the channel input feature Segment along the channel direction to obtain two second sub-features and , is the division coefficient, is the number of channels of the channel input feature, and are the height and width of the channel input features respectively; Then the second sub-feature Perform group convolution to obtain features , the second sub-feature Perform point convolution to obtain features ; Then the features and Features Add to get the intermediate features ; The pooling process includes: The global average pooling operation obtains the features , for the intermediate features Perform global maximum pooling operation to obtain features , then the feature and Features Add to get the pooling feature ; The attention mechanism refers to: pooling features Input to the softmax layer, and then combine the output of the softmax layer with the intermediate features Multiply to get the channel output features.

2. The lightweight image convolution method according to claim 1, characterized in that: The training process of the convolution module based on the spatial processing unit and the channel processing unit is as follows: Step T1, prepare data set; Step T2, preprocessing the images in the data set; Step T3: embed the constructed convolution module into the target network to replace the original convolution module in the target network; Step T4: training based on the replaced target network.

3. The lightweight image convolution method according to claim 2, characterized in that: The specific process of step T4 is as follows: Step T4-1, extracting input data blocks of m images from the training set; Step T4-2, inputting the input data block in step T4-1 into the target network; Step T4-3, after the target network is forward propagated, the network loss is obtained, and according to the obtained loss, the parameters of the constructed convolution module are adjusted using the stochastic gradient descent optimization method in the backward propagation of the target network to reduce the network loss; Repeat steps T4-1 to T4-3 until the training is completed.

Images