An adaptive multi-shape convolution method for object detection

Abstract

The present invention discloses an adaptive multi-shape convolution method for target detection, the purpose of which is to solve the shortcomings of traditional convolution kernels that cannot change shape, cannot match target geometric deformation, have a single receptive field in the same network layer, and extract invalid information features other than the target. The method of the present invention mainly learns the kernel mask first, then uses the kernel mask to perform adaptive multi-shape convolution, and finally uses gradient back propagation to update the kernel mask. The adaptive multi-shape convolution method of the present invention enables convolution to adapt to changes in the geometric shape of the target, enhances the semantic extraction ability and feature expression ability of the convolution network, and only adds a small amount of parameters and computational complexity compared to standard convolution. It can be added to the backbone network of any target detection model, and only needs to replace the standard convolutions such as 3x3 and 5x5 in its network, which can greatly improve the average detection accuracy of network model target detection.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to an adaptive multi-shape convolution method for target detection. Background Art

[0002] Convolutional neural networks are an important method in the field of computer vision and a hot research area in artificial intelligence. They play a very important role in processing various feature extraction tasks. Convolutional neural networks have greatly accelerated the development of computer vision tasks, including image classification, object detection, and object segmentation. The deep stacking of convolutional layers can simulate complex functions, the convolution operation can extract various semantic features of the image, and the gradient back propagation enables the network to learn and update a large number of parameters.

[0003] The different convolution kernels and convolution methods in convolutional neural networks give it powerful feature extraction capabilities. Standard convolution has different convolution kernel sizes, such as 3x3 and 1x1 convolution kernels. Grouped convolution divides the input into g groups, and the corresponding parameter amount can be reduced by 1 / g, which can greatly reduce the parameters of the network. Dilated convolution increases the distance between every two adjacent weights in the kernel, making the weight distribution discrete and expanding the receptive field of the original convolution kernel. Deconvolution is an upsampling operation used to expand the feature map.

[0004] However, the above convolution method can only cover the target body with a rectangular area, but most targets have irregular geometric shapes. In this way, in the process of convolution feature extraction, the convolution kernel not only extracts the features of the object, but also extracts the invalid background, which is obviously very inefficient. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the current convolution kernel cannot be changed into multiple shapes, resulting in inefficient convolution feature extraction. The present invention provides an adaptive multi-shape convolution method for target detection, which enables the convolution kernel to have the ability to change shape, thereby more efficiently extracting the main features of the target and eliminating the adverse effects of irrelevant information such as background on feature extraction.

[0006] In order to solve the above problems, the present invention is implemented by a technical solution of an adaptive multi-shape convolution method for target detection, which includes the following steps:

[0007] Step 1: The kernel mask is obtained by learning the standard convolution operation and argmax operation. The kernel mask calculation formula is as follows:

[0008] m(x)=argmax(conv(x))

[0009] Among them, m(x) represents the kernel mask, x is the input feature map, conv is the standard convolution operation, which consists of two 2D convolutions and a ReLU function; the first 2D convolution is used for dimensionality reduction, the convolution kernel size is 3x3, the number of input channels is C, the number of output channels is M, the stride is 1, the padding is 1, and the bias is True; followed by the ReLU function, which is used to increase the nonlinear expression ability of the module; the second 2D convolution increases the channel dimension and learns the final kernel mask, the convolution kernel size is 1x1, the number of input channels is M, the number of output channels is K, the stride is 2, the padding is 0, and the bias is True; among them, M is the height and width of the adaptive multi-shape convolution kernel, and K is the height and width product; finally, the shape of the learned kernel mask is K*H / 2*W / 2, where H and W are the width and height of the input feature map;

[0010] The argmax operation consists of a sigmoid function and a thresholding process. The resulting kernel mask has a value of 0 or 1 at each point, which can be expressed as:

[0011]

[0012] Among them, t is a hyperparameter, the default value is 0.25, s is the result of conv(x), is a sigmoid function; the argmax operation is implemented through the sigmoid function and thresholding techniques, in order to prevent the loss of gradients caused by direct argmax operation and the inability to perform back propagation updates of the kernel mask parameters;

[0013] Step 2: Use the learned kernel mask to perform adaptive multi-shape convolution, which is expressed as follows:

[0014]

[0015] Where, · represents the dot multiplication operation; w is the standard convolution kernel, m is the kernel mask, x is the input feature map, y is the output feature map; K is the number of convolution sampling points, K = 9 in 3x3 adaptive convolution; po is the pixel coordinate on the feature map, and pk is the coordinate point on the convolution kernel;

[0016] If the input feature map size is W*H, the stride of the adaptive multi-shape convolution kernel and the standard convolution kernel is 2, the number of convolution sampling points of the adaptive multi-shape convolution in the input feature map is the same as the number H / 2*W / 2 multiplied by the height and width of the kernel mask. Before the adaptive multi-shape convolution performs convolution at different positions in the feature map, the value of the corresponding coordinate point of the kernel mask on the K channels is multiplied by the K weights of the standard convolution kernel to obtain a new convolution kernel with a shape change, and then it is multiplied with the K pixel values ​​around the sampling point and accumulated to obtain the adaptive multi-shape convolution result of the point po; it can be seen that in the adaptive multi-shape convolution with a stride of 2, a feature map of size W*H corresponds to W / 2*H / 2 convolution kernels of different shapes; while in the standard convolution, a feature map only uses one shape convolution kernel;

[0017] Step 3: Back propagate and update the kernel mask. After using the kernel mask for forward propagation of adaptive multi-shape convolution, it is necessary to update the parameters according to the gradient back propagation to learn a kernel mask that is more suitable for the convolution sampling point. The gradient formula of the adaptive multi-shape convolution result with respect to the kernel mask is as follows:

[0018]

[0019] Among them, w is the standard convolution kernel, m is the kernel mask, x is the input feature map, y is the output feature map, K is the number of convolution sampling points, po is the pixel coordinate on the feature map, and pk is the coordinate point on the convolution kernel;

[0020] The kernel mask is learned by standard convolution and argmax operations, and its gradient formula with respect to the input feature map x is as follows:

[0021]

[0022] Among them, x is the input feature map, m(x) is the kernel mask, and conv(x) is the result of the standard convolution operation;

[0023] According to the above gradient formula, the adaptive multi-shape convolution is back-propagated and updated, and finally the kernel mask of the most suitable convolution sampling point can be learned;

[0024] Step 4: input the training data set into the target detection network model based on the adaptive multi-shape convolution obtained in steps 1, 2, and 3 for training; the target detection network model includes the Faster RCNN network model;

[0025] Step 5: Input the test data set data into the trained target detection network model to obtain the target detection result.

[0026] The beneficial effects of the present invention are as follows: the method designed by the present invention uses a learnable kernel mask to realize an adaptive multi-shape convolution kernel, which solves the shortcoming that the shape of the standard convolution kernel cannot adapt to the change of the target, realizes the variable receptive field of the convolution kernel, makes the convolution of the target area fit the target itself, and makes the convolution network have more powerful semantic extraction and feature expression capabilities. The use of convolutional networks to learn the kernel mask also makes the adaptive multi-shape convolution only increase the model operation amount almost negligible relative to the traditional standard convolution. Compared with the standard convolution kernel, it also has the excellent properties of weight sharing, sparse connection, translation and other variations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the overall flow chart of the adaptive multi-shape convolution of the present invention;

[0028] Figure 2 It is a diagram of the adaptive multi-shape convolution operation process of the present invention;

[0029] Figure 3 is a structural diagram of the adaptive multi-shape of the present invention in the ResNet 50 residual module;

[0030] Figure 4 It is a comparison diagram of the adaptive multi-shape convolution of the present invention and the standard convolution. DETAILED DESCRIPTION

[0031] The present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods. The specific steps are as follows:

[0032] Step 1: Prepare the object detection dataset. The training set is the union of PASCALVOC 2007trainval and 2012trainval, which contain 5011 images and 11540 images respectively, with a total of 12608 and 27450 objects annotated. The validation set is PASCALVOC 2007test, which contains 4952 images and 12032 objects. The dataset has 20 categories and the input image size is 512x512.

[0033] Step 2: Build a basic object detection model. The detection algorithm model uses Faster R-CNN, FPN for neck, and ResNet-50 for backbone. Faster R-CNN is one of the best two-stage object detection algorithms. Using the RPN network to generate candidate boxes can truly realize an end-to-end object detection framework.

[0034] Step 3: Replace all 3x3 standard convolutions in the residual block of the ResNet50 backbone network of the Faster R-CNN algorithm model with 3x3 adaptive multi-shape convolutions.

[0035] The comparison of the ResNet50 residual block before and after replacement is shown in the figure Figure 3 shown. Figure 3 The left picture shows the structure of the ResNet50 residual module, and the right picture shows the structure of the adaptive multi-shape convolution applied in the ResNet residual module. The kernel mask of the adaptive multi-shape convolution is learned from the result of 1x1 convolution, and the standard 3x3 convolution in the original ResNet50 residual block is replaced by the adaptive multi-shape convolution.

[0036] The overall algorithm flow of adaptive multi-shape convolution is as follows Figure 1 As shown in the figure, the learnable kernel mask of each convolution position on the feature map is obtained through standard convolution operation and argmax operation. When convolution is performed on each convolution sampling point of the feature map, the kernel mask value at the corresponding position is multiplied by the original convolution kernel weight of the feature map to obtain the adaptive multi-shape convolution kernel, and then convolved with the feature map to obtain the convolution result of the adaptive multi-shape convolution kernel. In the figure, the adaptive multi-shape convolution kernel is 3x3, the stride is 2, H and W are the width and height of the feature map, C is the input channel value, M and K are the channel values ​​of the kernel mask after two convolutions, M=6, K=9. It can be seen from the figure that when convolution is performed on the right body part of the sheep, the adaptive multi-shape convolution accurately convolves the right body part of the sheep, without taking the grass into calculation, and the convolution feature extraction is more efficient.

[0037] The specific process of using kernel mask to perform adaptive multi-shape convolution operation is as follows: Figure 2 As shown in the figure, the white block of the input image is the background, and the dark block is the target object. The stride of the convolution is 3 in the x direction and 1 in the y direction. The adaptive multi-shape convolution uses different kernel masks at different positions in the feature map, and multiplies them with the standard convolution kernel to obtain the adaptive multi-shape convolution kernel, which can accurately extract the features of the target object.

[0038] Step 4: Send the training set data to the network to train the model network. The model optimizer uses SGD, the learning rate is set to 0.02, the momentum is set to 0.9, the weight_decay is set to 0.0001, the single GPU batchsize is set to 4, and a total of 12 epochs are trained. In the first 500 iterations of the first epoch, the warm up training strategy is used, and the warm_upration is set to 0.001. The training environment is python3.7.3, PyTorch 1.7.1, the GPU uses 4 GeForce RTX2080Ti, and the cuda version is 10.1.

[0039] Step 5: After training, use the test set to test the model, and the evaluation indicators are mAP and accuracy in each category.

[0040] Table 1. Comparison of detection accuracy between the present invention and Faster RCNN on the VOC2007 test set

[0041]

[0042] Table 1 shows the comparison results of the detection accuracy of Faster R-CNN with adaptive multi-shape convolution and Faster R-CNN method on the PASCAL VOC 2007 test set. The first row shows all the categories contained in the PASCAL VOC 2007 test set pictures. The second row shows the detection accuracy of the Faster R-CNN method, and the third row shows the detection accuracy of the present invention. The accuracy and average precision mAP of 20 categories are compared, and the highest accuracy of the compared items is marked in bold font. It can be seen from Table 1 that the average detection precision mAP of the present invention is 1.7% higher than that of the Faster R-CNN method, and the detection accuracy of the "airplane" category is improved by 3.1%, the detection accuracy of the "bottle" category is improved by 4.6%, the detection accuracy of the "chair" category is improved by 5.5%, and the detection accuracy of the "green plant" category is improved by 4.3%. It can be seen that the adaptive multi-shape convolution has a significant effect on improving the target detection accuracy.

[0043] Figure 4 Explain why adaptive multi-shape convolution is better than standard convolution. Figure 4 The left picture shows the area of ​​convolution operation when using 4 standard 3x3 convolution kernels to extract convolution features of the dog's head in the picture; the right picture shows the area of ​​convolution operation when using 4 adaptive multi-shape 3x3 convolution kernels to extract convolution features of the dog's head in the picture. It can be seen that the traditional standard 3x3 convolution kernel can only cover the target to be convolved with a rectangular area, and the target often has an irregular geometric shape, which makes the result of the traditional standard convolution affected by invalid information such as the background. Adaptive multi-shape convolution can perceive the shape of the target according to the convolution kernel at different positions, adapt to the shape of the target to perform convolution feature extraction, and fully extract the target information.

[0044] Matters not covered by the present invention are known technologies.

[0045] The above specific implementations are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive multi-shape convolution method for target detection, characterized in that The following steps are involved: Step 1: The kernel mask is obtained by learning the standard convolution operation and argmax operation. The kernel mask calculation formula is as follows: m(x)=argmax(conv(x)) Among them, m(x) represents the kernel mask, x is the input feature map, conv is the standard convolution operation, which consists of two 2D convolutions and a ReLU function; the first 2D convolution is used for dimensionality reduction, the convolution kernel size is 3x3, the number of input channels is C, the number of output channels is M, the stride is 1, the padding is 1, and the bias is True; followed by the ReLU function, which is used to increase the nonlinear expression ability of the standard convolution operation; the second 2D convolution increases the channel dimension and learns the final kernel mask, the convolution kernel size is 1x1, the number of input channels is M, the number of output channels is K, the stride is 2, the padding is 0, and the bias is True; among them, M is the height and width of the adaptive multi-shape convolution kernel, and K is the height and width product; finally, the shape of the learned kernel mask is K*H / 2*W / 2, where H and W are the width and height of the input feature map; The argmax operation consists of a sigmoid function and a thresholding process. The resulting kernel mask has a value of 0 or 1 at each point, which can be expressed as: Among them, t is a hyperparameter, the default value is 0.25, s is the result of conv(x), is a sigmoid function; the argmax operation is implemented by using the sigmoid function and thresholding techniques to prevent the loss of gradients caused by direct argmax operation and the inability to perform backpropagation updates of the kernel mask parameters; Step 2: Use the learned kernel mask to perform adaptive multi-shape convolution, which is expressed as follows: Where, · represents the dot multiplication operation; w is the standard convolution kernel, m is the kernel mask, x is the input feature map, y is the output feature map; K is the number of convolution sampling points, K = 9 in 3x3 adaptive convolution; po is the pixel coordinate on the feature map, and pk is the coordinate point on the convolution kernel; If the input feature map size is W*H, the stride of the adaptive multi-shape convolution kernel and the standard convolution kernel is 2, the number of convolution sampling points of the adaptive multi-shape convolution in the input feature map is the same as the number H / 2*W / 2 multiplied by the height and width of the kernel mask. Before the adaptive multi-shape convolution performs convolution at different positions in the feature map, the value of the corresponding coordinate point of the kernel mask on the K channels is multiplied by the K weights of the standard convolution kernel to obtain a new convolution kernel with a shape change, and then it is multiplied with the K pixel values ​​around the sampling point and accumulated to obtain the adaptive multi-shape convolution result of the point po; it can be seen that in the adaptive multi-shape convolution with a stride of 2, a feature map of size W*H corresponds to W / 2*H / 2 convolution kernels of different shapes; while in the standard convolution, a feature map only uses one shape convolution kernel; Step 3: Back propagate and update the kernel mask. After using the kernel mask for forward propagation of adaptive multi-shape convolution, it is necessary to update the parameters according to the gradient back propagation to learn a kernel mask that is more suitable for the convolution sampling point. The gradient formula of the adaptive multi-shape convolution result with respect to the kernel mask is as follows: Among them, w is the standard convolution kernel, m is the kernel mask, x is the input feature map, y is the output feature map, K is the number of convolution sampling points, po is the pixel coordinate on the feature map, and pk is the coordinate point on the convolution kernel; The kernel mask is learned by standard convolution and argmax operations, and its gradient formula with respect to the input feature map x is as follows: Among them, x is the input feature map, m(x) is the kernel mask, and conv(x) is the result of the standard convolution operation; According to the above gradient formula, the adaptive multi-shape convolution is back-propagated and updated, and finally the kernel mask of the most suitable convolution sampling point can be learned; Step 4: input the training data set into the target detection network model based on the adaptive multi-shape convolution obtained in steps 1, 2, and 3 for training; the target detection network model includes the Faster RCNN network model; Step 5: Input the test data set data into the trained target detection network model to obtain the target detection result.

Images