A deformable convolution method, device and storage medium

Abstract

The application discloses a deformable convolution method, a device and a storage medium, wherein the deformable convolution method comprises the following steps: step 1, performing regular convolution on an input feature map to obtain offset vectors of multiple groups of sampling point coordinates; step 2, extracting features of offset sampling points from the input feature map respectively according to the coordinates of each group of sampling points, and performing weighted summation using a convolution kernel to obtain output features of each group; and step 3, averaging the convolution output features of the groups to obtain final output features. Through the multiple groups of sampling points, the expression ability of the deformable convolution is enhanced, meanwhile, the position consistency of each channel of the convolution input features is ensured, and experiments on a target detection data set COCO prove that the application effectively improves the performance of target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to a deformable convolution method of multiple groups of sampling points, and belongs to the technical field of computer vision. BACKGROUND

[0002] With the development of deep learning technology, target detection is widely used in many fields. For example, detecting defects of industrial products in industrial quality inspection; detecting pedestrians and vehicles in front of vehicles in the field of transportation; detecting abnormal events in public places such as elevators in the field of security, and the like.

[0003] Deformable convolution is a commonly used technology in target detection. Compared with ordinary convolution, deformable convolution can adaptively adjust the sampling points of convolution according to the input features, so as to adaptively realize the changes of shape and scale. However, the number of sampling points of most existing deformable convolution methods is fixed, which limits the expression ability of deformable convolution. The existing deformable convolution method using multiple groups of sampling points divides the channels of the convolution input features into multiple groups, and uses different sampling points in each group, which destroys the position consistency of different channels of the convolution input features. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a deformable convolution method, device and storage medium for enhancing the expression ability of the model and improving the accuracy of target detection.

[0005] In order to solve the above problems, the present application adopts the following technical scheme:

[0006] A deformable convolution method, characterized in that it comprises:

[0007] Step 1, performing regular convolution on the input feature map to obtain offset vectors of multiple groups of sampling point coordinates;

[0008] Step 2, extracting the features of the offset sampling points from the input feature map respectively for the coordinates of each group of sampling points, and using convolution kernel weighted summation to obtain the output features of each group;

[0009] Step 3, averaging the convolution output features of each group to obtain the final output features.

[0010] Step 1, performing regular convolution on the input feature map to obtain offset vectors of multiple groups of sampling point coordinates, comprising:

[0011] Step 1-1, for a position p0 on the output feature map y, using a 3x3 grid R to sample features on the input feature map x, wherein R={(-1,-1), (-1,0),..., (0,1), (1,1)};

[0012] Step 1-2, using convolution kernel weight wo The weighted sum is obtained as the offset value vector of the coordinates of each group of sampling points:

[0013]

[0014] In the formula, is the offset value vector of the position ; is the weight of the convolution kernel at the weight vector at the position ; is the feature vector of the input feature map x at the position

[0015] Step 1-3, divide the offset vector o into g groups to obtain the offset vectors o of each group 1 , o 2 ,..., o g ; wherein the dimension of each offset vector o i is 18, respectively corresponding to the horizontal and vertical offset values of each sampling point in the grid R.

[0016] Step 2 extracts the features of the offset sampling points from the input feature map for each group of sampling point coordinates, and uses the convolution kernel to obtain the output features of each group by weighted sum, including:

[0017] Step 2-1, for each group of offset vectors o i , offset the coordinates of the sampling points in the grid R;

[0018] Step 2-2, extract the convolution input features on the input feature map x using the bilinear interpolation method with the offset sampling points;

[0019] Step 2-3, use the convolution kernel w to weighted sum the sampling features to obtain the feature of the i-th group of output feature maps y i at the position p0 :

[0020]

[0021] wherein w(p n ) is the weight vector of the convolution kernel weight w at p n ; q enumerates each position on the input feature map x; x(q) is the feature vector of the input feature map x at the position q; q x and q y are the horizontal and vertical coordinates of the position q, respectively, (p0+ p n +o n i ) x and (p0+ p n +on i ) y Don't use position p0+ p n +o n i The horizontal and vertical coordinates.

[0022] As a preferred technical solution of the present invention, unlike the existing deformable convolution method that samples each channel in groups, step 2-2 uses only one set of sampling points, that is, each channel uses the same sampling points.

[0023] In steps 2-3, the same convolution kernel w is used when weighting and summing the sampling features corresponding to each group of offset vectors, and when convolving each group of sampling points separately.

[0024] Step 3, convert the convolutional output features y of each group into the desired features. i The final output features are obtained by averaging:

[0025]

[0026] In the formula, This represents the final output feature at position P0.

[0027] Compared with the prior art, the present invention has the following technical effects:

[0028] 1. This invention can effectively enhance the shape and scale variation of deformable convolution by using multiple sets of sampling points in deformable convolution, thereby enhancing the expressive power of the model.

[0029] 2. By using a shared convolution kernel when summing the features of each group of sampling points, the number of parameters remains constant, thus mitigating the risk of model overfitting.

[0030] 3. By using a set of sampling points for the convolutional input features and averaging the convolution results of multiple sets of sampling points, the positional consistency of each channel of the convolutional input features is ensured.

[0031] 4. By applying the deformable convolution of multiple sampling points of the present invention to the target detection network, the target detection accuracy on the COCO dataset is effectively improved. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of deformable convolution with multiple sampling points used in this invention. Detailed Implementation

[0033] The embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.

[0034] Example 1

[0035] like Figure 1 As shown, the deformable convolution method of the present invention includes the following steps:

[0036] Step 1: Perform regular convolution on the input feature map to obtain offset vectors for multiple sets of sampling point coordinates;

[0037] Step 1: Sample features from the input feature map, and sum the sampled features using convolutional kernel weights, including:

[0038] Step 1-1: For position p0 on the output feature map y, sample features on the input feature map x using a 3x3 grid R, where R={(-1,-1), (-1,0), ..., (0,1), (1,1)}.

[0039] Steps 1-2: Apply convolutional kernel weights w to the sampled features. o The offset vector o of the coordinates of each group of sampling points is obtained by weighted summation using the following formula:

[0040]

[0041] Steps 1-3: Divide the offset vector o into g groups to obtain the offset vector o of each group. 1 , o 2 , ..., o g Each set of offset vectors o i The dimension is 18, which corresponds to the horizontal and vertical offset values ​​of each sampling point in grid R.

[0042] In one embodiment, g is set to 2, meaning that two sets of offset vectors are used to obtain two sets of deformable convolution sampling points.

[0043] Step 2: Extract the features of the offset sampling points from the input feature map for the coordinates of each group of sampling points, and use a convolution kernel to weighted sum to obtain the output features of each group;

[0044] Step 2 includes:

[0045] Step 2-1, for each set of offset vectors o i The coordinates of the sampling points in grid R are offset to obtain new sampling point coordinates p0+p n +o n i , where p n Iterate through each sampling point of R, o n i For offset vector o i The nth value;

[0046] Step 2-2, using the offset sampling points to extract the feature x(p0+ p n +o n i ).

[0047] Step 2-3, using the convolution kernel w to weight and sum the sampled features to obtain the i-th group of output feature maps y i Feature at position p0:

[0048]

[0049] wherein,

[0050]

[0051]

[0052] g(q, p) = max(0, 1-|q-p|).

[0053] Step 3, average the convolution output features of each group to obtain the final output feature:

[0054]

[0055] wherein, y i is the convolution output feature of each group;

[0056] In an embodiment, the deformable convolution method using multiple groups of sampling points described above is applied to the target detection network Faster-RCNN. Specifically, all 3x3 convolutions in the 3rd to 5th stages of the backbone network of Faster-RCNN are replaced with deformable convolution using two groups of sampling points. The coco dataset, which is a scene with complex and authoritative data, is tested for classification, and compared with the method of using ordinary deformable convolution (DCN) and the deformable convolution using group sampling of each channel of the convolution input feature. The results are shown in Table 1:

[0057]

[0058] wherein, mAP is the overall class average accuracy, AP50 and AP75 represent the average accuracy at the intersection over union threshold of 50% and 75%, respectively. From Table 1, it can be seen that the application is applied to the target detection method for testing of the coco dataset. From the test results, compared with the deformable convolution and the deformable convolution method using group sampling of each channel, several indicators are improved, especially AP50, which is significantly improved.

[0059] Embodiment 2

[0060] The embodiment of the present application also provides an apparatus, comprising a processor and a memory; wherein the memory has stored programs or instructions, the programs or instructions are loaded and executed by the processor to realize the variable convolution method of the embodiment 1.

[0061] Embodiment 3

[0062] The present application also provides a computer readable storage medium, which can be a non-volatile computer readable storage medium, and can also be a volatile computer readable storage medium, the computer readable storage medium has stored instructions, when the instructions run on the computer, make the computer execute the variable convolution method of the embodiment 1.

[0063] Those skilled in the art can clearly understand that the technical solutions of the present application, or the essence of the technical solutions or the contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product, the computer software product is stored in a storage medium, includes a plurality of instructions to make a computer device (may be a personal computer, server, or network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and various program code storage media.

Claims

1. A deformable convolution method for target detection, characterized in that, Includes the following steps: Step 1: Perform regular convolution on the input feature map to obtain offset vectors for multiple sets of sampling point coordinates; Step 2: Based on the offset vectors of multiple sets of sampling point coordinates obtained in Step 1, extract the features of the offset sampling points from the input feature map respectively, and use convolution kernels to weighted sum to obtain the output features of each set; Step 3: Average the convolutional output features of each group to obtain the final output features; Step 2 includes: Step 2-1, for each set of offset vectors o i Offset the coordinates of the sampling points in grid R; Step 2-2: Extract features from the input feature map x using bilinear interpolation with the offset sampling points; Steps 2-3 involve weighted summation of the sampled features using a convolution kernel w to obtain the i-th output feature map y. i The feature y at position p0 i (p0): Among them, w(p) n ) represents the convolution kernel weights w in p n The weight vector at point; x(p0+p n +o n i ) represents the input feature map x at position p0+p n +o n i The eigenvector at position o n i This represents the position of the nth sampling point in the i-th group; Step 1-1: For position p0 on the output feature map y, sample features on the input feature map x using a 3x3 grid R, where R = {(-1,-1),(-1,0),...,(0,1),(1,1)}; Steps 1-2: Apply convolutional kernel weights w to the sampled features. o Weighted summation yields the offset vector of each group of sampling point coordinates: In the formula, o(p0) is the offset vector of position p0; w o (p n ) represents the convolution kernel weights w o In p n The weight vector at point x(p0+p) n ) represents the input feature map x at position p0+p n The eigenvector at that location; Steps 1-3: Divide the offset vector o into g groups to obtain the offset vector o of each group. 1 ,o 2 ,...,o g .

2. The method according to claim 1, characterized in that, Input feature map x at position p0+p n +o n i eigenvectors at location for: Where q is a sampling point on the feature map, q x and q y The x and y coordinates of q are respectively, (p0+p n +o n i ) x and (p0+p n +o n i ) y They are p0+p respectively n +o n i The x and y coordinates are given by x(q), where x(q) is the feature vector of the input feature map x at position q.

3. The method according to claim 1, characterized in that, Step 3 is: The convolution output features y of each group i The final output features are obtained by averaging using the following formula: In the formula, y(p0) is the final output feature at position P0.

4. A deformable convolution device, characterized in that, It includes a processor and a memory; the memory stores a program or instructions which are loaded and executed by the processor to implement the deformable convolution method as described in any one of claims 1 to 3.

5. A computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the deformable convolution method as described in any one of claims 1 to 3.

Images