A Deformable Convolutional Network Based on Deep Learning

Abstract

The present invention discloses a deformable convolution network based on deep learning, including a deformable convolution module and a deformable ROI pooling layer module; the deformable convolution module adds a two-dimensional offset to the position of the conventional grid sampling in the standard convolution, so that the sampling grid can be deformed freely; the offset is obtained from the mapping feature in front of it through an additional convolution layer; the deformable ROI pooling layer module adds an offset to each small partition position in the pooling layer, and the offset can be learned from the previous feature map and the region of interest, so as to achieve adaptive positioning for targets with different shapes. The two new modules of the present invention can replace the ordinary modules in the existing mainstream network, and can be trained end-to-end through standard back propagation, so as to generate a deformable convolution network, which greatly improves the detection accuracy when processing complex target situations.

Description

Technical Field

[0001] The present invention relates to a deformable convolutional network, and in particular to a deformable convolutional network based on deep learning. Background Art

[0002] At present, a key challenge in computer vision tasks is how the network architecture adapts to the geometric changes caused by the target size, posture, angle and deformation or the geometric transformation of the model. There are currently two main solutions to the above problems. The first is to construct a training data set with sufficient expected changes. This is usually achieved by increasing the existing data samples, such as resizing or cropping, rotating the existing data samples. Stable fitting representations can be learned from the data, but usually at the cost of expensive training and complex model parameters; the second is to use transformation invariant features and algorithms. The above methods have two disadvantages. First, it is assumed that the geometric transformation is fixed and known, and this prior knowledge is used to expand the data, design features and algorithms. This assumption prevents the generalization of new tasks with unknown geometric transformations, so that the established model does not have good generalization performance. Secondly, for overly complex transformations, the difficulty of designing features and algorithms with transformation invariance increases, and the feasibility is extremely low. In recent years, convolutional neural networks have achieved remarkable success in visual recognition tasks such as image classification, semantic segmentation, and object detection. However, they still have the above two disadvantages.

[0003] The ability of convolutional neural network-based models to model geometric transformations mainly comes from extensive data augmentation, large model capacity, and some simple hand-designed modules. In short, CNN is limited to modeling large unknown transformations. This limitation stems from the fixed geometric structure of CNN modules: convolution units sample input features at fixed positions; pooling layers reduce spatial resolution at a fixed ratio; ROI (region-of-interest) pooling layers divide regions of interest into fixed spatial containers, etc., all of which lack internal mechanisms to handle geometric transformations. In previous convolutional neural networks, the receptive field size of all activated units in the same layer of the network is the same, which is undesirable for deep convolutional neural network layers that encode semantics in spatial positions. Since different positions may correspond to objects of different scales or deformations, visual recognition that requires fine positioning requires adaptive determination of the scale or receptive field size. On the other hand, although object detection has made great progress in recent years, all detection methods still rely on feature extraction based on the original detection box. This is not the best method, especially when dealing with complex target situations, such as complex target contours, multiple target overlaps, etc. Summary of the invention

[0004] The main purpose of the present invention is to provide a deformable convolutional network based on deep learning.

[0005] The technical solution adopted by the present invention is: a deformable convolutional network based on deep learning, including a deformable convolutional module and a deformable ROI pooling layer module;

[0006] The deformable convolution module adds a two-dimensional offset to the position of the regular grid sampling in the standard convolution, so that the sampling grid can be deformed freely; the offset is obtained from the mapping feature in front of it through an additional convolution layer;

[0007] The deformable ROI pooling layer module adds an offset to each small partition position in the pooling layer, and the offset can be learned from the previous feature map and the region of interest, thereby achieving adaptive positioning for targets with different shapes.

[0008] Furthermore, the deformable convolution module includes two steps:

[0009] S1, use a regular grid R to sample the input mapping feature x;

[0010] S2, uses w to represent the sum of weighted samples, the grid R defines the size and expansion of the receptive field, and R uses array coordinates to represent the position of the convolution kernel.

[0011] Furthermore, the steps S1 and S2 are specifically as follows:

[0012] For each position p on the output map feature y 0 ,have:

[0013]

[0014] Among them, p n Enumerate the positions in R;

[0015] In deformable convolution, the regular grid R is augmented with an offset {Δp n |n=1,...,N}, where N=|R|, so:

[0016]

[0017] At this time, sampling is done at irregular and offset locations p n +Δp n , due to Δp n Often appears as a decimal.

[0018] Furthermore, the steps S1 and S2 further include:

[0019] For Δp n To process:

[0020]

[0021] Where p represents any position (p = p 0 +p n +Δp n ), q enumerates all complete spatial locations of the mapped feature x, and G(., ) represents the kernel of bilinear interpolation; G is two-dimensional and is then divided into two one-dimensional kernels:

[0022] G(q,p)=g(q x , p x )·g(q y , p y )

[0023] Wherein, g(a, b) = max(0, 1-|ab|);

[0024] The offset is obtained by applying a convolutional layer on the same input mapping features, and the output offset has the same spatial resolution as the input mapping features; during training, the output features and offsets of the convolution kernel are learned simultaneously.

[0025] Furthermore, the deformable ROI pooling layer module converts an input rectangular detection box of any size into a fixed-size feature, giving any input mapping feature x, a ROI rectangular detection box of size w×h and a top left corner p 0 ; ROI pooling divides the ROI rectangular detection box T into k×k small areas, where k is a free parameter; and outputs a k×k mapping feature y. For the (i, j)th small area (0≤i, j<k), there is

[0026]

[0027] Among them, j is the pixel point of the (i, j)th small area, and the range of the (i, j)th small area is:

[0028] as well as

[0029] In the deformable ROI pooling layer, the offset {Δp ij |0≤i, j<k} is added to the position of the small area of ​​space, so that

[0030] y(i, j) = ∑ p∈T(i，j) x(p 0 +p+Δp ij ) / n ij .

[0031] Furthermore, the deformable ROI pooling layer module also includes:

[0032] For Δp ij Processing:

[0033]

[0034] G(q,p)=g(q x , p x )·g(q y , p y )

[0035] For the deformable ROI pooling layer, the pooled mapping features are first generated by pooling, and the generated mapping features are passed through the fully connected layer to generate standardized offsets Then, through element-level calculation, the width and height of the detection box are converted to Δp ij , specifically:

[0036]

[0037] Here, γ is a pre-set scalar used to adjust the size of the offset; the deformable ROI pooling layer first converts all input mapping features of each target category into k 2 There is no need to distinguish each category. Such a score mapping is represented by {x i，j}, where (i, j) enumerates each small area in the detection box; pooling is performed on the mapping of these scores, and the output value for the (i, j)th small area is obtained by a mapping feature {x i，j}Sum the results.

[0038] Advantages of the present invention:

[0039] The present invention enhances the conversion modeling capability of convolutional neural networks by inventing two new modules. Both modules are based on using additional offsets to increase the spatial sampling position in the module, and continuously learning the offsets from the target task without the need for additional supervised learning. These two new modules can replace ordinary modules in existing mainstream networks, and can be trained end-to-end through standard back propagation to produce a deformable convolutional network, which greatly improves the detection accuracy when dealing with complex target situations, such as complex target contours, multiple target overlaps, etc.

[0040] The present invention proposes a deformable convolutional neural network, which is based on the basic idea of ​​adding additional offsets in the spatial sampling position in the convolution and the ROI pooling layer, and learning the offsets from the target task. When the deformable convolutions are superimposed, the influence of the compound deformation is far-reaching. They can be adaptively adjusted according to the target scale and shape in the deformable convolution, which greatly improves the detection accuracy of target detection, semantic segmentation, etc. in complex situations.

[0041] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0043] Figure 1 This is a deformable convolutional network framework diagram based on deep learning of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] refer to Figure 1 ,like Figure 1 As shown, a deformable convolutional network based on deep learning includes a deformable convolution module and a deformable ROI pooling layer module;

[0046] The two new modules proposed greatly enhance the ability of convolutional neural networks to model geometric transformations. The first is the deformable convolution module, which adds a two-dimensional offset to the position of the regular grid sampling in the standard convolution, so that the sampling grid can be deformed freely. The offset is obtained from the previous mapping features through an additional convolution layer, so the deformable convolution network is constrained by the input features in a local, dense and adaptive manner. The deformable ROI pooling layer module adds an offset to each small partition position in the pooling layer. Similarly, the offset can be learned from the previous feature map and the region of interest, thereby achieving adaptive positioning for objects with different shapes. Both modules are lightweight modules that provide a small number of parameters and calculations for offset learning. They can easily replace ordinary peers in deep convolutional neural networks and can be easily trained end-to-end using standard backpropagation.

[0047] First, for deformable convolution, the general two-dimensional convolution includes two steps: (1) use a regular grid R to sample the input mapping feature x; (2) use w to represent the sum of weighted samples, and the grid R defines the size and expansion of the receptive field. R uses array coordinates to represent the position of the convolution kernel, such as defining a 3×3 convolution kernel with an expansion parameter of 1:

[0048] R={(-1,-1), (-1,0),..., (0,1), (1,1)}

[0049] For each position p on the output map feature y 0 ,have:

[0050]

[0051] Among them, p n Enumerate the positions in R.

[0052] In deformable convolution, the regular grid R is augmented with an offset {Δp n |n=1,...,N}, where N=|R|, so:

[0053]

[0054] At this time, sampling is done at irregular and offset locations p n +Δp n , due to Δp n It often appears in the form of decimals. This patent handles it in the following way:

[0055]

[0056] Where p represents any position (p = p 0 +p n +Δp n ), q enumerates all complete spatial locations of the mapped feature x, and G(·,·) represents the kernel of bilinear interpolation. G is two-dimensional and is then divided into two one-dimensional kernels:

[0057] G(q,p)=g(q x , p x )·g(q y , p y )

[0058] Among them, g(a,b)=max(0,1-|ab|).

[0059] The offset is obtained by applying a convolutional layer on the same input map features, and the output offset has the same spatial resolution as the input map features. During training, the output features and offsets of the convolution kernel are learned simultaneously.

[0060] For the second module, the deformable ROI pooling layer. All target detection methods based on region proposals use the ROI pooling method, which converts an input rectangular detection box of any size into a fixed-size feature. Given any input mapping feature x, a ROI rectangular detection box of size w×h and the upper left corner p 0 ROI pooling divides the ROI rectangular detection box T into k×k small areas (k is a free parameter) and outputs a k×k mapping feature y. For the (i, j)th small area (0≤i, j<k), there is

[0061]

[0062] Among them, n ij is the pixel point of the (i, j)th small area, and the range of the (i, j)th small area is:

[0063] as well as

[0064] In the deformable ROI pooling layer, the offset {Δp ij |0≤i, j<k} is added to the position of the small area of ​​space, so that

[0065]

[0066] Similarly, for Δp ij Most of the cases are also decimals, which are also processed by the above method:

[0067]

[0068] G(q,p)=g(q x , p x )·g(q y , p y )

[0069] For the deformable ROI pooling layer, the pooled mapping features are first generated by pooling, and the generated mapping features are passed through the fully connected layer to generate standardized offsets Then, through element-level calculation, the width and height of the detection box are converted to Δp ij , specifically:

[0070]

[0071] Here, γ is a pre-set scalar used to adjust the size of the offset. The deformable ROI pooling layer first converts all input mapping features of each target category into k 2There is no need to distinguish each category, such a score mapping is expressed as {x i，j}, where (i, j) enumerates each small area in the detection box. Pooling is performed on the mapping of these scores, and the output value for the (i, j)th small area is obtained by a mapping feature {x i，j}Sum the results.

[0072] In general, the outputs and inputs of deformable convolution and deformable ROI pooling are the same as other common modules, so they can easily replace their common counterparts in existing convolutional neural networks, thereby improving the detection accuracy of the entire detection algorithm.

[0073] In the present invention, two new modules are invented to enhance the conversion modeling capability of convolutional neural networks. Both modules are based on using additional offsets to increase the spatial sampling position in the module, and continuously learning the offsets from the target task without the need for additional supervised learning. These two new modules can replace the ordinary modules in the existing mainstream network, and can be trained end-to-end through standard back propagation to produce a deformable convolutional network, which greatly improves the detection accuracy when dealing with complex target situations, such as complex target contours, multiple target overlaps, etc.

[0074] The STN (Spatial Transform Networks) algorithm is the first to use data to learn spatial transformations in a deep learning framework. It distorts mapping features through affine transformations of global parameter transformations. This distortion is expensive, and learning known transformation parameters is very difficult. STN has been successful in solving the difficult problem of classifying small-scale images. The inverse method of STN replaces the expensive feature distortion with effective transformation parameter propagation, but requires a large number of model parameters. The deformable convolutional network in this patent does not use global parameter transformation and feature distortion, but samples mapping features in a local and dense manner, and generates new mapping features through weighted summation. The deformable convolutional network is easy to integrate into other CNN frameworks and is simple to train.

[0075] Active Convolution network is the current mainstream method, which uses offset to increase the sampling position in convolution and learns the offset through end-to-end backpropagation, but the versatility and adaptability of this method are poor. First, it shares the offset at different spatial positions; second, the offset in this method is a static model parameter for each task or each training, while the offset generated by the deformable convolution network in this patent is a dynamic model output according to the change of image position.

[0076] ERF (Effective Receptive Field) found that the pixels in the receptive field do not contribute equally to the output. Pixels close to the center have a greater impact. The effective receptive field only accounts for a small part of the theoretical receptive field and presents a Gaussian distribution. Although the theoretical receptive field size increases linearly with the convolution layer, the size of the effective receptive field increases linearly with the square root of the number of convolution layers. Therefore, even the top-level units of deep CNNs may not have a large enough receptive field, which illustrates the need for adaptive receptive field learning. The deformable convolution proposed in the patent has the ability to adaptively learn receptive fields.

[0077] TIF (Transformation invariant features) has achieved some results in designing transformation invariant features, such as size-invariant feature transformation and ORB, but these designs are based on prior transformations, which are used to manually craft the structure of feature extraction algorithms, or are fixed in SIFT or have learnable parameters based on CNN. They cannot handle unknown transformations in new tasks.

[0078] The deformable module proposed in this paper generalizes various transformations, which are learned from the target task.

[0079] In summary, this patent proposes a deformable convolutional neural network, which is based on the basic idea of ​​adding additional offsets in the spatial sampling position and ROI pooling layer in the convolution and learning the offsets from the target task. When the deformable convolutions are superimposed, the impact of the compound deformation is far-reaching. They can be adaptively adjusted according to the target scale and shape in the deformable convolution, greatly improving the detection accuracy of target detection, semantic segmentation, etc. in complex situations.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image classification method based on a deep learning deformable convolutional network, characterized in that: The deformable convolutional network includes a deformable convolution module and a deformable ROI pooling layer module; The deformable convolution module adds a two-dimensional offset to the position of the regular grid sampling in the standard convolution, so that the sampling grid can be deformed freely; the offset is obtained from the mapping feature in front of it through an additional convolution layer; The deformable ROI pooling layer module adds an offset to each small partition position in the pooling layer, and the offset can be learned from the previous feature map and the region of interest, so as to achieve adaptive positioning for image objects with different shapes; The deformable ROI pooling layer module converts an input rectangular detection box of any size into a fixed-size feature, giving any input mapping feature , a size of The ROI rectangular detection box and the upper left corner ; ROI pooling divides the ROI rectangular detection box T into A small area, is a free parameter; and outputs a Mapping features , for the Small area ,have ； in, It is pixels in a small area, and the The range of each small area is: as well as ; In the deformable ROI pooling layer, the offset is added to the location of the small area of ​​space, so that ； The deformable ROI pooling layer module also includes: right Processing: ； ； For the deformable ROI pooling layer, the pooled mapping features are first generated by pooling, and the generated mapping features are passed through the fully connected layer to generate standardized offsets , and then through element-level calculation, the width and height of the detection box are converted to , specifically: ； Here It is a pre-set scalar used to adjust the size of the offset; the deformable ROI pooling layer first converts all input mapping features of each image target category into There is no need to distinguish the target category of each image. Such a score mapping is expressed as ,in Enumerate each small area in the detection box; pool on the map of these scores, for the first The output value of a small area is obtained by a mapping feature corresponding to the small area. The sum of the results.

2. The image classification method based on a deformable convolutional network based on deep learning according to claim 1, characterized in that: The deformable convolution module consists of two steps: S1, mapping features in the input Use a regular grid Take samples; S2, with represents the sum of weighted sampling, grid Define the receptive field size and dilation, Use array coordinates to represent the convolution kernel position.

3. The image classification method based on a deformable convolutional network based on deep learning according to claim 2, characterized in that: The steps S1 and S2 are specifically as follows: For output mapping features Each position on ,have: ； in, enumerate Position in In deformable convolution, the regular grid Augmenting with offset ,in , so we have: ； At this time, sampling is at irregular and offset locations ,because Often appears as a decimal.

4. The image classification method based on a deformable convolutional network based on deep learning according to claim 2, characterized in that: The steps S1 and S2 further include: right To process: ； in, Represents any position ( ), Enumerates the mapping characteristics All complete spatial locations, Represents the kernel of bilinear interpolation; is two-dimensional, and is then divided into two one-dimensional kernels: ； in, ; The offset is obtained by applying a convolutional layer on the same input mapping features, and the output offset has the same spatial resolution as the input mapping features; during training, the output features and offsets of the convolution kernel are learned simultaneously.

Images