Method for sharing offset between convolutional neurons by lightweight deformable module

Abstract

A method for sharing offset between convolutional neurons by a lightweight deformable module belongs to the technical field of deep learning, and comprises the following steps: S1, designing a lightweight deformable module TDCN: introducing an offset field representation method, reducing the channel dimension of an offset domain from 2N to 2, N being the number of sampling points, no matter how large a convolution kernel is selected, and the channel dimension of the offset domain is reduced from 2N to 2N, and no matter how large the convolution kernel is selected, the channel dimension of the offset domain is reduced from 2N to 2N; the channel number of the offset domain is always 2; sharing offset through adjacent convolution operations, so that neurons of the convolution operations learn a proper offset, and establishing association between the neurons; s2, the TDCN is integrated into a Convnet system, and various different backbone networks are used; s3, the TDCN is integrated into the Faster R-CNN system, and various different backbone networks are used; and S4, the TDCN is integrated into the RetinaNet system, and various different backbone networks are used. According to the method, the problem of traditional deformable convolution is solved, meanwhile, the capacity of adaptively adjusting the convolution kernel to adapt to the geometric change of the object is reserved, and wider applicability is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and in particular relates to a method for sharing offsets between convolutional neurons in a lightweight deformable module. Background Art

[0002] With the popularization of large-scale datasets and computing resources, convolutional neural networks (CNNs) have become the mainstream in visual recognition. Starting from AlexNet[1], many deeper and more effective neural network architectures have been proposed, such as VGG[2], GoogleNet[3], ResNet[4], ResNeXt[5], EfficientNet[6,7], etc. Recent research results such as InternImage[8] and ConvNeXt[9] have shown that visual models based on convolutional networks maintain robust performance, efficiency, simplicity, and have suitable inductive biases[10,11]. In particular, in fields such as image generation[12,13], convolutional methods are still the first choice. This highlights the enduring value of convolutional methods. In addition to the development of architectural design, more complex convolution operations such as deep convolution

[14] and deformable convolution[15,16] have been formulated. The challenges of geometric changes faced by object recognition and detection, including scale, pose, viewpoint, and part deformation, have driven the development of these technologies. The current state-of-the-art approach to addressing these issues is the Deformable Convolutional Networks (DCN)

[16] , which introduces two modules to help CNNs model these changes. The first module is the deformable convolution, which adapts to geometric changes by offsetting the grid sampling positions of the standard convolution, which are determined by the displacement learned from the previous feature map. The second module is the deformable region of interest pooling (ROI pooling), in which the offsets of the bin positions in the ROI pooling

[17] are learned to enhance the feature representation. The introduction of these modules enables the neural network to dynamically adjust its feature representation to adapt to the configuration and structure of the object, thereby significantly improving the accuracy of object detection. This approach enables the network to better adapt to the geometric changes of the object by deforming the sampling and pooling patterns, thereby improving the performance and accuracy of the model in practical applications.

[0003] When using deformable convolution, we find that the samples of an activated unit tend to be concentrated around the object in which it is located. However, this coverage is not precise and the samples often extend outside the region of interest. To address this problem,

[15] proposed a new version of deformable convolutional networks, called DeformableConvNetsv2 (DCNv2). Based on DCNv1 (i.e. DCN), DCNv2 adds a modulation mechanism so that each sample not only undergoes a learned offset, but is also modulated by a learned feature amplitude. In this way, the network module is able to simultaneously adjust the spatial distribution and relative influence of its samples. However, similar to conventional convolution, different convolution neurons in the original DCNv2 have independent linear projection weights and offsets, so its parameter and memory complexity are proportional to the total number of sampling points, which significantly limits the efficiency of the model, especially in large-scale models. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes: a method for sharing offsets between convolutional neurons using a lightweight deformable module, comprising the following steps:

[0005] S1. Design of lightweight deformable module TDCN:

[0006] The offset field representation method is introduced to reduce the offset field channel dimension from 2N to 2, where N is the number of sampling points. No matter how large the convolution kernel is, the number of channels in the offset field is always 2.

[0007] By sharing offsets between adjacent convolution operations, the neurons of these convolution operations learn a suitable offset and establish associations between them;

[0008] S2, integrate TDCN into the Convnet system and use various backbone networks;

[0009] S3. Integrate TDCN into the Faster R-CNN system and use various backbone networks.

[0010] S4. Integrate TDCN into the RetinaNet system and use various backbone networks.

[0011] Furthermore, the step S1 includes the following steps:

[0012] TDCN maps the offset in one direction to a frame in the offset domain channel dimension, and the convolution kernel generates offsets in two directions, corresponding to the width direction and the height direction, namely the W-offset field and the H-offset field.

[0013] In DCN, the same position of the channel dimension on the offset domain is extracted to form the convolution kernel offset when the position is the sampling center. In TDCN, the size of the input feature map is the same as the offset domain W and H dimensions. The offset domain is at the same position as the convolution sampling grid on the input feature map, corresponding to the convolution kernel offset at this time; adjacent pixels that overlap in convolution operations will jointly learn an offset that adapts to them.

[0014] Furthermore, in step S2, the convolutional neural network CNN is used as a feedforward neural network, and its artificial neurons respond to surrounding units to perform large-scale image processing; a convolutional neural network ConvNet is designed to process image classification tasks on the MNIST dataset, and the network combines traditional convolutional layers and deformable convolutional layers to enhance its adaptability to spatial transformations in the input image, and Conv3 and Conv4 are used as deformable convolutional replacement layers.

[0015] Furthermore, in step S3, based on the feature extraction network ResNet50, the FPN network structure is integrated, and the feature maps of the high and low layers of ResNet50 are superimposed and fused to obtain the fused five feature maps P2 to P6; these five feature maps use multiple predefined anchor boxes at each spatial position to form a group of candidate boxes; for each anchor box, the regression layer predicts the coordinates of the box, and the classification layer predicts whether the box contains the target; these candidate boxes are then sent to R-CNN for accurate classification and positioning of the target, so as to realize the detection and recognition of the target in the image.

[0016] Furthermore, in step S4, the RetinaNet network is composed of ResNet50, FPN feature pyramid, classification subnetwork and bounding box regression subnetwork connected in sequence. ResNet50 and FPN together constitute the backbone network to realize feature extraction of the target. The classification subnetwork and bounding box regression subnetwork are respectively responsible for target classification and position regression of the feature map output by FPN.

[0017] The beneficial effects of the present invention are as follows: the present invention enables the neurons of these convolution operations to learn a suitable offset and establish associations between them by sharing offsets between adjacent convolution operations. This association still exists when the network depth increases, thereby protecting local features. This innovative method not only solves the problems existing in traditional deformable convolution, but also maintains the ability of deformable convolution to adaptively adjust the convolution kernel to adapt to the geometric changes of the object, thereby obtaining more widely applicable characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a comparison chart of standard convolution and deformable convolution sampling;

[0019] Figure 2 Schematic diagram of deformable convolution structure;

[0020] Figure 3 Schematic diagram of TDCN structure;

[0021] Figure 4 Comparison diagram between DCN and TDCN;

[0022] Figure 5 It is the ConvNet network architecture;

[0023] Figure 6 It is a simplified diagram of the characteristic pyramid structure;

[0024] Figure 7 This is the structure diagram of FasterRCNN after integrating FPN network;

[0025] Figure 8 This is the Retinanet structure diagram;

[0026] Fig. 9 The number of parameters (left) and computation (right) added to ConvNet using DCNv1, DCNv2, and TDCN.

[0027] Fig.10 Bar graphs showing the amount of parameters (a) and computational effort (b) added to FasterR-CNN using DCNv1, DCNv2, and TDCN.

[0028] Fig.11 Bar graphs showing the amount of parameters (a) and computational effort (b) added when using DCNv1, DCNv2, and TDCN on Retinanet. Fig.12 This is the result of Faster R-CNN's target detection on the Pascal VOC 2007test. Fig.13 This is a diagram showing the target detection results of Retinanet on the Pascal VOC 2007test. DETAILED DESCRIPTION

[0029] In order to make the technical means and objectives of the present invention easy to understand, the present invention is further described below in combination with a specific implementation method. A method for sharing offsets between convolutional neurons using a lightweight deformable module includes the following steps:

[0030] S1. Design of lightweight deformable module TDCN:

[0031] The offsetfield representation method is introduced to reduce the offset field channel dimension from 2N to 2, where N is the number of sampling points. No matter how large the convolution kernel is, the number of channels in the offset field is always 2.

[0032] By sharing offsets between adjacent convolution operations, the neurons of these convolution operations learn a suitable offset and establish associations between them;

[0033] S2, integrate TDCN into the Convnet system and use various backbone networks;

[0034] S3. Integrate TDCN into the FasterR-CNN system and use various backbone networks;

[0035] S4. Integrate TDCN into the RetinaNet system and use various backbone networks.

[0036] To address the above problems, we proposed a new method to share offsets between convolutional neurons. With this improvement, we designed a lightweight deformable module (TinyDeformableConvolution, TDCN), which not only maintains the lightweight characteristics, but can also be easily integrated into existing network architectures. Specifically, we integrated TDCN into Convnet, FasterR-CNN

[17] , RetinaNet

[18] systems, and used a variety of different backbone networks. Extensive experiments conducted on the MNIST and PascalVOC benchmarks show that compared with DCNv1 and DCNv2, TDCN has achieved significant improvements in object detection, especially in small object detection. TDCN reduces the number of parameters and computation while maintaining excellent experimental results.

[0037] Deformable Convolution:

[0038] The inherent geometric structure of standard convolutions is insufficient for modeling objects of unknown shapes

[19] . Figure 1 As shown in the figure, taking a 3×3 convolution kernel as an example, the convolution kernel of the standard convolution is a fixed rectangle, while the deformable convolution allows each sampling point of the convolution kernel to learn an offset to adapt to the geometric shape of the object. Figure 1 Different convolution sampling shapes are shown: (a) The sampling shape of the standard convolution is a fixed rectangle; (b) The sampling position changes according to the offset, realizing deformable convolution; (c) By changing the offset, a dilation rate of 2 is formed to achieve the effect of scale transformation; (d) In special cases, the rotation effect of the convolution kernel is achieved through an appropriate offset.

[0039] Standard convolution samples from a fixed grid R, and each sampling point is weighted by the convolution kernel. Deformable convolution is based on the calculation of standard convolution, but adds an offset when sampling. For example, a 3×3 convolution kernel with a dilation rate of 1

[16] :

[0040] R={(-1,-1),(-1,0),...,(0,1),(1,1)}(1)

[0041] In the formula, each element in R represents the offset of all positions of the convolution kernel relative to the center position.

[0042] For each position P on the output feature map 0 , is calculated by the following expression, which is expressed as

[16] :

[0043]

[0044] In the formula, x is the input feature map, y is the output feature map, w(p n ) indicates p n The weight of the position, x(p n ) indicates that the input feature map is at point p n The pixel value at the location.

[0045] After sampling the input feature map x, an offset {Δp n |n=1,2,…,N}, where N is the number of sampling points.

[0046]

[0047] In the formula, Δp n For p n The position offset. Due to Δp n Basically a decimal, x(p 0 +p n +Δp n ) may not exist at a point on the input feature map, so a bilinear interpolation algorithm is used

[16] .

[0048] like Figure 2 As shown in the figure, during the deformable convolution process, an additional convolution layer is used to convolve the input feature map to obtain the offset, and an offset domain with the same size as the input feature map is obtained, where the number of channels in the offset domain is 2N, because the two-dimensional offset contains horizontal and vertical offsets, and each input channel does not share the offset.

[0049] DCN and CNN have the same input and output. The difference is that DCN adds convolutional layers and fully connected layers for learning offsets. However, due to the limited adaptive learning ability of convolution kernels, DCNv1 still has the problem of feature sampling that is irrelevant to image content. The advantage of DCN over CNN lies in its ability to adapt to geometric changes of objects. The feature sampling area is closer to the shape of the object, but the sampling area may far exceed the area of ​​interest, causing the features to be affected by irrelevant image content

[15] .

[0050] Modulated Deformable Convolution:

[0051] In 2019, the paper

[15] proposed DCNv2, which improved DCNv1. By more comprehensively integrating deformable convolution within the network and introducing a modulation mechanism that expands the range of deformable modeling, the geometric modeling capability of deformable convolution is comprehensively enhanced, and the ability of the network model to focus on the key areas of the image is improved. The deformable convolution module in DCNv2 adds a modulation mechanism, which can not only learn the offset of the sampling point, but also learn the weight of each sampling point. Let x p and p Denoting the features of position p in the input feature map x and the output feature map y respectively, the modulated deformable convolution can be expressed as:

[0052]

[0053] Where K represents the total number of sampling points, and k enumerates the sampling points. k ∈R C×C represents the projection weight of the kth sampling point, m k ∈R represents the modulation scalar of the kth sampling point, which is normalized by the sigmoid function. k represents the kth position sampled on a predefined grid, e.g., {(-1,-1),(-1,0),...,(0,+1),...,(+1,+1)}, just like in regular convolution, Δp k is the offset corresponding to the kth grid sampling position. From the equation, we can see that (1) the sampling offset w k ∈R C×C is too flexible and may cause feature loss in deep networks; (2) For adaptive spatial aggregation, each pixel needs to learn the corresponding sampling offset Δp k and the modulation scalar m k , and is subject to the input x.

[0054] DCNv2 needs to independently calculate offsets and weights at each sampling point, which not only increases the computational burden, but also increases the memory usage during model training and inference. In addition, since each convolution kernel requires separate weights and offsets, this independence makes the model very time-consuming when processing large-scale datasets, especially when high-resolution inputs or complex tasks are required. Due to hardware limitations, large-scale models are very sensitive to the parameters and memory costs of core operators, which are still problems that need to be solved in practice. To deal with these challenges, you can consider reducing the number of independent weights and offsets, reducing the computational and memory burdens by sharing offsets or reducing the number of sampling points. These optimization methods can significantly improve the efficiency of DCNv2 in large-scale models, making it more practical in practical applications, so we propose TDCN, introducing a strategy of shared offsets between convolutional neurons to optimize the performance and efficiency of the model.

[0055] Lightweight Deformable Convolution:

[0056] This paper introduces a new offset field representation method, such as Figure 3 As shown in the figure, the offset domain channel dimension is reduced from 2N to 2, where N is the number of sampling points. In traditional deformable convolution, the offset is unlimited on the feature map, and local features gradually disappear as the network depth increases. At the same time, as the size of the convolution kernel increases, the number of parameters and computational complexity also increase dramatically, which imposes restrictions on the selection of the convolution kernel. However, the improved method proposed in this paper (see Table 1) breaks through this limitation. No matter how large the convolution kernel is selected, the number of channels in the offset domain is always 2. In addition, the method in this paper shares the offset through adjacent convolution operations, so that the neurons of these convolution operations learn a suitable offset and establish associations between them. This association still exists when the network depth increases, thereby protecting local features. This innovative method not only solves the problems existing in traditional deformable convolution, but also maintains the ability of deformable convolution to adaptively adjust the convolution kernel to adapt to the geometric changes of the object, thereby obtaining more widely applicable characteristics.

[0057] Table 1 Comparison of the number of channels in the offset domain between DCN and TDCN

[0058]

[0059]

[0060] like Figure 3 As shown in the figure, TDCN maps the offset in one direction to a frame of the offset domain channel dimension, and the convolution kernel generates offsets in two directions, corresponding to the width direction (W direction) and the height direction (H direction), namely W-offsetfield and H-offsetfield.

[0061] In DCN, such as Figure 4 As shown in , the same position of the channel dimension in the offset domain is extracted to form the convolution kernel offset when the position is the sampling center. However, in TDCN (such as Figure 4 As shown in the figure, the size of the input feature map is the same as the offset domain W and H dimensions, and the offset domain is at the same position as the convolution sampling grid on the input feature map, corresponding to the convolution kernel offset at this time. Pixels that overlap in adjacent convolution operations will jointly learn an offset that adapts to them. This feature not only reduces the number of parameters, but also preserves the connection between adjacent pixels, solving the problem of feature loss in deep neural networks.

[0062] For a feature map with an input of W×H, where W represents the width of the input feature map and H represents the height of the input feature map, the size of the DCN offset domain P is expressed as follows:

[0063] Size P =2N×W×H(5)

[0064] In the formula, Size P Represents the size of the offset domain P, and N is the number of sampling points.

[0065] The size of the TDCN offset field P is expressed as follows:

[0066] Size P =2×W×H(6)

[0067] According to equations (5) and (6), it can be seen that the TDCN proposed in this paper reduces the channel dimension of the offset domain from 2N to 2, greatly reducing the number of parameters.

[0068] Network Model:

[0069] In order to verify the effectiveness of the lightweight deformable convolution (TDCN) proposed in this paper, we conducted a variety of comparative experiments, using DCNv1, DCNv2 or TDCN to replace the specified layer of ordinary convolution on different basic networks for comparative analysis. Specifically, a convolutional neural network Convnet was designed as a baseline model for verification in the image classification scenario. In addition, the two-stage detector FasterR-CNN with ResNet-50 as the backbone network and the single-stage detector RetinaNet were selected for verification in the target detection scenario. ResNet-50 goes through four blocks, and conv3~conv5 are used as replacement layers in this paper. The recognition accuracy (Accuracy), the average precision (mAP) of the bounding box, the number of parameters (params) and the amount of calculation (FLOPs) are used as evaluation indicators for comparison. The application on different networks helps to verify the effectiveness and robustness of TDCN in practical applications.

[0070] Convolutional Neural Network (CNN) is a feedforward neural network in which artificial neurons can respond to surrounding units and can process large images. CNN is mainly used in image recognition, image classification and retrieval, target location detection, target segmentation and other fields. For example, MNIST handwritten data recognition, Cats and Dogs War, etc. It can also be applied to natural language processing and speech recognition. We designed a convolutional neural network Convnet to handle image classification tasks on the MNIST dataset. The network combines traditional convolutional layers and deformable convolutional layers to enhance its adaptability to spatial transformations in the input image. The ConvNet network architecture is as follows: Figure 5 As shown in Figure 1, it is an ideal benchmark to demonstrate the effectiveness of TDCNv3’s improvements in real-world image processing tasks. In this paper, Conv3 and Conv4 are used as deformable convolution replacement layers.

[0071] Faster R-CNN is a very effective object detection algorithm and is still the basis of many object detection algorithms to this day

[20] . As a two-stage algorithm, Faster R-CNN is more complex and slower than one-stage algorithms, but its detection accuracy is generally higher

[21] . In fact, Faster R-CNN performs very well in detection, but its detection speed and training speed still have room for improvement

[22] . Faster R-CNN can use a variety of backbone feature extraction networks, commonly used ones include VGG, ResNet and Xception

[23] . This paper selects ResNet-50 as the backbone network for experiments. In traditional convolutional neural networks, image features are usually extracted by stacking multiple convolutional layers. However, in object detection tasks, objects of different scales may require features at different levels for detection. Lin et al.

[18] proposed Feature Pyramid Networks (FPN). FPN allows the network to simultaneously utilize semantic information at different scales, so that the detector can better identify objects of different scales. The schematic diagram of the feature pyramid structure is shown in Figure 6 shown.

[0072] This study integrates the FPN network structure (such as Figure 7As shown in the figure, the feature maps of the high and low layers of ResNet50 are superimposed and fused to obtain the fused five feature maps P2 to P6. These five feature maps use multiple predefined anchor boxes at each spatial position to form a set of candidate boxes. For each anchor box, the regression layer predicts the coordinates of the box, and the classification layer predicts whether the box contains the target. These candidate boxes are then sent to R-CNN for accurate classification and positioning of the target, realizing the detection and recognition of the target in the image.

[0073] After continuous development, although the detection speed of single-stage object detection can meet the real-time requirements, there is still a certain gap in accuracy compared with the two-stage object detection represented by the R-CNN series. One of the main reasons is the imbalance in the number of positive and negative samples, which affects the effect of the training process. To solve this problem, the FocalLoss RetinaNet model

[18] was proposed to improve the imbalance between samples and achieve the accuracy of the two-stage detection method.

[0074] like Figure 8 As shown in Figure 1, the RetinaNet network is mainly composed of the ResNet50 backbone network, the FPN feature pyramid, the classification subnetwork, and the bounding box regression subnetwork

[24] . The backbone network is composed of ResNet50 and FPN to achieve feature extraction of the target. The classification subnetwork and the bounding box regression subnetwork are responsible for target classification and position regression of the feature map output by FPN respectively

[25] .

[0075] Dataset:

[0076] In order to comprehensively evaluate the performance of TDCN on different tasks and different data types, this paper uses MNIST

[26] and PascalVOC

[27] respectively. (1) The MNIST dataset has 10 categories, 60,000 images for training, and 10,000 images for testing. (2) The PascalVOC07+12 dataset has 20 categories, 16,551 images for training, and 4,952 images for testing. Testing on different datasets can verify the generalization ability of TDCN and evaluate whether the proposed method can adapt to different types of data and maintain good performance in different tasks. The task of the MNIST dataset is image classification, while the tasks of the PascalVOC dataset include image classification and object detection. By testing on these two datasets, we can fully understand the performance of TDCN in different tasks (classification and detection). By testing on two datasets with different characteristics, we can determine the scope of applicability and practicality of TDCN in practical applications. This is of great significance for applying TDCN to practical scenarios, such as handwritten digit recognition and object detection.

[0077] Experimental results and analysis:

[0078] Image Categories:

[0079] We integrated TDCN, DCNv1, and DCNv2 into the ConvNet network architecture to verify their effectiveness in image classification tasks. We selected the MNIST dataset as the verification benchmark, which is representative and widely used, and can ideally demonstrate the effectiveness of TDCN's improvements in actual image processing tasks. The final experimental results on the MNIST dataset are shown in Table 3. We compared the number of parameters, computational complexity, and image recognition accuracy of network models based on CNN, with shared offsets (TDCN), and non-shared offsets (DCNv1, DCNv2).

[0080] Table 3 Comparison of classification accuracy of conventional convolution, DCNv1, DCNv2 and TDCN on the MNIST dataset

[0081]

[0082] The results show that when two layers of CNN are replaced by DCNv1, DCNv2 or TDCN in the network, the accuracy of TDCN (the method in this paper) is improved by 2.31% compared with CNN. Fig. 9 As shown in the figure, the increased parameters and increased computational complexity when using DCNv1, DCNv2 or TDCN are compared. Specifically, the accuracy of DCNv1 is only improved by 1.73%, while the increased parameters are twice that of TDCN (0.06Mvs0.03M), and the increased computational complexity is more than 11 times that of TDCN (10.16Mvs0.88M); the accuracy of DCNv2 is only improved by 1.99%, while the increased parameters are more than three times that of TDCN (0.10Mvs0.03M), and the increased computational complexity is more than 24 times that of TDCN (15.24Mvs0.88M).

[0083] In summary, on the MNIST dataset, the TDCN we proposed not only reduces the number of parameters and computations, but also outperforms DCNv1 and DCNv2 in recognition accuracy. The additional number of parameters and computations using two layers of TDCN is much smaller than that of DCNv1 and DCNv2, which proves that our method brings more significant performance improvements under the premise of increasing the limited number of parameters and computations.

[0084] Object Detection:

[0085] We verified the effectiveness of TDCN in the object detection task on the PascalVOC dataset and applied it to two representative object detection frameworks: FasterR-CNN and RetinaNet. According to the results in Tables 4 and 5, after replacing the conventional convolutional layer, the mAP of FasterR-CNN and Retinanet increased by 3.88% and 3.56% respectively. The number of parameters and computations of using three layers of TDCN is comparable to that of using one layer of DCNv1.

[0086] Table 4 Comparison of target detection results of conventional convolution, DCNv1, DCNv2 and TDCN on FasterR-CNN

[0087]

[0088] Table 5 Comparison of target detection results of conventional convolution, DCNv1, DCNv2 and TDCN on Retinanet

[0089]

[0090] like Fig.10 , Fig.11 As shown in the figure, we give the increased number of parameters and the increased amount of computation when using DCNv1, DCNv2 or TDCN on Faster R-CNN and Retinanet. Compared with DCNv1 and DCNv2, TDCN uses the least number of parameters and computation. In general, the benefits brought by our proposed method TDCN with the same increase in the number of parameters are much higher than those of DCNv1 and DCNv2. Therefore, the method proposed in this paper is better than DCNv1 and DCNv2.

[0091] The detection results of TDCN and DCNv2 on FasterR-CNN and Retinanet are as follows: Fig.12 , Fig.13 As shown. It can be seen that Fig.12 In (a), the detection effect of the relatively large target car is slightly worse than that of DCNv2 (99.9% vs 99.8%), while the detection effect of the person with fewer pixels is significantly better than that of DCNv2 (85.5% vs 48.2%). Fig.12 (b) shows the same conclusion. Using TDCN and DCNv2, the detection accuracy of person is as high as 99.9%. TDCN can detect the smaller target horse at 94.3%. DCNv2 even misidentifies the horse as a sheep. Fig.12In (c), the detection accuracy of TDCNbicycle is 73.4%, while DCNv2 is only 48.3%, and TDCN detects the person that DCNv2 does not detect. Fig.13 The detection results in also show the same result. We can conclude that TDCN has a significantly better detection effect on smaller objects in the image than DCNv2, and the detection effect on larger objects is comparable to that of DCNv2.

[0092] [1]Krizhevsky A, Sutskever I, Hinton G E.ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6):84–90.

[0093] [2]Simonyan K, Zisserman A.Very Deep Convolutional Networks for Large-Scale Image Recognition[J].arXiv,2014:1409.1556.

[0094] [3]Szegedy C, Liu W, Jia Y. Going Deeper With Convolutions[C]. Proceedings of the IEEE conference on computer vision and pattern recognition, 2015:1–9.

[0095] [4]He K, Zhang

[0096] [5]Xie S,Girshick R,Dollar P.Aggregated Residual Transformations forDeep Neural Networks[C].Proceedings of the IEEE conference on computer visionand pattern recognition,2017:1492–1500.

[0097] [6]Tan M,Le Q.Efficientnet:Rethinking model scaling for convolutionalneural networks[C].International conference on machine learning.PMLR,2019:6105-6114.

[0098] [7]Tan M,Le Q V.Efficientnetv2:Smaller models and fastertraining.arXiv 2021[J].arXiv preprint arXiv:2104.00298.

[0099] [8]Wang W,Dai J,Chen Z,et al.Internimage:Exploringlarge-scale visionfoundation models with deformableconvolutions[C].Proceedings of the IEEE / CVFconference oncomputer vision and pattern recognition.2023:14408-14419.

[0100] [9]Liu Z,Mao H,Wu C Y,et al.A convnet for the 2020s[C].Proceedings ofthe IEEE / CVF conference on computer vision andpattern recognition.2022:11976-11986.

[0101]

[10] He K,Gkioxari G,Dollár P,et al.Mask r-cnn[C].Proceedings of theIEEE international conference on computervision.2017:2961-2969.

[0102]

[11] Xiao T,Liu Y,Zhou B,et al.Unified perceptual parsingfor sceneunderstanding[C].Proceedings of the Europeanconference on computer vision(ECCV).2018:418-434.

[0103]

[12] Saharia C,Chan W,Saxena S.PhotorealisticText-to-Image DiffusionModels with Deep LanguageUnderstanding[J].Advances in Neural InformationProcessingSystems,2022,35:36479–36494.

[0104]

[13] Rombach R,Blattmann A,Lorenz D,et al.High-resolution imagesynthesis with latent diffusionmodels[C] / / Proceedings of the IEEE / CVFconference on computervision and pattern recognition.2022:10684-10695.

[0105]

[14] Howard A G,Zhu M,Chen B,et al.Mobilenets:Efficientconvolutionalneural networks for mobile visionapplications[J].arXiv preprint arXiv:1704.04861,2017.

[0106]

[15] Zhu X,Hu H,Lin S,et al.Deformable convnets v2:Moredeformable,better results[C].Proceedings of the IEEE / CVFconference on computer visionand pattern recognition.2019:9308-9316.

[0107]

[16] Dai J,Qi H,Xiong Y,et al.Deformable convolutionalnetworks[C].Proceedings of the IEEE international conferenceon computer vision.2017:764-773.

[0108]

[17] Ren S,He K,Girshick R,et al.Faster r-cnn:Towardsreal-time objectdetection with region proposal networks[J].Advances in neural informationprocessing systems,2015,28.

[0109]

[18] Lin T Y,Goyal P,Girshick R,et al.Focal loss for denseobjectdetection[C].Proceedings of the IEEE internationalconference on computervision.2017:2980-2988.

[0110]

[19] Tao H.A label-relevance multi-direction interactionnetwork withenhanced deformable convolution for forest smokerecognition[J].Expert Systemswith Applications,2024,236:121383.

[0111]

[20] He Z,Zhang L,Gao X.Multi-adversarial Faster-RCNNwith ParadigmTeacher for Unrestricted Object Detection[J].International Journal ofComputer Vision,2023,131(3):680–700.

[0112]

[21] Li J,Zhu Z,Liu H.Strawberry R-CNN:Recognition andcounting modelof strawberry based on improved faster R-CNN[J].Ecological Informatics,2023,77:102210.

[0113]

[22] Xu J,Ren H,Cai S.An improved faster R-CNN algorithmfor assisteddetection of lung nodules[J].Computers in Biology and Medicine,2023,153:106470.

[0114]

[23] Zhang W,Zhu Q,Li Y.MAM Faster R-CNN:Improved Faster R-CNN basedon Malformed Attention Module for object detection on X-ray securityinspection[J].Digital Signal Processing,2023,139:104072.

[0115]

[24] Peng H,Chen H,Zhang X.Retinanet_G2S:a multi-scale feature fusion-based network for fruit detection of punna navel oranges in complex fieldenvironments[J].Precision Agriculture,2024,25(2):889–913.

[0116]

[25] Huang L, Wang Z, Fu

[0117]

[26] Lecun Y, Bottou L, Bengio Y.Gradient-based learning applied todocument recognition[J].Proceedings of the IEEE,1998,86(11):2278–2324.

[0118]

[27] Everingham M,Van Gool L,Williams CK I.The Pascal Visual ObjectClasses(VOC)Challenge[J].International Journal of Computer Vision,2010,88(2):303–338.

[0119] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for sharing offsets between convolutional neurons in a lightweight deformable module, characterized in that: The steps include: S1. Design of lightweight deformable module TDCN: The offset field representation method is introduced to reduce the offset field channel dimension from 2N to 2, where N is the number of sampling points. No matter how large the convolution kernel is, the number of channels in the offset field is always 2. By sharing offsets between adjacent convolution operations, the neurons of these convolution operations learn a suitable offset and establish associations between them; S2, integrate TDCN into the Convnet system and use various backbone networks; S3. Integrate TDCN into the Faster R-CNN system and use various backbone networks. S4. Integrate TDCN into the RetinaNet system and use various backbone networks.

2. The method for sharing offsets between convolutional neurons of a lightweight deformable module as claimed in claim 1, characterized in that The step S1 includes the following steps: TDCN maps the offset in one direction to a frame in the offset domain channel dimension, and the convolution kernel generates offsets in two directions, corresponding to the width direction and the height direction, namely the W-offset field and the H-offset field. In DCN, the same position of the channel dimension on the offset domain is extracted to form the convolution kernel offset when the position is the sampling center. In TDCN, the size of the input feature map is the same as the offset domain W and H dimensions. The offset domain is at the same position as the convolution sampling grid on the input feature map, corresponding to the convolution kernel offset at this time; adjacent pixels that overlap in convolution operations will jointly learn an offset that adapts to them.

3. The method for sharing offsets between convolutional neurons of a lightweight deformable module as claimed in claim 1, characterized in that In the step S2, the convolutional neural network CNN is used as a feedforward neural network, and its artificial neurons respond to surrounding units to perform large-scale image processing; a convolutional neural network ConvNet is designed to process image classification tasks on the MNIST dataset, and the network combines traditional convolutional layers and deformable convolutional layers to enhance its adaptability to spatial transformations in the input image, and Conv3 and Conv4 are used as deformable convolutional replacement layers.

4. The method for sharing offsets between convolutional neurons of a lightweight deformable module as claimed in claim 1, characterized in that In the step S3, based on the feature extraction network ResNet50, the FPN network structure is integrated, and the feature maps of the high and low layers of ResNet50 are superimposed and fused to obtain the fused five feature maps P2 to P6; these five feature maps use multiple predefined anchor frames at each spatial position to form a group of candidate frames; for each anchor frame, the regression layer predicts the coordinates of the frame, and the classification layer predicts whether the frame contains the target; these candidate frames are then sent to R-CNN for accurate classification and positioning of the target, so as to realize the detection and recognition of the target in the image.

5. The method for sharing offsets between convolutional neurons of a lightweight deformable module as claimed in claim 1, characterized in that: In step S4, the RetinaNet network is composed of ResNet50, FPN feature pyramid, classification subnetwork and bounding box regression subnetwork connected in sequence. ResNet50 and FPN together constitute the backbone network to realize feature extraction of the target. The classification subnetwork and bounding box regression subnetwork are respectively responsible for target classification and position regression of the feature map output by FPN.

Images