Optimization method of deformable convolution for enhancing geometric modeling

Abstract

The invention discloses an optimization method of deformable convolution for enhancing geometric modeling, which adopts limited deformable convolution, enhances the modeling capability of a model by increasing an adaptive limiting unit constraint offset, and enhances the image focusing capability by adjusting the offset weight constraint. On the basis of limited deformable convolution, lightweight work is carried out on the limited deformable convolution, three LDBottleneck are designed to adapt to different scenes, an LDC2f module is further obtained and can be applied to a target detector network, and brand new possibility is provided for efficiency and accuracy of a deep learning model.

Description

Technical Field

[0001] The present invention relates to an improvement of algorithms in the field of target detection, and more specifically to a method for further optimizing deformable convolution. Background Art

[0002] The fixed geometric structure of CNNs limits their ability to model large, unknown transformations. Operations such as convolutional units, pooling layers, and region-of-interest pooling layers are all fixed in their approach of sampling inputs at specific locations, reducing spatial resolution at fixed ratios, and segmenting regions of interest into fixed spatial boxes. These operations lack intrinsic mechanisms for handling geometric transformations. This fixed structure leads to significant problems. For example, all activations in a CNN layer have the same receptive field size, which hinders the semantic encoding of spatial locations by higher-level CNN layers. Standard convolution (SC) is widely used in various deep learning models, such as VGGNet, ResNet, and Inception, due to its simplicity, efficiency, and scalability. However, the receptive field of SC is fixed, making it insensitive to geometric variations in the input data. This makes the model less sensitive to geometric changes such as scale, pose, viewpoint, and local deformation. The fixed geometric structure of CNNs limits their flexibility and adaptability in handling geometric transformations. While SC's simplicity and efficiency make it a common choice, its fixed receptive field limits its ability to handle complex geometric transformations. These limitations have driven researchers to seek more flexible model structures and convolution operations that are better able to adapt to geometric transformations, thereby improving the model's ability to model various geometric variations. Unlike standard convolution, deformable convolution allows the convolution kernel to deform on the feature map. This means that the position of the convolution kernel weights is no longer fixed but can be dynamically adjusted based on learned offsets. This enables DCN to better adapt to irregular shapes and structures in the input data. The learned offsets can dynamically adjust the receptive field during feature extraction, thereby more accurately capturing subtle features and structural information of the object. Deformable convolution is considered to have great potential, but the learning process of DCN, in order to enhance the model's adaptability to irregular shapes and structures, may introduce irrelevant contextual information in certain situations. When visualizing the changes in the receptive field of DCNv1, the authors found that the coverage of objects was inaccurate, with samples spreading beyond the region of interest, and samples used for activation functions also overlapping the surrounding objects. DCNv2 expands on the deformable convolution layer and equips more convolution layers with offset learning capabilities, enabling DCNv2 to control sampling at a wider range of feature levels. And increase the modulation mechanism in the deformable convolution module, where each sample not only undergoes a learned offset but is also modulated by the learned feature amplitude. These two strategies are improved.

[0003] Even though DCNv2 has made some progress in trying to solve the problem of sample diffusion, there are still cases where samples diffuse beyond the region of interest. This diffusion may cause the model to introduce irrelevant contextual information in a specific task, thus affecting the performance and accuracy of the model. Summary of the Invention

[0004] In response to the problem of limited modeling capabilities of traditional neural network convolution, the present invention proposes an adaptive convolution module, which uses restricted deformable convolution to solve the problem. The present invention proposes a new deformable convolution, which is called restricted deformable convolution (LDConv). It has two ways to complement the modeling capabilities. The first is to add a learnable restriction unit to the activation sample offset, retain the offset capability, and enhance the constraint capability of irrelevant areas during modeling. The second is to increase the threshold modulation weight to limit the contribution of the offset sample. In the lightweight process, the feature correlation is reduced by channel shuffling, more representative features are extracted, and the sensitivity to irrelevant context is reduced.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0006] An optimization method for deformable convolution for enhanced geometric modeling, comprising the following steps:

[0007] S1. Collect target scene image data and perform data enhancement, label classification, and establish a paired dataset with classification labels.

[0008] S2. Establish a constrained deformable convolution LDConv module, and use the LDConv module to establish an LDBottleneck backbone network module to adapt to different application scenarios; further establish an LDC2f module based on the LDBottleneck backbone network module; improve the YOLOv8 detector based on the constrained deformable convolution LDConv module and the LDC2f module to obtain an improved deformable convolution detector with enhanced geometric modeling, perform target detection, and train and optimize the detector network;

[0009] S3. Collect image data, input the optimized and trained target detector network, automatically locate the target position prediction box and output the predicted classification label.

[0010] The label classification is to classify the target.

[0011] The Conv module in the YOLOv8 detector adopts the structure of a restricted deformable convolution LDConv module, and the C2f module adopts the structure of an LDC2f module.

[0012] The LDC2f structure in the backbone network includes a CBS module, a split operation, an n-way LDBottleneck module, a concat splicing, and a CBS module connected in sequence, which is used to extract features in the deep network.

[0013] The LDBottleneck structure includes three structures: lightweight restricted deformable convolution L, balanced restricted deformable convolution B, and ordinary restricted deformable convolution N.

[0014] The algorithm improvement of the restricted deformable convolution LDConv includes three stages:

[0015] Stage 1: Introducing a learnable constraint unit to restrict the sampling point position to a certain range. This unit is continuously updated with iterations to better adapt to different image features and scales. For offsets outside the range, its weight is set to a maximum value.

[0016] The formula is as follows:

[0017]

[0018] Among them, y(P0) represents the pixel value of position P0 in the output feature map, W(P n ) represents the weight of each position in the convolution kernel, X(P0+P n +ΔP n ) indicates that the input feature map is at position P0+P n +ΔP n The pixel value of R, where the size of R is 3x3, and the sampling position adjustment amount of the convolution kernel on the input feature map is ΔP n , Δm n is the sampling point weight coefficient, δ min ,δ max Represents the scope of restrictions that can be updated;

[0019] Stage 2: Determine δ min ,δ max Initial threshold size;

[0020] Stage 3: Lightweight LDConv by halving the number of channels.

[0021] The θ is:

[0022]

[0023] The present invention has the following beneficial effects and advantages:

[0024] First, this paper proposes a new restricted deformable convolution, or LDConv, which enhances modeling capabilities in neural networks, significantly improving upon DCNv1 and DCNv2. This innovative approach significantly improves existing convolution techniques and opens up new possibilities for improving the efficiency and accuracy of deep learning models.

[0025] Secondly, the present invention lightweights LDConv and designs three types of LDBottleneck to adapt to different application scenarios.

[0026] Finally, applying the restricted deformable convolution LDConv of the present invention to the target detector network can effectively improve the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 LDConv structure diagram of the present invention;

[0029] Figure 3 This is a working range diagram of the common convolution, deformable convolution v1, deformable convolution v2, and restricted deformable convolution of the present invention;

[0030] Figure 4 The LDBottleneck and LDC2f structure diagrams and the lightweight LDConv structure diagram of the present invention;

[0031] Figure 5(a) shows an example of the original YOLOv8 object detector network;

[0032] FIG5( b ) is a diagram of the improved network structure of the present invention; DETAILED DESCRIPTION

[0033] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, the specific implementation methods of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the invention. Therefore, the present invention is not limited to the specific implementation methods disclosed below.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of the art to which the present invention pertains. The terms used in the specification of the invention herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention.

[0035] The present invention proposes an adaptive convolution module to restrict deformable convolution to address the problem of limited convolution modeling capabilities of traditional neural networks. When processing geometric transformations, traditional convolutional neural networks are limited in their modeling capabilities of geometric transformations due to their fixed geometric structures. To overcome this problem, traditional deformable convolutions allow the convolution kernel to be deformable on the feature map. However, deformable convolutions may introduce irrelevant contextual information during the learning process, thereby affecting model performance. DCNv2 introduces a modulation mechanism to control the contribution of the offset through weights to control the diffusion of sampling points, but such problems still exist in actual use. Therefore, the present invention proposes a new restricted deformable convolution to address this problem, which enhances the modeling capability of the model by adding an adaptive restriction unit to constrain the offset, and adjusts the offset weight constraint to enhance the image focusing capability. Based on the restricted deformable convolution, the present invention has carried out lightweight work on it and designed three LDBottlenecks to adapt to different scenarios.

[0036] like Figure 1 As shown, it is a flow chart of the method of the present invention, comprising the following steps:

[0037] S1. Collect target scene image data and perform data enhancement, label classification, and establish a paired dataset with classification labels.

[0038] S2. Establish a constrained deformable convolution (LDConv) module, and use the LDConv module to establish an LDBottleneck backbone network module to adapt to different application scenarios; further establish an LDC2f module based on the LDBottleneck backbone network module; improve the YOLOv8 detector based on the constrained deformable convolution (LDConv) module and the LDC2f module to obtain an improved deformable convolution detector with enhanced geometric modeling, perform target detection using a paired dataset with classification labels, and train and optimize the detector network;

[0039] S3. Collect image data, input the optimized and trained target detector network, automatically locate the target position prediction box and output the predicted classification label.

[0040] Among them, label classification is to manually classify the target, label the image data with a category label, and create a paired data set for training. The output of the detection network of the present invention is the target position prediction box and the predicted classification label.

[0041] Figure 2This is the limited deformable convolution of the present invention, in which the input channel is expanded to 3 times after one convolution. Two copies are offsets, which are used to control the sampling position offset, and one copy is a mask to control the sampling point weight, that is, the contribution of the current sampling position to the entire neural network. The improvement is to add a limit offset +-2 to further limit the offset. It should be noted that the limit offset is updated with the iteration, which is specifically reflected in the θ function in the formula. The mask selection strategy is demonstrated below. Specifically, in the formula Δm n .

[0042] Its specific theories and functions are as follows:

[0043] Ordinary convolution operations fix the size and shape of the convolution kernel. This static feature remains unchanged throughout the training process, limiting the model's ability to perceive complex geometric structures. Due to the fixed size and shape of the convolution kernel, ordinary convolution has limitations in extracting features of different scales or multi-scales. The fixed nature of ordinary convolution may not be able to flexibly adapt to feature changes at different scales or positions. Therefore, when processing complex scenes or objects, the fixed size of the convolution kernel may not be able to fully capture key geometric features, limiting the model's ability to extract diverse features. In contrast, deformable convolution allows dynamic adjustment of the center position of the convolution kernel, improving the perception of complex geometric structures. This dynamic adjustment allows the model to better adapt to feature changes at different scales and positions. Assume that ordinary convolution is defined as follows:

[0044]

[0045] y(P0) represents the pixel value at position P0 in the output feature map, W(P n ) represents the weight of each position in the convolution kernel, X(P0+P n ) indicates that the input feature map is at position P0+P n Pixel value. Where R is the sliding window size, temporarily set at 3x3, then:

[0046] R={(-1,-1),(-1,0),…,(0,1),(1,1)

[0047] DCNv1 dynamically adjusts the sampling position adjustment amount ΔP of the convolution kernel on the input feature map n , the expression is as follows:

[0048]

[0049] However, the global feature extraction range of deformable convolution sometimes introduces some irrelevant or unnecessary sampling points, so DCNv2 increases the sampling point weight coefficient Δm n , then we have:

[0050]

[0051] Δm n Obtained by applying separate convolutional layers on the same input feature map x. These convolutional layers have the same spatial resolution and dilation rate as the current convolutional layer. The output of the convolutional layer is 3k channels, of which the first 2K channels correspond to the learned offsets, while the remaining k channels are further input to the Sigmoid layer to obtain the modulation scalar.

[0052] However, applying mask weight coefficients does not completely resolve the issue of irrelevant sample points. The sigmoid coefficients result in weights greater than 0, meaning that completely redundant features cannot contribute zero. Furthermore, the convolution kernels are relatively small, making it prone to slow network convergence due to incorrect sampling points. In the shallow layers of the network, feature channels are relatively shallow and process basic image features. These features are often local, small, or fundamental elements of the image, such as edges and texture. Therefore, for these simple features, a smaller receptive field and a smaller number of feature channels are more suitable. A smaller receptive field helps preserve local details and avoids excessive blurring or compression of image details. However, as the network becomes deeper, the feature channels gradually narrow. In deep networks, a larger receptive field becomes crucial to capture broader contextual information. This adjustment enables the network to understand the overall image structure and extract more abstract and high-level features. Deep networks learn more complex and global patterns through larger receptive fields, which is crucial for processing more complex scenes and abstract concepts.

[0053] To follow this concept, the present invention introduces a learnable restriction unit to restrict the sampling point position to a certain range, which is continuously updated with iterations to better adapt to different image features and scales. The formula is as follows:

[0054]

[0055] δ min ,δ max Represents the updateable limit range. For θ:

[0056]

[0057] For out-of-range offsets, it is unwise to set the eigenvalue to zero to avoid sensing signals at specific locations or positions. Since there is a large error between the sampling point and the actual position when the number of iterations is small in the early stage of training, it is assumed that there is a target sampling point boundary α, the sampling point position is denoted as β, and the limit boundary is denoted as γ. If β>α>γ exists, [DCNv2] chooses to gradually minimize m in the iterative process. n , but in fact when α>γ, γ is an effective feature in the image, so the initial maximization m is selectedn , after sigmoid, it approaches 1 infinitely. As β gradually approaches α during the iteration process, and when β>γ, γ is still a valid feature, m n It also gradually increases and approaches the initial value, omitting the gradual change process and setting the weight to the maximum. When β, α < γ, no additional operation is required. By learning m n The ideal result will be achieved gradually. The adjustment strategy aims to adjust the network training parameters when the number of iterations is small to more effectively optimize the network performance.

[0058] For the γ threshold, LDC replaces the 3rd and 5th layers of ordinary convolution, and the initial threshold is 5. During the adjustment process, δ min , δ max The trend of converging centers is shown, and a comparison of the parameter update amplitudes of shallow and deep networks reveals interesting characteristics. Shallow networks exhibit larger parameter update amplitudes relative to deep networks. This reflects that the shallow layers receive less information and therefore require larger adjustments to adapt to the needs of feature learning. In contrast, deep networks require a wider receptive field to cover more input information, so their parameter update amplitudes are smaller. We further limit γ to 3, which is the convolution kernel size. Shallow layers δ min , δ max The deep layers also show a central clustering trend, which is consistent with the above analysis. In order to retain more features, a conservative strategy is adopted, that is, the initial threshold is selected to be large, such as 5.

[0059] like Figure 3 The figure shows a comparison of the operating ranges of ordinary convolution, deformable convolution v1, deformable convolution v2, and restricted deformable convolution. The conclusion is that the restricted deformable convolution proposed in this invention has more variable and adaptable sampling positions, which is undoubtedly more adaptable to complex and changing real-world scenarios.

[0060] Figure 4 This is the LDBottleneck, LDC2f structure diagram and the lightweight LDConv structure diagram of the present invention.

[0061] The lightweighting process follows the following steps: The input feature map is split into channels, denoted as m1 and m2. A constrained deformable convolution is applied to m2 to control the complexity of the convolution. The processed m2 is then merged with m1, similar to a residual connection, to preserve some of the original features. Finally, the channels are recombined to reduce correlation between features, thereby obtaining a more representative feature representation.

[0062] The lightweight LDConv structure is an L-LDC, s = 1, k = 3 module, which is expanded to include: the outputs of the Harf-Split module and the LDConv module are connected in sequence, and after the contact operation with the output of the Harf-Split module (the number of channels is halved), they are connected to the shuffle module; the role of the L-LDC, s = 1, k = 3 module is lightweight.

[0063] The LDBottleneck structure consists of three branches: lightweight restricted deformable convolution (L), balanced restricted deformable convolution (B), and normal restricted deformable convolution (N). Each branch consists of two L-LDC (s=1, k=3) modules, whose output is added to the initial input in series for feature fusion. The LDBottleneck structure improves feature specificity and reduces the number of parameters.

[0064] The LDC2f structure expansion includes: CBS modules, split operations, n-way LDBottleneck modules, concat splicing, and CBS modules connected in sequence, which are used to extract features in deep networks.

[0065] Figure 5(a) is an example of the original YOLOv8 target detector network; Figure 5(b) is the improved network structure diagram of the present invention; after the improvement, any Conv module of the original YOLOv8 network can be selectively replaced with a constrained deformable convolution LDConv module, and the C2f module of the original YOLOv8 network can be selectively replaced with an LDC2f module.

[0066] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. An optimization method for deformable convolution for enhanced geometric modeling, characterized in that: The following steps are involved: S1. Collect target scene image data and perform data enhancement, label classification, and establish a paired dataset with classification labels. S2. Establish a constrained deformable convolution LDConv module, and use the LDConv module to establish an LDBottleneck backbone network module to adapt to different application scenarios; further establish an LDC2f module based on the LDBottleneck backbone network module; improve the YOLOv8 detector based on the constrained deformable convolution LDConv module and the LDC2f module to obtain an improved deformable convolution detector with enhanced geometric modeling, perform target detection, and train and optimize the detector network; S3. Collect image data, input the optimized and trained target detector network, automatically locate the target position prediction box and output the predicted classification label.

2. The method for optimizing deformable convolution for enhanced geometric modeling according to claim 1, wherein: The label classification is to classify the target.

3. The method for optimizing deformable convolution for enhanced geometric modeling according to claim 1, wherein: The Conv module in the YOLOv8 detector adopts the structure of a restricted deformable convolution LDConv module, and the C2f module adopts the structure of an LDC2f module.

4. The method for optimizing deformable convolution for enhanced geometric modeling according to claim 1 or 3, wherein: The LDC2f structure in the backbone network includes a CBS module, a split operation, an n-way LDBottleneck module, a concat splicing, and a CBS module connected in sequence, which is used to extract features in the deep network.

5. The method for optimizing deformable convolution for enhancing geometric modeling according to claim 1 or 3, characterized in that: The LDBottleneck structure includes three structures: lightweight restricted deformable convolution L, balanced restricted deformable convolution B, and ordinary restricted deformable convolution N.

6. The method for optimizing deformable convolution for enhanced geometric modeling according to claim 1 or 3, wherein: The algorithm improvement of the restricted deformable convolution LDConv includes three stages: Stage 1: Introducing a learnable constraint unit to restrict the sampling point position to a certain range. This unit is continuously updated with iterations to better adapt to different image features and scales. For offsets outside the range, its weight is set to a maximum value. The formula is as follows: Among them, y(P0) represents the pixel value of position P0 in the output feature map, W(P n ) represents the weight of each position in the convolution kernel, X(P0+P n +ΔP n ) indicates that the input feature map is at position P0+P n +ΔP n The pixel value of R, where the size of R is 3x3, and the sampling position adjustment amount of the convolution kernel on the input feature map is ΔP n , Δm n is the sampling point weight coefficient, δ min ,δ max Represents the scope of restrictions that can be updated; Stage 2: Determine δ min ,δ max Initial threshold size; Stage 3: Lightweight LDConv by halving the number of channels.

7. The method for optimizing deformable convolution for enhanced geometric modeling according to claim 6, wherein: The θ is:

Images