A weighted deformable convolution based object detection method and device

Abstract

The application relates to a weighted deformable convolution target detection method and device, and the following optimizations are carried out on the basis of a classic method Faster R-CNN: a weighted deformable convolution and a global context module are introduced in a feature extraction network ResNet-101; in a feature fusion network, a lightweight feature fusion network is used to replace a feature pyramid network to learn a multi-scale feature representation; a position loss function is introduced in a loss function, the position loss function comprises a width-height ratio loss function and an intersection-over-union loss function; in a post-processing process, a non-maximum suppression algorithm based on a center distance is adopted. Compared with the prior art, the application can effectively extract and enhance the boundary features of a target, the training speed is accelerated while the detection precision is ensured, the position and the width-height ratio of a prediction box are closer to those of a real box, and the problem of missing detection of overlapping targets is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method and apparatus for target detection based on weighted deformable convolution. Background Technology

[0002] The biggest challenge in object detection is dealing with the various geometric transformations of objects in images. Objects of the same category can exhibit different sizes, shapes, and even non-rigid deformations in images due to shooting angle, environment, and the characteristics of the objects themselves. How to adapt the model to these complex geometric transformations is a major challenge that object detection needs to address.

[0003] Convolutional neural networks have made significant progress in object detection. However, most of them use fixed-shape convolutional kernels such as 1×1 and 3×3, which are poorly adapted to targets with complex geometric structures and are prone to losing key features. The adaptability of existing models to geometric deformations of targets largely stems from the diversity of the dataset; the models themselves do not have the ability to handle geometric deformations. Therefore, for targets with geometric deformations outside the dataset, the detection performance of the models is poor and the generalization ability is insufficient. Especially for targets with diverse shapes, blurred boundaries, and large scale differences, classic deep learning-based object detection networks have limited ability to extract target features, are easily confused with surrounding features, have inaccurate bounding box regression positions, and are affected by environmental and target-specific characteristics, resulting in some targets being missed and causing low detection accuracy.

[0004] Chinese Patent Publication No. CN114519819A discloses a remote sensing image target detection method based on global context awareness. This method uses a deep residual network (ResNet 101) to extract image features, and then utilizes a feature pyramid network (FPN) to further extract features and generate candidate regions. After generating candidate regions, feature pooling is used to align the features. A global context extraction module is added to the highest layer of the feature extraction network, and the extracted features are fused with the original features through addition to obtain new features. Finally, a fully connected layer is used to classify the new features, generating target categories and bounding boxes. This method uses a deep residual network to extract image features, which has a fast detection speed, but it uses convolutional kernels of fixed shapes for feature extraction, making it prone to losing key features in scenarios with complex target structures. Furthermore, the method uses a feature pyramid network to learn multi-scale feature representations, which has good training accuracy, but its structure is relatively complex and the training time is long.

[0005] In summary, current remote sensing image target detection methods have the following drawbacks:

[0006] (1) In the feature extraction process, most of the convolution kernels with fixed shapes are used. They are not adaptable to targets with complex geometric structures and are prone to losing key features. For targets with other geometric deformations outside the dataset, the detection performance of the model is poor and the generalization ability is insufficient, especially for targets with diverse shapes, blurred boundaries and large scale differences.

[0007] (2) The loss function lacks design for the intersection-union ratio and aspect ratio of the prediction box;

[0008] (3) During the feature fusion process, the feature pyramid network structure is complex and the training time is long. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a detection method that introduces a weighted deformable convolutional network to adaptively adjust the sampling position according to image features, thereby improving the detection capability of targets with complex geometric structures.

[0010] The objective of this invention can be achieved through the following technical solutions:

[0011] One aspect of the present invention provides a target detection method based on weighted deformable convolution, comprising the following steps: acquiring an image to be detected; converting the image to be detected into a preset format to obtain a unified format image to be detected; inputting the unified format image to be detected into a pre-trained target detection model to obtain a target detection result, wherein the target detection model comprises: a feature extraction network, including a global context sub-network and a weighted deformable convolution sub-network, wherein the global context sub-network is used to acquire long-range dependency features based on the unified format image to be detected; the weighted deformable convolution sub-network calculates a two-dimensional offset based on the standard sampling point position of the image to be detected to obtain the offset sampling point position; and performs sampling based on the offset sampling point position and offset weight to acquire complex target features; and a feature fusion network, used to obtain a fused multi-scale feature pyramid based on the long-range dependency features and complex target features; and outputting a target detection result based on the fused multi-scale feature pyramid.

[0012] As a preferred technical solution, the method for obtaining the two-dimensional offset and offset weight is as follows: input the image to be detected in the uniform format into a separate pre-trained neural network to obtain the two-dimensional offset and offset weight.

[0013] As a preferred technical solution, the pre-trained neural network is trained using the gradient descent method.

[0014] As a preferred technical solution, the global context sub-network includes: a context modeling unit, used to obtain attention weights based on the unified format image to be detected; a feature transformation unit, used to obtain channel weights for each channel based on the unified format image to be detected and the attention weights; and a fusion unit, used to obtain the remote dependency features based on the unified format image to be detected and the channel weights.

[0015] As a preferred technical solution, the method for obtaining the channel weights specifically involves: obtaining the dependency relationships between channels in the image to be detected in the unified format through one or more bottleneck structures, and obtaining the channel weights based on the dependency relationships between the channels.

[0016] As a preferred technical solution, the method for obtaining the fused multi-scale feature pyramid includes the following steps: transforming the source feature map, including the remote dependency feature and the complex target feature, to obtain the transformed feature map; upsampling the transformed source feature map; adding the transformed feature maps pixel by pixel and taking the average value to obtain the fused feature map; and obtaining the fused multi-scale feature pyramid based on the fused feature map using methods including upsampling and downsampling.

[0017] As a preferred technical solution, the steps for obtaining the pre-trained target detection model include: obtaining a training sample set, wherein each sample in the training sample set includes an RGB sample image and a corresponding target annotation file; training the target detection model based on the training sample set, and obtaining the pre-trained target detection model after the loss function value reaches a preset convergence condition, wherein the loss function value is the loss function value between the predicted image of the RGB sample image of each sample and the annotation file of the target corresponding to the RGB sample image, and the loss function includes a positional loss function term, which is used to represent the degree of difference between the predicted box and the actual box; the smaller the degree of difference, the smaller the loss function value.

[0018] As a preferred technical solution, the location loss function includes an aspect ratio loss sub-function and an intersection-union ratio (IURR) loss sub-function. The IURR loss sub-function is used to measure the overlap between the predicted bounding box and the ground truth bounding box. The higher the overlap, the smaller the IURR loss sub-function value. The aspect ratio loss sub-function is used to measure the consistency between the predicted bounding box and the ground truth bounding box in length and width. The higher the consistency, the smaller the aspect ratio loss sub-function value.

[0019] As a preferred technical solution, after obtaining the target detection result, the final detection result is output after nonmaximum suppression processing.

[0020] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the weighted deformable convolution-based target detection method and device described above.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] (1) Weighted deformable convolution and global context network are introduced. Based on standard convolution, a two-dimensional offset and weight are learned by gradient descent to represent the weights of the x and y directions and the position of the sampling point after offset. The learned offset is added to the sampling points of the input feature map, and the regular sampling points are replaced by the offset sampling points, thus obtaining weighted deformable convolution. It can adaptively adjust the sampling position according to the image features. Weighted deformable convolution improves the feature extraction ability and generalization of complex geometric target. The global context module extracts the long-range dependency features of the image, so that the feature map focuses on the target of interest and enhances the feature representation of the target of interest.

[0023] (2) Based on the classification loss function and regression loss function of Faster RCNN network, the intersection-union ratio and aspect ratio of the predicted box are introduced as the location loss function. The intersection-union ratio measures the overlap between the predicted box and the real box to extract more accurate target boundaries. The aspect ratio is used to determine whether the predicted box and the real box are consistent in width and height. After introducing the location loss function, the position and aspect ratio of the predicted box are closer to the real box, thereby improving the accuracy of regression of solid waste targets with different shapes and blurred boundaries.

[0024] (3) In the feature fusion network, a lightweight feature fusion network is used to replace the feature pyramid network to learn multi-scale feature representations, which speeds up the training speed while ensuring detection accuracy.

[0025] (4) Based on the original non-maximum suppression algorithm, the center distance between the two detection boxes is introduced as the suppression index. The non-maximum suppression algorithm based on the center distance is considered to be two targets after calculation, and will not suppress one of the prediction boxes, thus effectively solving the problem of missed detection. Attached Figure Description

[0026] Figure 1 This is a flowchart of a remote sensing image target detection method based on global context awareness, according to an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of a weighted variable convolutional subnetwork according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the global context subnetwork according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of a lightweight fusion network according to an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] Example 1

[0032] This embodiment provides a weighted deformable convolutional target detection method and apparatus. Based on the classic target detection method Faster R-CNN, the feature extraction network, feature fusion network, loss function and post-processing are optimized respectively.

[0033] like Figure 1 The above is a flowchart of this embodiment. Taking remote sensing detection of solid waste as an example, in step S1, the image to be detected is input into a trained target detection model, which includes a feature extraction network and a feature fusion network. In step S2, the feature extraction network is used to extract features from the image to be detected. The feature extraction network includes a weighted deformable convolutional subnetwork and a global context subnetwork. The weighted deformable convolutional subnetwork is used to extract complex target features from the image to be detected, and the global context subnetwork is used to extract long-range dependency features from the image to be detected. In step S3, the features obtained in step S2 are fused using the feature fusion network. In step S4, the detection result is generated, and after post-processing in step S5, the final detection result is obtained.

[0034] like Figure 2 The diagram shows a weighted deformable convolutional subnetwork. Weighted deformable convolution adds a learnable two-dimensional offset Δp and weight coefficient w to each position of the regular grid RG of standard convolution. RF (p n The standard convolution is offset and then multiplied by a weight coefficient to indicate whether the offset sampling position is within the region of interest. This reduces the influence of irrelevant background information, increases the model's ability to focus on the region of interest, and improves the model's detection capability. Wherein, Δp n Let w be the two-dimensional offset. RF (p n ) represents the offset weight, p0 represents the sampling point of the input feature map, and p n This is a position within the regular grid RG of a standard convolution. Δp and w RF(p n The weighted deformable convolution is obtained by using a separate convolutional layer on the input feature map, with the same size as the input feature map, ultimately resulting in an offset map with 3N channels. The first 2N layers represent N two-dimensional offsets, and the 3Nth layer is input to a sigmoid layer to obtain the weight coefficients. The weighted deformable convolution and the standard convolution have the same input and output feature map size and number of channels, making it easy to apply in mainstream object detection networks.

[0035]

[0036] like Figure 3 The diagram illustrates the global context sub-network. The global context module consists of three parts: a context modeling unit, a feature transformation unit, and a fusion unit. The context modeling unit transforms the C×H×W features into HW×1 features using a 1×1 convolutional layer, and then calculates the attention weights of the feature map using a softmax function to obtain the importance of the query position on the feature map. The feature transformation unit captures the dependencies between channels using a bottleneck structure, calculating the importance of each channel in the feature map separately and assigning different weights to each channel based on its importance. Simultaneously, layer normalization (LN) is used to facilitate network optimization and improve the network's generalization ability. The feature fusion unit uses pixel-level addition to aggregate global context features to each position in the input feature map, ultimately outputting a feature map with global context information. The global context sub-network is combined with a weighted deformable convolutional sub-network and embedded into the feature extraction network to extract and enhance the boundary features of solid waste. The specific process is as follows: The number of channels of the output feature map of the weighted deformable convolution is converted to 256 layers through a 1×1 convolutional layer, and then a global context module is inserted. This allows the global context sub-network to be combined with the weighted deformable convolution sub-network and embedded into the feature extraction network to extract and enhance the boundary features of the target.

[0037] like Figure 4The diagram illustrates the feature fusion network. ResNet-101 network conv3_4, conv4_23, and conv5_3 are selected as source features for the fusion operation. A 1×1 convolutional layer is used to transform the source feature maps; then, bilinear interpolation is used to upsample the feature maps so that their size is the same as that of conv3_4, ensuring all features have the same spatial dimension; finally, the transformed feature maps are summed pixel-by-pixel, and the average is taken to obtain the fused feature map. After fusion, the feature pyramid network needs to be reconstructed using upsampling or downsampling to obtain multi-scale features. The specific operation process is as follows: a 3×3 convolutional kernel is applied to the fused feature map to enhance the features; then, the enhanced feature map is sequentially subjected to a 2x upsampling operation and two max-pooling operations with a stride of 2, resulting in feature maps P2, P4, and P5, ultimately yielding the fused multi-scale feature map {P2, P3, P4, P5}.

[0038] The training process of the pre-trained target detection model includes: creating a training sample set using solid waste data; training the target detection model based on the training sample set; and obtaining the pre-trained target detection model after the loss function value reaches the preset convergence condition.

[0039] During the creation of the training sample set, LabelImg was used to annotate the images with rectangular boxes to obtain the bounding boxes surrounding the targets, which were then saved as XML files. The XML files and images served as inputs for training the detection network.

[0040] The intersection-union ratio (IU) and aspect ratio (AUM) of the predicted bounding boxes are introduced into the loss function as optimization directions, resulting in IU and AUM loss functions. This ensures that training optimizes the results in a direction where the IU and AUM of the predicted and ground truth bounding boxes are more consistent. The IU loss function is the ratio of the intersection to the union of the predicted and ground truth bounding boxes. The AUM loss function is the normalized difference between the width and height of the predicted and ground truth bounding boxes. IU measures the overlap between the predicted and ground truth bounding boxes, and its definition is shown in the following equation: where P is the predicted bounding box, T is the ground truth bounding box, and L... IOU This is the crossover ratio loss function.

[0041] IOU=|P∩T| / |P∪T|

[0042] L IOU =1-IOU

[0043] Aspect ratio is used to determine whether the predicted bounding box and the ground truth bounding box are consistent in width and height. It is defined as follows: where w t and h t w represents the width and height of the ground truth bounding box of the target object, respectively. p and h pThese are the width and height of the prediction box, respectively, L AR This is the aspect ratio loss function.

[0044]

[0045] The intersection-to-union (IoU) loss function and the aspect ratio loss function constitute the position loss function, which is shown in the following formula.

[0046] L loc =L IOU +k·L AR

[0047] Ultimately, the loss function of the object detection network consists of three parts: classification loss, regression loss, and location loss, as defined in the following equation.

[0048]

[0049] This embodiment addresses the problem of low target detection accuracy caused by the characteristics of targets with irregular shapes, indistinct appearances, blurred boundaries, and large size differences. It proposes a detection network based on weighted deformable convolution and global context. By optimizing and combining multiple modules, the robustness of features is improved, the problem of missed detection of small and overlapping targets is alleviated, and thus the performance of target detection is improved.

[0050] Example 2

[0051] Compared to Example 1, this example, in addition to considering the intersection-union ratio of the two predicted boxes, also uses a non-maximum suppression method for center distance in the post-processing stage. The formula for calculating the center distance is shown below. Where, c(B,b) i ) represents the distance between the center points of the predicted box and the ground truth box, and d represents the diagonal distance between the minimum outer bounding boxes of the two detection boxes.

[0052]

[0053] The NMS (Non-maximum Suppression) formula based on center distance is shown below. (Explanatory variables)

[0054]

[0055] The specific operation of the nonmaximum suppression algorithm is as follows: Assume that the set of all target detection boxes is S; sort the boxes according to their scores, and take the highest-scoring target box B from list S and put it into set F; calculate the scores of target boxes b in S in turn. iIf the intersection-union ratio (IU) of the predicted bounding box (B) is greater than a threshold θ, the target bounding box is deleted; otherwise, it is added to set F. The predicted bounding boxes in S are then processed in steps ① through ④ until S is an empty set. Set F represents the final detection result. In addition to considering the IU between two predicted bounding boxes, the center distance between them is also introduced as a suppression factor. The center distance is the ratio of the Euclidean distance between the center points of the two detected bounding boxes to the diagonal distance between their smallest bounding boxes. When two predicted bounding boxes have high overlap but a large center distance, the NMS based on the center distance calculates that they are two separate targets and will not suppress one of the predicted bounding boxes, thus effectively solving the problem of missed detections.

[0056] Example 3

[0057] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing any of the weighted deformable convolution-based target detection methods and devices in Embodiments 1 and 2.

[0058] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A target detection method based on weighted deformable convolution, characterized in that, Includes the following steps: Acquire the image to be detected, convert the image to be detected into a preset format, and obtain an image to be detected in a uniform format; The uniformly formatted image to be detected is input into a pre-trained object detection model to obtain the object detection result. The target detection model includes: The feature extraction network includes a global context subnetwork and a weighted deformable convolutional subnetwork. The global context subnetwork is used to obtain remote dependency features based on the image to be detected in the unified format. The weighted deformable convolutional network calculates a two-dimensional offset based on the standard sampling point position of the image to be detected, obtains the offset sampling point position, and performs sampling based on the offset sampling point position and offset weight to obtain complex target features. A feature fusion network is used to obtain a fused multi-scale feature pyramid based on the aforementioned long-range dependency features and complex target features, and to output target detection results based on the fused multi-scale feature pyramid. The method for obtaining the fused multi-scale feature pyramid includes the following steps: The source feature map, including the aforementioned remote dependency features and complex target features, is transformed to obtain the transformed feature map. Upsample the transformed source feature map; The converted feature maps are summed pixel by pixel and the average value is taken to obtain the fused feature map; A multi-scale feature pyramid is obtained from the fused feature map by employing methods including upsampling and downsampling.

2. The target detection method based on weighted deformable convolution according to claim 1, characterized in that, The method for obtaining the two-dimensional offset and offset weight is as follows: input the image to be detected in the uniform format into a separate pre-trained neural network to obtain the two-dimensional offset and offset weight.

3. The target detection method based on weighted deformable convolution according to claim 2, characterized in that, The pre-trained neural network is trained using the gradient descent method.

4. The target detection method based on weighted deformable convolution according to claim 1, characterized in that, The global context sub-network includes: The context modeling unit is used to obtain attention weights based on the uniformly formatted image to be detected; The feature transformation unit is used to obtain the channel weights of each channel based on the uniformly formatted image to be detected and the attention weights. The fusion unit is used to obtain the remote dependency features based on the unified format image to be detected and the channel weights.

5. The target detection method based on weighted deformable convolution according to claim 4, characterized in that, The method for obtaining the channel weights is as follows: the dependency relationship between each channel in the image to be detected in the unified format is obtained through one or more bottleneck structures, and the channel weights are obtained based on the dependency relationship between each channel.

6. The target detection method based on weighted deformable convolution according to claim 1, characterized in that, The steps for obtaining the pre-trained object detection model include: Obtain a training sample set, wherein each sample in the training sample set includes an RGB sample image and a corresponding target annotation file; The target detection model is trained based on the training sample set. After the loss function value reaches a preset convergence condition, the pre-trained target detection model is obtained. The loss function value is the loss function value between the predicted image of the RGB sample image of each sample and the annotation file of the target corresponding to the RGB sample image. The loss function includes a position loss function term, which is used to represent the degree of difference between the predicted box and the actual box. The smaller the degree of difference, the smaller the loss function value.

7. The target detection method based on weighted deformable convolution according to claim 6, characterized in that, The location loss function includes an aspect ratio loss subfunction and an intersection-union ratio (IURR) loss subfunction. The IURR loss subfunction measures the overlap between the predicted bounding box and the ground truth bounding box; the higher the overlap, the smaller the IURR value. The aspect ratio loss subfunction measures the consistency between the predicted bounding box and the ground truth bounding box in length and width; the higher the consistency, the smaller the aspect ratio loss subfunction value.

8. The target detection method based on weighted deformable convolution according to claim 1, characterized in that, After obtaining the target detection results, the final detection results are output after nonmaximum suppression processing.

9. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the weighted deformable convolution-based target detection method and apparatus as described in any one of claims 1-8.

Images