A hollow object detection method based on gradient image fusion

Abstract

The present invention discloses a hollow target detection method that fuses gradient maps. The method comprises the following steps: preparing a hollow object data set, processing an original image to obtain a gradient map; simultaneously feeding the original image and the gradient map into two target detection networks to obtain original image features and corresponding gradient map features; fusing the original image features and the gradient map features through a lightweight gradient attention module, and further forward-propagating the fused features through the network to ultimately obtain the target's bounding box information and category information; continuously iterating to obtain an optimal target detection model; inputting the image to be identified into the optimal target detection model, and calculating and outputting a classification result with a confidence level greater than a threshold as the recognition result of the image to be detected. The present invention uses gradient information through the gradient attention module to enhance the hollow object features in the original image, enabling the network to learn more meaningful features. At the same time, the present invention can be easily extended to various detection frameworks, thereby improving the detection of almost all hollow targets.

Description

Technical Field

[0001] The present invention relates to computer vision processing technology, and in particular to a hollowed-out target detection method integrating gradient maps. Background Art

[0002] The goal of object detection is to predict the bounding box and category label for each object of interest. With the national trend toward digital economy and intelligent industries, investment in the intelligent sector is increasing. Smart city, smart healthcare, smart education, and other smart development initiatives are underway, and object detection technology is essential in almost all of these scenarios. Currently, object detection technology is widely used in real life.

[0003] Due to the rapid development of deep neural networks in recent years, neural network-based object detection methods have made great progress compared to traditional hand-designed methods. Generally speaking, deep learning-based object detection methods can be divided into two types, namely two-stage methods and single-stage methods. For example, Faster R-CNN and Cascade R-CNN are two-stage detection frameworks, also known as candidate region-based object detection algorithms. They first select candidate regions for the input image and then classify and regress the candidate regions to achieve the detection task. YOLO, SSD, and FCOS are single-stage detection frameworks that omit the candidate region generation step and integrate the feature extraction, object classification, and position regression processes into a convolutional neural network, thereby simplifying the object detection process into an end-to-end regression problem.

[0004] In the development of object detection, many researchers have proposed different frameworks to solve detection problems in various practical scenarios. For example, SNIP, SNIPER, and TridentNet solve the multi-scale problem in object detection, FA-SSD is used to improve the performance of small object detection, and BorderDet is designed for dense object detection. However, there is currently no method specifically designed for hollow object detection scenarios. Hollow objects have very special properties. The proportion of pixels within the bounding box that actually belong to the object is very small, and most of the pixels are messy and useless background information, which has a significant negative impact on the detection of these objects. Therefore, these hollow objects are often difficult to define during the detection process, resulting in multiple dense bounding boxes for a single object.

[0005] For example, patent CN112001400A discloses a method for detecting hollow objects. This prior art solution simplifies hollow object detection by designing candidate boxes, but relies on manually designed parameters and lacks robustness. Another example is patent CN108549089B, which discloses a hollow obstacle detection device and method for SLAM. This prior art solution uses lidar and depth cameras for detection, which is highly dependent on equipment and is not conducive to widespread application in real-world scenarios. Summary of the Invention

[0006] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and provide a hollow target detection method that fuses gradient maps. By effectively fusing the gradient map and the original image, the problem of poor hollow target detection effect in the prior art is solved, thereby improving the detection effect of hollow objects and improving the robustness and generalization performance of the hollow object detection network.

[0007] Technical solution: The present invention provides a hollow target detection method by integrating gradient images, comprising the following steps:

[0008] Step 1: Create a hollow object dataset

[0009] Collect images containing corresponding hollowed-out targets to form a hollowed-out object dataset. Use the labelImg annotation tool to annotate the bounding boxes and category information of the targets to be detected in all collected images, and generate corresponding label files with the same name as the image.

[0010] Step 2: Process the unlabeled original images in the hollow object dataset to obtain the corresponding gradient map;

[0011] Step 3: The original image and the corresponding gradient map are simultaneously fed into two target detection networks with the same structure. Features are extracted through the first few stages of the backbone network in the target detection network to obtain the original image features and the corresponding gradient map features respectively.

[0012] Step 4: The original image features and gradient image features obtained in step 3 are fused through the lightweight gradient attention module to obtain fused features. The lightweight gradient attention module first performs a convolution operation on the gradient image features, and the number of output channels of the convolution is the same as the input. Then, the feature map of the convolution output is fed into the sigmoid function. At the same time, the result of multiplying the original image features and the gradient image features is convolved, and then the output result is passed through the normalization layer and the nonlinear activation layer and the residual connection is performed on the output result. Finally, the result of the residual connection, the original image features and the result of the sigmoid function are multiplied to complete the feature fusion of the original image and the gradient image.

[0013] Step 5: The fused features obtained in step 4 are fed into the subsequent structure of the target detection network in step 3 to achieve network forward propagation and finally obtain the target's bounding box information and category information; then, a confidence threshold is set to perform non-maximum suppression, and the union of bounding boxes with confidence higher than the threshold is taken as the final prediction result; finally, the error between the predicted bounding box and the true value of the labeled bounding box is calculated through the loss function, and the error is back-propagated to update the network parameters;

[0014] Repeat the above steps and iterate continuously to reduce the loss function to a stable state, obtain the optimal network parameters, obtain the optimal target detection model, and improve the detection effect of hollow objects;

[0015] Step 6: Input the image to be identified into the optimal target detection model trained in step 5, and calculate the classification result with an output confidence greater than the threshold as the recognition result of the image to be detected.

[0016] Furthermore, in step 1, the images in the hollow object data set are randomly divided into a training set and a validation set according to a certain ratio; data augmentation is performed on the training set as needed to expand the data volume, and the data augmentation method includes flipping.

[0017] Furthermore, the step 2 uses the Sobel operator to calculate the gradient map G. The specific method is: the Sobel operator performs convolution based on the image space domain and calculates it according to the weighted difference of the grayscale values of the four neighboring points above, below, left and right of the pixel point. The formula is as follows:

[0018]

[0019]

[0020] G=|G x |+|G y |

[0021] Among them, I is the original image, G x and G y Represents the image grayscale values of horizontal and vertical edges detected respectively.

[0022] Furthermore, the two target detection networks in step 3 have exactly the same structure and share parameters. In the backbone network of the target detection network structure, there is a convolution layer with a stride of 2 after every few convolution blocks. The feature maps are downsampled to half the resolution. The feature extraction process is as follows:

[0023] F I =ReLU(BN(Conv(…ReLU(BN(Conv(I)))…)))

[0024] F G=ReLU(BN(Conv(…ReLU(BN(Conv(G)))…)))

[0025] Among them, F I and F G They represent the original image and gradient map features respectively, ReLU is the nonlinear activation layer, BN is the normalization layer, Conv is the convolution layer, and the ellipsis represents the specific number of convolution blocks, which can be dynamically adjusted according to the detection network used.

[0026] Furthermore, the target detection network adopts the YOLOv5 detection network, and performs feature extraction through the first 6 stages of its backbone network, with a convolution layer with a stride of 2 after every 2 stages.

[0027] Furthermore, the subsequent structure of the target detection network in step 5 includes the second half of the backbone network, the FPN structure and the final detection head. The bounding box information and category information of the target obtained by the network forward propagation include the center coordinates, width and height of the bounding box, the confidence that the bounding box contains the target object, and the probability that the target object in the bounding box belongs to a certain category.

[0028] Beneficial Effects: This invention uses gradient information through a gradient attention module to enhance the features of hollowed-out objects in the original image, enabling the network to learn more meaningful features. Furthermore, this invention can be easily generalized to various detection frameworks, improving the detection of nearly all hollowed-out objects. Extensive experiments have validated this conclusion, improving the detection accuracy of hollowed-out objects by 7.4% mAP on YOLOv5, 1.7% mAP on Faster R-CNN, and 3.2% mAP on Cascade R-CNN. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall flow chart of the present invention;

[0030] Figure 2 Schematic diagram of the target detection network structure of the present invention;

[0031] Figure 3 is a schematic diagram of a lightweight gradient attention module in one embodiment of the present invention;

[0032] Figure 4 It is a diagram of the experimental results shown in the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0034] like Figure 1As shown, the present invention provides a hollow target detection method by integrating gradient images, comprising the following steps:

[0035] Step 1: Create a hollow object dataset

[0036] Collect images containing corresponding hollowed-out targets to form a hollowed-out object dataset, use annotation tools to annotate the bounding boxes and category information of the targets to be detected in the images, and generate corresponding label files with the same name as the images;

[0037] Step 2: Process the unlabeled original images in the hollow object dataset to obtain the corresponding gradient map;

[0038] Step 3: The original image and the corresponding gradient map are simultaneously fed into two target detection networks with the same structure to obtain the original image features and the corresponding gradient map features respectively;

[0039] Step 4: The original image features and gradient image features obtained in step 3 are fused through a lightweight gradient attention module to obtain fused features.

[0040] Step 5: The fusion features obtained in step 4 are fed into the subsequent structure of the target detection network in step 3 to train and obtain the optimal network parameters, thereby obtaining the optimal target detection model and improving the detection effect of hollow objects.

[0041] Step 6: Input the image to be identified into the optimal target detection model trained in step 5, and calculate the classification result with an output confidence greater than the threshold as the recognition result of the image to be detected.

[0042] Example 1

[0043] This embodiment specifically includes the following steps:

[0044] Step 1: Create a hollow object dataset.

[0045] like Figure 4 As shown, this example collects 3131 images containing corresponding hollowed-out objects. These images are then randomly divided into training and validation sets according to a certain ratio (85% and 15%). Annotation tools are then used to annotate the objects to be detected in the images with bounding boxes and category information. There are seven categories: safety belts, poles, workers working at heights, cross fences, square fences, ladders, and zebra crossings. A corresponding XML tag file with the same name as the image is generated. The training set is then randomly flipped to increase the data volume.

[0046] Step 2: Obtain the corresponding gradient map of the sample in step 1 through traditional algorithm processing. The overall network structure is as follows Figure 2 shown.

[0047] This embodiment uses the Sobel operator to calculate the gradient map G. The Sobel operator is essentially based on image spatial domain convolution and is obtained based on the weighted difference in the grayscale values of the four neighboring points above, below, left, and right of the pixel. The calculation formula is as follows:

[0048]

[0049]

[0050] G=|G x |+|G y |

[0051] Among them, I is the original image, G x and G y Represents the image grayscale values of horizontal and vertical edges detected respectively.

[0052] Step 3: The original image and the gradient map obtained in step 2 are simultaneously fed into two feature extraction networks with the same structure.

[0053] like Figure 2 As shown, this example uses the first six stages of the YOLOv5 backbone network for feature extraction. After every two stages, there is a convolutional layer with a stride of 2, and the feature maps are downsampled to half the resolution. The input image size is 1600×1600, and the resulting feature map resolution is 1 / 8 the original image. To save costs, the feature extraction network structure used for the original image and the gradient map is the same, and parameters are shared. The feature extraction process is as follows:

[0054] F I =ReLU(BN(Conv(…ReLU(BN(Conv(I)))…)))

[0055] F G =ReLU(BN(Conv(…ReLU(BN(Conv(G)))…)))

[0056] Among them, F I and F G They represent the original image and gradient image features respectively, ReLU is a nonlinear activation layer, BN is a normalization layer, Conv is a convolution layer, and the ellipsis represents the specific number of convolution blocks, which can be dynamically adjusted according to the detection network used. In the embodiment of the present application, there are 6 of them.

[0057] Step 4: Effectively fuse the original image feature map and the gradient image feature map obtained in step 3. In order to better utilize the gradient information, this embodiment involves a lightweight gradient attention module to perform some multiplication operations on the two types of feature maps, such as Figure 3 shown.

[0058] First, two 1×1 convolutions are performed on the gradient feature map. The output channels of both convolutions are the same as the input, 256. The resulting convolutional output feature map is then fed into a sigmoid function to ensure the output is between 0 and 1. Simultaneously, the result of multiplying the original image feature map with the gradient image feature map is subjected to a 1×1 convolution. The result is then passed through a batch normalization layer and a nonlinear activation layer (ReLU), and a residual connection is performed on the output. Finally, the residual connection result, the original image feature map, and the result of the sigmoid function are multiplied together to complete the feature fusion of the original image and the gradient map.

[0059] Through step 4, the contour and edge information in the gradient image can effectively guide the learning of semantic information of the original image.

[0060] Step 5: First, the features obtained in step 4 are continued to be sent to the subsequent structure of the detection network used, including the second half of the backbone network, the FPN structure and the final detection head, to realize the forward propagation of the network, and finally obtain the bounding box information and category information of the target, including the center coordinates, width and height of the bounding box, the confidence that the bounding box contains the target object, and the probability that the target object in the bounding box belongs to a certain category. Then, the confidence threshold is set to 0.5, non-maximum suppression is performed, and the union of the bounding boxes with confidence higher than the threshold is used as the final prediction result. Finally, the error between the predicted bounding box and the true value of the marked bounding box is calculated by the loss function. The loss function of the embodiment of the present application is the sum of the bounding box regression loss, confidence loss and classification loss. Finally, the error is back-propagated to update the network parameters.

[0061] Repeat the above steps for 100 iterations to reduce the loss function and stabilize it, obtain the optimal network parameters, and improve the detection effect of hollow objects. During the iteration process, the learning rate is set to 0.001 and the training batch size is 4.

[0062] Step 6: Input the image to be identified into the target detection model trained in step 5, set the threshold to 0.6, and calculate the classification result with an output confidence greater than the threshold as the recognition result of the image to be detected. The final experimental results are as follows: Figure 4 As shown in the figure, the accuracy of hollow target detection is greatly improved.

[0063] Example 2:

[0064] This example adds a lightweight gradient attention module to multiple excellent detection frameworks, including the one-stage detection networks SSD, RetinaNet, YOLOv3, and YOLOv5, and the two-stage detection networks Faster R-CNN and Cascade R-CNN. The accuracy of the models before and after adding gradient attention is compared on the hollowed-out target dataset. The comparison results are shown in Table 1.

[0065] Table 1

[0066]

[0067] Through the above embodiments, it can be seen that the technical solution of the present invention has achieved an improvement in detection accuracy in almost all categories, proving the effectiveness of the present invention.

Claims

1. A hollowing target detection method using gradient image fusion, characterized by: The following steps are involved: Step 1: Create a hollow object dataset Collect images containing corresponding hollowed-out targets to form a hollowed-out object dataset. Use annotation tools to annotate the bounding boxes and category information of the targets to be detected in all collected images, and generate corresponding label files with the same name as the image. Step 2: Process the unlabeled original images in the hollow object dataset to obtain the corresponding gradient map; Step 3: The original image and the corresponding gradient map are simultaneously fed into two target detection networks with the same structure. Features are extracted through the first few stages of the backbone network in the target detection network to obtain the original image features and the corresponding gradient map features respectively. Step 4: The original image features and gradient image features obtained in step 3 are fused through the lightweight gradient attention module to obtain fused features. The lightweight gradient attention module first performs a convolution operation on the gradient image features, and the number of output channels of the convolution is the same as the input. Then, the feature map of the convolution output is fed into the sigmoid function. At the same time, the result of multiplying the original image features and the gradient image features is convolved, and then the output result is passed through the normalization layer and the nonlinear activation layer and the residual connection is performed on the output result. Finally, the result of the residual connection, the original image features and the result of the sigmoid function are multiplied to complete the feature fusion of the original image and the gradient image. Step 5: The fused features obtained in step 4 are fed into the subsequent structure of the target detection network in step 3 to achieve network forward propagation and finally obtain the target's bounding box information and category information; then, a confidence threshold is set to perform non-maximum suppression, and the union of bounding boxes with confidence higher than the threshold is taken as the final prediction result; finally, the error between the predicted bounding box and the true value of the labeled bounding box is calculated through the loss function, and the error is back-propagated to update the network parameters; Repeat the above steps and iterate continuously to reduce the loss function to a stable state, obtain the optimal network parameters, and obtain the optimal target detection model; Step 6: Input the image to be identified into the optimal target detection model trained in step 5, and calculate the classification result with an output confidence greater than the threshold as the recognition result of the image to be detected.

2. The hollowing target detection method based on gradient image fusion according to claim 1, wherein: The step 1 randomly divides the images in the hollow object dataset into a training set and a validation set according to a certain ratio; and performs data augmentation on the training set as needed to expand the data volume, and the data augmentation method includes flipping.

3. The hollowing target detection method based on gradient image fusion according to claim 1, wherein: The step 2 uses the Sobel operator to calculate the gradient map G. The specific method is: calculate it according to the weighted difference of the grayscale values of the four neighboring points above, below, left and right of the image pixel point. The formula is as follows: in, I It's the original picture. and Represents the image grayscale values of horizontal and vertical edges detected respectively.

4. The hollowing target detection method of fused gradient map according to claim 1, characterized in that: In step 3, the two target detection networks have exactly the same structure and share parameters. In the backbone network of the target detection network structure, there is a convolution layer with a stride of 2 after every few convolution blocks. The feature maps are downsampled to half the resolution. The feature extraction process is as follows: in, and They represent the original image and gradient map features respectively, ReLU is the nonlinear activation layer, BN is the normalization layer, and Conv is the convolution layer.

5. The hollowing target detection method of fused gradient map according to claim 4, characterized in that: The target detection network adopts the YOLOv5 detection network, and performs feature extraction through the first 6 stages of its backbone network. There is a convolution layer with a stride of 2 after every 2 stages.

6. The hollowing target detection method of fused gradient map according to claim 1, characterized in that: The subsequent structure of the target detection network in step 5 includes the second half of the backbone network, the FPN structure and the final detection head. The bounding box information and category information of the target obtained by the network forward propagation include the center coordinates, width and height of the bounding box, the confidence that the bounding box contains the target object, and the probability that the target object in the bounding box belongs to a certain category.

Images