Highway dangerous goods small target detection method and device based on convolutional neural network

Abstract

The method comprises the following steps: acquiring an image training data set with a highway dangerous small target detection frame labeled, enhancing the image training data set; adjusting the enhanced image training data set to an input picture size, inputting the image training data set into a backbone network, and acquiring a feature map output by the backbone network; performing feature processing on the feature map, integrating feature layers of different resolutions, applying an attention mechanism to construct a dependency relationship, balancing semantic and position information in non-adjacent layers, and finally obtaining a fused feature map; inputting the fused feature map into a decoupling prediction head to complete object classification and positioning; splicing the classification branch and the regression branch, stretching the connected feature map into one dimension, and then connecting to obtain a final feature map; calculating the loss and updating the network parameters through back propagation to complete the training of the network; and inputting the image to be detected into the trained network to obtain a detection result.

Description

Technical Field

[0001] This invention belongs to the field of deep learning image processing technology, and particularly relates to a method and device for detecting small hazardous targets on highways based on convolutional neural networks. Background Technology

[0002] Object detection is a core problem in machine vision, encompassing various visual tasks such as instance segmentation, object tracking, and action recognition. It has wide applications in fields such as autonomous driving, aerial image analysis, and face recognition. Existing object detection algorithms typically employ anchor-box-based methods, but this often leads to imbalanced positive and negative samples, which is particularly challenging for small object detection. Therefore, improving the accuracy of small object detection remains a significant research challenge.

[0003] Currently, object detection primarily employs two mainstream technical approaches: one-stage algorithms and two-stage algorithms. A representative example of two-stage algorithms is the Faster R-CNN series, which first filters a large number of candidate regions that may contain objects, and then detects these candidate regions. While this algorithm boasts high accuracy, it is relatively slow and unsuitable for real-time image detection. In contrast, one-stage algorithms such as the YOLO series can directly perform end-to-end object prediction, offering faster model detection speeds, but to some extent reducing the accuracy of object detection. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a method for detecting small hazardous targets on highways based on convolutional neural networks.

[0005] This invention introduces a Balanced FPN, integrating feature layers of different resolutions to balance semantic and positional information in non-adjacent layers, making the features more discriminative. Simultaneously, it employs a Task-Specific decoupled prediction head, utilizing task-specific encoding to address the imbalance between classification and regression tasks. This method improves the low accuracy of detecting small hazardous targets on highways.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A method for detecting small hazardous targets on highways based on convolutional neural networks includes the following steps:

[0008] Obtain an image training dataset with bounding boxes labeled with small targets that pose hazards on highways, and perform Mosaic data augmentation and MixUp data augmentation on the image training dataset;

[0009] The enhanced image training dataset was adjusted to the size of the input image and input into the backbone network CSPDarknet-53. The feature maps L2, L3, L4, and L5 of the dark2, dark3, dark4, and dark5 units in the backbone network CSPDarknet-53 were obtained.

[0010] Four feature maps of different sizes, L2, L3, L4, and L5, are input into the feature fusion network Balance FPN for feature processing. Feature layers of different resolutions are integrated, and an attention mechanism is applied to build dependencies, thereby balancing semantic and positional information in non-adjacent layers, and finally obtaining fused feature maps P2, P3, P4, P5, and P6.

[0011] The fused feature maps P2, P3, P4, P5, and P6 are fed into the Task-Specific decoupled prediction head. High-level semantic information features are used to classify objects, while low-level rich fine-grained information features are introduced to locate objects. The classification branch and the regression branch are concatenated along the channel portion, and the resulting feature map is stretched to one dimension. The stretched feature maps are then concatenated to obtain the final feature map. The loss is calculated, and backpropagation is performed to update the network parameters, completing the network training.

[0012] The image to be detected is input into the trained network to obtain the detection result.

[0013] The process of inputting feature maps L2, L3, L4, and L5 of four sizes into the Balance FPN feature fusion network for feature processing, ultimately obtaining fused feature maps P2, P3, P4, P5, and P6, includes:

[0014] The feature map L5 is directly input into the feature fusion network Balance FPN. First, it is processed from top to bottom, through a 1×1 convolution, and then upsampled to obtain L. 5_0 The feature map L4 is obtained by performing Concat fusion along the channels and then CSP convolution. 4_1 Continue with feature map L 4_1 After 1×1 convolution and upsampling, L is obtained. 4_0 The feature map L3 is obtained by performing Concat fusion along the channels and then CSP convolution. 3_1 Then, the feature map L 3_1 After 1×1 convolution and upsampling, L is obtained. 3_0 The feature map L is obtained by performing Concat fusion along the channels of feature map L2 and then CSP convolution. 2_1 ; the feature map L 2_1 The feature map C2 is directly output; then a bottom-up fusion is performed, where C2 is convolved with L by a 3×3 convolution. 3_0After fusing and convolution, feature map C3 is obtained; C3 is then convolved with L. 2_0 The feature map C4 is obtained by fusion; C4 is then convolved with L. 5_0 Feature maps C5 and C6 are obtained through fusion. To enhance feature recognition capabilities by enabling information interaction between non-adjacent layers, C3, C4, and C5 are used as inputs for feature integration. Considering that shallow background information can cause interference, C2 is discarded. In the feature integration stage, C4 is used as the baseline. C3 and C5 are downsampled and upsampled respectively to make them the same size as C4. After summing using SUM, long-distance dependencies between features are obtained through a non-local attention mechanism. Finally, C3, C4, and C5 are added back to obtain P3, P4, and P5. P2 and P6 are directly obtained from C2 and C6.

[0015] Specifically, P2, P3, P4, P5, and P6 are fed into the Task-Specific decoupled prediction head. In the classification branch, the features of the (i+1)th layer are upsampled and fused with the features of the ith layer along the channel. After passing through a 3×3 convolution and a fully connected layer, the result is obtained. In the regression branch, the features of layer (i+1) are upsampled. Then, the upsampled features of layer (i) are combined with the features of layer (i-1) and subjected to a SUM operation. After downsampling and dimensionality reduction, the upsampled features of layer (i) are combined with the features of layer (i) and the upsampled features of layer (i+1) and then subjected to two 3×3 convolutions to obtain the final result.

[0016] The calculated loss includes: classification loss, target score loss, and bounding box regression loss. The classification loss and target score loss are binary cross-entropy loss (BCELoss), and the bounding box regression loss is the effective cross-union ratio loss (EIOULoss).

[0017] A second aspect of the present invention relates to a small target detection device for hazardous objects on highways based on a convolutional neural network, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the small target detection method for hazardous objects on highways based on a convolutional neural network of the present invention.

[0018] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the convolutional neural network-based method for detecting small hazardous targets on highways according to the present invention.

[0019] The advantages of this invention are: it introduces a Balanced FPN, integrating feature layers of different resolutions to balance semantic and positional information in non-adjacent layers, making the features more discriminative. Simultaneously, it employs a Task-Specific decoupled prediction head, utilizing task-specific encoding to adjust the imbalance between classification and regression tasks. This method improves the low accuracy of detecting small hazardous targets on highways. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention.

[0021] Figure 2 This is a detailed design drawing of the Balance FPN of the present invention.

[0022] Figure 3 This is a detailed design diagram of the Task-Specific decoupling prediction head of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention.

[0024] Example 1

[0025] This invention relates to a method for detecting small hazardous targets on highways based on neural networks, which mainly includes the following steps: First, the images are subjected to data augmentation processing. Then, training begins in batches. For each batch, the images are processed through a convolutional neural network to obtain feature maps L2, L3, L4, and L5. Subsequently, the obtained feature maps are fused using a Balanced FPN to obtain P2, P3, P4, P5, and P6. These feature maps are then fed into a Task-Specific decoupled prediction head for classification and regression to obtain predicted values. The predicted values ​​are compared with the true values ​​of the images to calculate the loss. After each batch of training, backpropagation is performed to reduce the loss and update the network parameters to complete the network training.

[0026] In one embodiment, such as Figure 1 As shown, a method for detecting small hazardous targets on highways based on neural networks is proposed, including:

[0027] Step S1: Obtain an image training dataset with labeled human figure small target detection boxes, and perform Mosaic and MixUp data augmentation on the image training dataset.

[0028] This embodiment performs Mosaic and MixUp data augmentation on the training dataset. Mosaic data augmentation involves taking four images and stitching them together using random scaling, cropping, and arrangement. Its advantages include enriching the background and small targets of the detected objects, and calculating the data of all four images at once, requiring minimal overhead; a single GPU can achieve good results. MixUp data augmentation involves superimposing two images together, which reduces the memorization of incorrect labels, thus enhancing robustness.

[0029] Step S2: Adjust the enhanced image training dataset to the size of the input image and input it into the backbone network CSPDarknet-53. Obtain the feature maps L2, L3, L4, and L5 of four sizes output by the dark2, dark3, dark4, and dark5 units in the backbone network CSPDarknet-53.

[0030] This invention uses CSPDarknet-53 as the backbone network for feature extraction. The CSPDarknet-53 network is pre-trained with weights trained on COCO and is trained in batches with a batch size of 8 (i.e., 8 images are processed per batch). The learning rate starts from 0.0025 and is updated using cosine annealing instead of a learning rate warm-up method.

[0031] Since the original image is large, this application scales the original image to 640×640 proportionally according to the long side, and fills the part of the short side that is less than 640 with 0. The scaled image is input into the backbone network CSPDarknet-53. After a series of convolution and other operations, feature maps L2, L3, L4 and L5 of four sizes are output successively: 20×20, 40×40, 80×80 and 160×160.

[0032] like Figure 2 As shown, the size of the feature map is determined by the backbone network CSPDarknet-53, which will not be elaborated here. This embodiment uses the features output from dark2, dark3, dark4, and dark5 for multi-scale fusion operation, outputting feature maps of four sizes. This allows for the incorporation of shallow, fine-grained information, which is beneficial for small target detection and achieves better detection results.

[0033] Step S3: Input the feature maps L2, L3, L4, and L5 of four sizes into the feature fusion network Balance FPN for feature processing.

[0034] First, from top to bottom, the feature map L5 (the feature map output by dark5) is convolved with 1×1 and then upsampled to obtain L... 5_0The feature map L4 (the feature map output by dark4, and so on) is fused along the channels using Concat and then subjected to CSP convolution to obtain the feature map L. 4_1 Continue with feature map L 4_1 After 1×1 convolution and upsampling, L is obtained. 4_0 The feature map L3 is obtained by performing Concat fusion along the channels and then CSP convolution. 3_1 Then, the feature map L 3_1 After 1×1 convolution and upsampling, L is obtained. 3_0 The feature map L is obtained by performing Concat fusion along the channels of feature map L2 and then CSP convolution. 2_1 ; the feature map L 2_1 The feature map C2 is directly output; then a bottom-up fusion is performed, where C2 is convolved with L by a 3×3 convolution. 3_0 After fusing and convolution, feature map C3 is obtained; C3 is then convolved with L. 2_0 The feature map C4 is obtained by fusion; C4 is then convolved with L. 5_0 Feature maps C5 and C6 are obtained through fusion. To enhance feature recognition capabilities by enabling information interaction between non-adjacent layers, C3, C4, and C5 are used as inputs for feature integration. Considering that shallow background information can cause interference, C2 is discarded. During the feature integration stage, C4 is used as the baseline. C3 and C5 are downsampled and upsampled respectively to make them the same size as C4. After summing using SUM, long-distance dependencies between features are obtained through a non-local attention mechanism. Finally, C3, C4, and C5 are added back to obtain P3, P4, and P5. P2 and P6 are directly obtained from C2 and C6.

[0035] Specifically, the 20×20 feature map L5 is directly input into the top-down feature pyramid network Balance FPN. It is first convolved into the same channel by 1×1, and then upsampled to become a 40×40 feature map, resulting in L. 5_0 The feature map L is obtained by performing Concat fusion with the 40×40 feature map L4 along the channels and then performing CSP convolution. 4_1 (40×40). Continue with L 4_1 L is obtained through 1×1 convolution and upsampling. 4_0 After becoming an 80×80 feature map, it is fused with feature map L3 along the channels using Concat and then subjected to CSP convolution to obtain feature map L. 3_1 (80×80); Continue to L 3_1 L is obtained through 1×1 convolution and upsampling. 3_0After obtaining a 160×160 feature map, it is fused with feature map L2 along the channel using a Concat convolution followed by a CSP convolution to obtain feature map C2 (160×160). Next, a bottom-up fusion is performed, where C2 is convolved with L2 using a 1×1 convolution. 3_0 Concat fusion along the channels using CSP convolution yields feature map C3 (80×80). C3 is then convolved with L using a 1×1 convolution. 4_0 Concat fusion along the channels using CSP convolution yields feature map C4 (40×40). C4 is then convolved with L using a 1×1 convolution. 5_0 Concat convolutions along the channels yield feature maps C5 and C6 (20×20). To enhance feature recognition by enabling information exchange between non-adjacent layers, C3, C4, and C5 are used as input features. Specifically, using C4 as a baseline, C3 and C5 are downsampled and upsampled respectively to achieve a feature map size of 40×40, followed by a SUM operation. A non-local attention mechanism is then used to capture long-distance dependencies between features. Finally, C3, C4, and C5 are added back to obtain P3, P4, and P5, while P2 and P6 are directly derived from C2 and C6.

[0036] Step S4: Input the fused feature maps P2, P3, P4, P5, and P6 into the Task-Specific decoupled prediction head, use high-level semantic information features to complete object classification, and introduce low-level rich fine-grained information features to locate objects.

[0037] Specifically, in the classification branch, the features of layer i+1 are upsampled and fused with the features of layer i along the channel using Concat. This is then processed through a 3×3 convolution and a fully connected layer to obtain the final result. In the regression branch, the features of layer (i+1) are upsampled. Then, the upsampled features of layer (i) are combined with the features of layer (i-1) and subjected to a SUM operation. After downsampling and dimensionality reduction, the upsampled features of layer (i) are combined with the features of layer (i) and the upsampled features of layer (i+1) and then subjected to two 3×3 convolutions to obtain the final result. Finally, the classification branch and the regression branch are spliced ​​along the channel, and the resulting feature map is stretched into one dimension. The stretched feature map is then spliced ​​to obtain the final feature map. The loss is calculated and backpropagation is performed to update the network parameters, thus completing the training of the network.

[0038] In this embodiment, after convolution of the prediction head classification branch and the regression branch, the concatenation along the channel portion generates three new feature maps, each a tensor of size {W×H×[(cls+reg+obj)]×N}, where W×H is the feature map size, cls is the detection class, reg is the predicted bounding box, obj is the target score prediction, and N is the number of predicted anchor boxes. Then, W and H are multiplied to stretch the spatial dimension to one dimension, resulting in feature maps F1, F2, and F3. Finally, F1, F2, and F3 are concatenated along W*H to obtain the final feature map F.

[0039] Finally, the classification loss, target score loss, and bounding box regression loss are calculated, and backpropagation is performed to reduce the loss while updating the network parameters.

[0040] Specifically, after convolution of F1, F2, and F3 for classification and regression branches respectively, each feature map generates three new feature maps F. cls ∈{N×W×H×cls}、F obj ∈{N×W×H×1}、F reg For each feature map ∈ {N×W×H×4}, first connect along the channel portion, resulting in three new feature maps of size {N×W×H×[(cls+reg+obj)]}, where W and H ∈ {20, 40, 80}. Then multiply W and H to stretch the spatial dimension to one dimension, resulting in three tensors of size {N×(cls+reg+obj)×(W×H)}. Finally, connect F1, F2, and F3 along W*H to obtain the final feature map F ∈ {N×(cls+reg+obj)×8400}.

[0041] Where cls is the category in the dataset, reg predicts the bounding box including the predicted top left corner (x1, y1) and bottom right corner (x2, y2), and N is the preset number of anchor boxes, which is 1 in this embodiment.

[0042] In this embodiment, the prediction head uses a decoupled approach, performing convolution operations on the classification and regression branches separately, which achieves better detection results. Furthermore, the prediction at each location is reduced from 3 to 1 through a concatenation operation, and an anchor-free method is used to avoid the problem of imbalanced positive and negative samples.

[0043] Since the output feature values ​​cannot be directly used for loss calculation, regression is required to obtain the actual predicted values. Classification loss, bounding box loss, and target score loss are applied to the feature map F according to the following formulas: Classification loss and target score loss are BCELoss, and bounding box regression loss is EIOULoss. The specific formulas are as follows:

[0044] BCELoss=-(ylog(p(x))+(1-y)log(1-p(x)))(1)

[0045]

[0046] It should be noted that the grid setting in this invention is an abstract concept on the final feature map, and its purpose is to facilitate bounding box regression calculation. For feature maps of 20*20, 40*40, and 80*80, there are 20×20, 40×40, and 80×80 grids respectively. Dividing the feature map into multiple grids is a relatively mature technology in this field, and will not be elaborated here.

[0047] 1. Calculate the classification loss and target score loss using the binary cross-entropy loss function:

[0048] BCELoss=-(ylog(p(x))+(1-y)log(1-p(x))) (3)

[0049] Where y represents whether it is a target, with a value of 1 or 0, and p(x) is the predicted target score.

[0050] 2. Calculate the bounding box loss. Calculate the Intersection of Union (IOU) using the predicted bounding box information and the ground truth bounding box information calculated from the labels. IOU is the intersection-union ratio between the predicted and ground truth bounding boxes. Post-processing using NMS yields the predicted bounding box with the highest IOU value.

[0051]

[0052] Where G represents the ground truth bounding box and P represents the predicted bounding box, the intersection-union ratio (IU / I) of the ground truth and predicted bounding boxes is calculated. Simultaneously, the minimum bounding rectangle R and w of both are calculated. c and h c The center represents the width and height of the shape R. dis convex represents the distance between the centers of the ground truth bounding box and the predicted bounding box. dis The diagonal distance of R represents the distance along the edge. w and h represent the width and height of the predicted bounding box. gt and h gt Represents the width and height of the actual bounding box. ρ represents the Euclidean distance calculation.

[0053] It should be noted that the calculation of classification loss, target score loss and bounding box loss is a relatively mature technology in this field, and will not be elaborated here.

[0054] This yields the loss between the predicted and actual values. Before the end of each batch, backpropagation is performed to reduce the loss. Simultaneously, the network parameters are updated, and training begins for the next batch, continuing until all batches of training data have been completed. Finally, the trained weights are obtained, and all updated parameters are saved in the Outputs weights file.

[0055] Step S5: Input the image to be detected into the trained network to obtain the detection result.

[0056] This invention scales the image to be detected to 640×640 and inputs it into the network. The CSPDarknet-53 backbone network outputs feature maps of four sizes. After regression of the feature values, the predicted values ​​are obtained, including the category cls, the target score obj, and the bounding box regression reg, resulting in a final prediction result.

[0057] Example 2

[0058] This embodiment relates to a small target detection device for hazardous objects on highways based on convolutional neural networks, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the small target detection method for hazardous objects on highways based on convolutional neural networks of Embodiment 1.

[0059] Example 3

[0060] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the method for detecting small hazardous targets on highways based on convolutional neural networks as described in Embodiment 1.

[0061] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting small hazardous targets on highways based on convolutional neural networks, characterized in that, The method for detecting small hazardous targets on highways based on convolutional neural networks includes: Obtain an image training dataset with bounding boxes labeled with small targets that pose hazards on highways, and perform Mosaic data augmentation and MixUp data augmentation on the image training dataset; The enhanced image training dataset was resized to the input image size and fed into the backbone network CSPDarknet-53. Feature maps of four sizes were obtained from the outputs of the dark2, dark3, dark4, and dark5 units in the backbone network CSPDarknet-53. ; Four different sized feature maps The input is fed into the Balance FPN feature fusion network for feature processing, integrating feature layers of different resolutions and applying an attention mechanism to build dependencies, thereby balancing semantic and positional information in non-adjacent layers, and finally obtaining the fused feature map. ; The fused feature map The Task-Specific decoupled prediction head is input, and high-level semantic information features are used to classify objects. At the same time, low-level rich fine-grained information features are introduced to locate objects. The classification branch and the regression branch are concatenated along the channel part, and the connected feature map is stretched into one dimension. The stretched feature map is then concatenated to obtain the final feature map. The loss is calculated and backpropagation is performed to update the network parameters, thus completing the training of the network. The image to be detected is input into the trained network to obtain the detection result; The four different sized feature maps The input is fed into the Balance FPN feature fusion network for feature processing, and finally the fused feature map is obtained. ,include: feature map The input is directly fed into the Balance FPN feature fusion network. First, it is processed from top to bottom through 1×1 convolutions, and then upsampled to obtain the desired result. , and feature map Concat fusion along the channels using CSP convolutions yields feature maps. Continue with the feature map After 1×1 convolution and upsampling, we get , and feature map Concat fusion along the channels using CSP convolutions yields feature maps. Then the feature map After 1×1 convolution and upsampling, we obtain , and feature map Concat fusion along the channels using CSP convolutions yields feature maps. ; feature map The feature map is obtained directly from the output. Then, perform bottom-up integration, After 3×3 convolution and The feature map is obtained by fusing and convolution. ;Will After 3×3 convolution and Feature map obtained by fusion ;Will After 3×3 convolution and Feature map obtained by fusion To achieve information exchange between non-adjacent layers and enhance feature recognition capabilities, As input for feature integration; considering that shallow background information would cause interference, it was abandoned. In the feature integration stage, As a benchmark, respectively for and Perform downsampling and upsampling operations to make it the same as For features of the same size, after summing using SUM, long-distance dependencies between features are obtained through a non-local attention mechanism; finally, they are added back together. Go up and get , Directly by get.

2. The method for detecting small hazardous targets on highways based on convolutional neural networks as described in claim 1, characterized in that, The The Task-Specific decoupled prediction head is passed in. In the classification branch, the features of layer i+1 are upsampled and fused with the features of layer i along the channel. After passing through a 3×3 convolution and a fully connected layer, the result is obtained. In the regression branch, the features of layer (i+1) are upsampled, and then the upsampled features of layer (i) are summed with those of layer (i-1) using a SUM operation. After downsampling to reduce dimensionality, the upsampled features of layer (i) are summed with the upsampled features of layer (i+1) using a SUM operation. Finally, the result is obtained after two 3×3 convolutions. .

3. The method for detecting small hazardous targets on highways based on convolutional neural networks as described in claim 1, characterized in that, The calculated losses include: classification loss, target score loss, and bounding box regression loss, where classification loss and target score loss are... The bounding box regression loss uses an effective intersection-union loss. .

4. A small target detection device for hazardous objects on highways based on convolutional neural networks, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the method for detecting small hazardous targets on highways based on convolutional neural networks as described in any one of claims 1-3.

5. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the method for detecting small hazardous targets on highways based on convolutional neural networks as described in any one of claims 1-3.

Images