A deep learning video detection method for targets with limited viewing angle in tunnels

Abstract

A deep learning video detection method for target detection in tunnels with limited viewing angles includes the following steps: Step 1: Constructing a library of annotated and labeled images as a training sample set for a network model; Step 2: Constructing a network model: Building a detection network based on an SSD network, using VGG-16 with a Mish activation function and a batch normalization layer as the backbone of the detection network, using bidirectional multi-scale feature fusion at the jump layer as the neck of the detection network, and using the SSD network head as the detection network head; Step 3: Training the network model, using adaptive online hard-to-distinguish sample discovery based on target size to obtain a detection model; Step 4: Inputting a test sample, and using the trained detection model to predict the target location and category in the tunnel image. This method is suitable for target detection in tunnels and other environments with limited viewing angles, extremely small targets, and environmental interference, and exhibits excellent detection performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent video detection, relates to the recognition and positioning of targets in videos, and is a deep learning video detection method for targets with limited viewing angles in tunnels. Background Art

[0002] Tunnels are rapidly developing due to their superior capabilities, such as crossing rivers, through mountains, and across lakes, while avoiding ground-based construction. However, the airtight nature of tunnels makes it easy for traffic accidents to lead to secondary accidents and other serious consequences. Tunnels present a high risk of accidents, and relying solely on manual monitoring of vehicle operations within tunnels is no longer sufficient to meet the increasing pressure of traffic. Tunnel situation monitoring and abnormal event warnings based on video analysis have become a new trend. Although AI-based deep learning video detection methods have been widely used in the transportation sector, tunnel cameras are generally installed at an oblique angle at a height of 4.5 meters. Distant vehicles are small in size and difficult to extract features from videos. Furthermore, the complex scenes within tunnels, such as severe interference from headlight glare and severe occlusion when vehicles are densely packed, make it difficult for existing video target detection methods to accurately detect vehicle targets in real time, severely restricting the development of intelligent tunnel monitoring systems. Summary of the Invention

[0003] The problem this invention aims to solve is that tunnel camera height restrictions and limited viewing angles result in distant vehicles appearing extremely small in the video. These objects, in a 1920x1080 resolution, are only about 40x30 pixels in size, occupying approximately 0.058% of the entire image. Extracting these small objects is a major challenge in applying deep learning methods to video detection in tunnels. Furthermore, video object detection in tunnels is hampered by environmental factors such as glare and severe occlusion.

[0004] The technical solution of the present invention is: a deep learning video detection method for targets with limited viewing angles in tunnels, comprising the following steps:

[0005] Step 1: Build a training dataset: Collect video images from a high-definition camera in the tunnel and build a library of images with annotations and labels as training samples for the network model. The annotations and labels are the target boxes of the targets.

[0006] Step 2: Build the network model: Build a detection network based on the SSD network, use VGG-16 with Mish activation function and BN layer as the backbone of the detection network, use jump-layer bidirectional multi-scale feature fusion as the neck of the detection network, and use the head of the SSD network as the head of the detection network;

[0007] Step 3: Use the training set in step 1 to train the network model in step 2. During the training process, adaptive online hard-to-distinguish samples based on target size are used to train the detection model.

[0008] Step 4: Input the test sample and use the trained detection model to predict the target location and category in the tunnel image.

[0009] The present invention is suitable for extracting small moving targets caused by the limited viewing angle in a tunnel environment, and for identifying and locating targets such as vehicles, pedestrians, and non-motor vehicles in the tunnel. The detection results can be used as a basic detection module, and the detection results can be subsequently called to obtain traffic situation parameters and abnormal event warning information.

[0010] Compared with the prior art, the present invention has the following advantages:

[0011] First, the detection method of the present invention can detect extremely small targets in tunnel scenarios, occupying approximately 0.058% of the entire video screen area. In tunnel scenarios, the existing SSD algorithm, which uses the same backbone as the present invention, has a detection accuracy of 93.2% and a detection rate of only 36.24%, failing to detect most small targets. The detection accuracy of the present invention is 98.89% and the detection rate is 95.21%, which can detect the vast majority of small targets and significantly improve the detection rate.

[0012] Second, the network inference speed of the present invention is fast, meeting the real-time requirements. Currently, the commonly used video detection algorithms FasterR-CNN and Cascade R-CNN have detection accuracy rates of over 95%, but their inference speeds are 13FPS and 5FPS respectively, which cannot meet the real-time requirements. The present invention has an inference speed of 30FPS under the conditions of a detection accuracy of 98.89% and a detection rate of 95.21%, meeting the real-time requirements.

[0013] Third, the present invention proposes an adaptive online difficult-to-distinguish sample mining method based on target size, so that the model can better learn the characteristics of small targets. The existing difficult-to-distinguish sample mining method is to sort the pre-selected boxes in training according to the loss, and send the positive and negative samples into the calculation back propagation at a ratio of 1:3. However, since the loss of large targets is greater than that of small targets, more large targets are sent to the network to update parameters, so that the network has a good recognition effect on large targets and a weak recognition effect on small targets. The present invention calculates the average area of ​​the pre-selected boxes for each batch Not artificially but through Adaptively divide all pre-selected boxes into large objects and small objects, and calculate the loss of each pre-selected box according to its area and The ratio of is scaled to eliminate the effect of target size on the loss. Furthermore, after sorting the scaled losses, a certain number of large-target negative samples and small-target negative samples are retained as difficult samples and kept until the next batch of training. This method improves the model's accuracy by 5.23% and its detection rate by 23.71%.

[0014] Fourth, the present invention further proposes a new loss function SIoU. The commonly used loss functions in existing video detection technologies are DIoU or CIoU, both of which require calculating the diagonal distance between the pre-selected box and the minimum enclosing box of the real box. This distance is affected by the size of the target, resulting in a greater loss for larger targets, making it easier for the network to learn large targets and ignore small targets. The SIoU proposed in the present invention calculates the ratio of the distance between the horizontal and vertical coordinates of the center points of the two boxes to the width and height of the minimum enclosing box, and then calculates the sum of their squares. The ratio is used to reduce the impact of the target size on the loss, so that the network can better learn the features of small targets and improve the detection performance of small targets. This loss function enables the model to improve the accuracy by 3.23% and the detection rate by 15.71%;

[0015] Fifth, the present invention proposes a bidirectional multi-scale feature fusion method for leap layers. Figure 2 Compared with the existing FPN, this method adopts a jump-layer approach, which reduces the amount of network computation and the impact of manual interpolation methods on training parameters. More parameters are completely determined by the network. On the other hand, it adopts bidirectional fusion, which, compared with the unidirectional FPN, enhances the detection performance of small targets while not significantly weakening the detection performance of large targets.

[0016] Sixth, the method of the present invention can also be transplanted to scenarios with extremely small targets, such as drones, and has great potential. When drones fly at high altitudes, ground targets are extremely small, similar to the scene in a tunnel, and the present invention can be used to detect extremely small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the process of the present invention.

[0018] Figure 2 This is a network structure diagram of the deep learning network of the method of the present invention.

[0019] Figure 3 This is an example tunnel video detection effect diagram of the present invention. DETAILED DESCRIPTION

[0020] Based on the established road monitoring network within tunnels, this paper proposes a deep learning video detection method suitable for tunnels with limited viewing angles. This method addresses the challenges of accurately detecting small, distant targets in complex tunnel environments, such as low camera height, narrow viewing angles, severe occlusion, and glare. The method is highly practical and can be easily integrated into embedded devices such as camera front ends, enabling widespread deployment. Its application to real-time tasks such as tunnel situation monitoring and abnormal event warnings can effectively accelerate the intelligent, information-based, and networked development of tunnels and the entire highway network.

[0021] like Figure 1 As shown, the specific implementation process of the present invention is as follows:

[0022] Step 1: Build a training dataset, collect video images from high-definition cameras in the Yangtze River Tunnel, and build a library of annotated and labeled images as a training sample set for the network model;

[0023] Step 2: Build a network model: The detection network of this invention is based on the existing object detection algorithm SSD (SingleShot MultiBox Detector) and is improved. It uses VGG-16 with Mish activation function and BN layer as the backbone of the detection network, uses bidirectional multi-scale feature fusion of jump layers as the neck of the detection network, and uses the head of the SSD algorithm as the head of the detection network. The details are as follows:

[0024] Step 2.1: The activation function in VGG-16 is configured as the Mish function, and a BN layer is inserted between the activation function and the convolution layer;

[0025] Step 2.2: Add 5 convolutional layers after the VGG-16 network to obtain 5 layers of feature maps with sizes of 19*19, 10*10, 5*5, 3*3, and 1*1 respectively;

[0026] Step 2.3: Compare the 5-layer feature map in step 2.2 with the last layer feature map of the VGG-16 network Figure 1 From the smallest to the largest, they are numbered 1, 2, 3, 4, 5, and 6;

[0027] Step 2.4: Upsample the feature map numbered 1 to obtain the feature map numbered 3 Figure 1 The upsampled image of the same size is fused with the feature map numbered 3. Similarly, the feature map numbered 3 is fused with the feature map numbered 5, and the feature map numbered 5 is fused with the feature map numbered 6.

[0028] Step 2.5: Downsample the feature map numbered 6 to obtain the same feature map numbered 4. Figure 1The downsampled image of the sample size is fused with the feature map numbered 4. Similarly, the feature map numbered 4 is fused with the feature map numbered 2, and the feature map numbered 2 is fused with the feature map numbered 1.

[0029] Step 2.6: At this point, the fused 6-layer feature maps form 6 prediction feature maps of different sizes, forming a leap-layer bidirectional feature fusion network as the neck of the detection network;

[0030] step2.7: Use the head of the SSD network as the head of the detection network.

[0031] Step 3: Use the training set in step 1 to train the detection network model in step 2. During the training process, an adaptive online hard-to-distinguish sample discovery method based on target size is used to obtain the detection model. The details are as follows:

[0032] Step 3.1: Perform K-means clustering on the width and height of the target box of the training sample set, and modify the size of the anchor in the detection network model according to the clustering results;

[0033] Step 3.2: Use data set expansion methods to expand the training set. Data set expansion methods include rotation, color conversion, translation and superposition, and reducing the four images to one-quarter and then piecing them together into a whole image. The reduction and piecing method can make the target smaller and increase the number of small targets. These are commonly used data expansion methods in the prior art and will not be described in detail.

[0034] Step 3.3: Perform adaptive online hard-to-distinguish sample discovery based on target size: Divide the expanded training set into batches. Divide the training set into batches. Calculate the average area of ​​all target boxes for all training samples in each batch. The area S of each pre-selected box i Divide by Get the loss scaling ratio R i , the loss of each pre-selected box is divided by R i Then we get the scaling loss that eliminates the influence of the target box area, and divide all pre-selected boxes into positive and negative samples and sort them from large to small according to the scaling loss. Then we get m positive samples, and calculate the back propagation for the loss of all positive samples. Then we take the first 3*m negative samples of the sort and calculate the back propagation. Then we take all the negative samples with an area larger than The negative samples of m are recorded as the "large target" group, and the remaining negative samples are recorded as the "small target" group. Each group is sorted by the scaling loss, and the first m negative samples (2m negative samples) are taken as the difficult samples for the next batch to continue training;

[0035] Step 3.4: In the training of the detection network, the present invention further proposes a new loss function based on the target size SIoU loss function. The specific loss function is:

[0036]

[0037] Where IoU is the intersection-over-union ratio between the pre-selected box and the target box, x, y, w, and h are the horizontal coordinate, vertical coordinate, width, and height of the pre-selected box, respectively. bt 、y bt 、w bt 、h bt They are the horizontal coordinate, vertical coordinate, width and height of the pre-selected box respectively.

[0038] Step 4: Input the test video and use the trained model to identify the location and category of the target in the video. The location is the target box and the category is the vehicle category, such as car, bus, truck, etc. Figure 3 It can be seen that even at the farthest end of the viewing angle, the present invention can still accurately detect the target.

Claims

1. A deep learning video detection method for targets with limited viewing angle in tunnels, characterized by The following steps are involved: Step 1: Build a training dataset: Collect video images from a high-definition camera in the tunnel and build a library of images with annotations and labels as training samples for the network model. The annotations and labels are the target boxes of the targets. Step 2: Build the network model: Build a detection network based on the SSD network, use VGG-16 with Mish activation function and BN layer as the backbone of the detection network, use jump-layer bidirectional multi-scale feature fusion as the neck of the detection network, and use the head of the SSD network as the head of the detection network; Step 3: Use the training set in step 1 to train the network model in step 2. During the training process, adaptive online hard-to-distinguish sample mining based on target size is used to train the detection model. The specific steps of hard-to-distinguish sample mining are as follows: Step 3.1: Perform K-means clustering on the width and height of the target box of the training sample set, and modify the size of the pre-selected box anchor in the detection network model according to the clustering results; Step 3.2: Use the data set expansion method to expand the training set. The data set expansion method includes rotation, color conversion, translation and superposition, and reducing the four images to one-quarter and then piecing them together into a whole image; Step 3.3: Adaptive online hard-to-distinguish sample discovery based on target size: Divide the expanded training set into batches, and calculate the average area of ​​all target boxes for all training samples in each batch The area S of each pre-selected box i Divide by Get the loss scaling ratio R i , the loss of each pre-selected box is divided by R i Then we get the scaling loss that eliminates the influence of the target box area, and divide all pre-selected boxes into positive and negative samples. We sort them from large to small according to the scaling loss, and get m positive samples. We calculate the back propagation for the loss of all positive samples, and take the first 3*m negative samples of the sort to calculate the back propagation, and then calculate all the negative samples with an area larger than The negative samples of are recorded as the "large target" group, and the remaining negative samples are recorded as the "small target" group. Each group is sorted by the scaling loss, and the first m negative samples of each group are taken, a total of 2m negative samples are taken as difficult samples, and used for the next batch to continue training; In the training of Step 3, the loss function SIoU based on the target size is adopted: Where IoU is the intersection-over-union ratio between the pre-selected box and the target box, x, y, w, and h are the horizontal coordinate, vertical coordinate, width, and height of the pre-selected box, respectively. bt 、y bt 、w bt 、h bt They are the horizontal coordinate, vertical coordinate, width and height of the pre-selected box respectively; Step 4: Input the test sample and use the trained detection model to predict the target location and category in the tunnel image.

2. The method for detecting targets with deep learning video in tunnels with limited viewing angles according to claim 1 is characterized in that Step 2 is as follows: Step 2.1: The activation function in VGG-16 is configured as the Mish function, and a BN layer is inserted between the activation function and the convolution layer; Step 2.2: Add 5 convolutional layers after the VGG-16 network to obtain 5 layers of feature maps with sizes of 19*19, 10*10, 5*5, 3*3, and 1*1 respectively; Step 2.3: Number the 5 layers of feature maps in step 2.2 and the last layer of feature maps of the VGG-16 network as 1, 2, 3, 4, 5, and 6 according to their sizes from small to large. Step 2.4: Upsample the feature map numbered 1 to obtain an upsampled map of the same size as the feature map numbered 3, and fuse the upsampled map with the feature map numbered 3. Similarly, fuse the feature map numbered 3 with the feature map numbered 5, and fuse the feature map numbered 5 with the feature map numbered 6. Step 2.5: Downsample the feature map numbered 6 to obtain a downsampled map of the same size as the feature map numbered 4, and fuse the downsampled map with the feature map numbered 4. Similarly, fuse the feature map numbered 4 with the feature map numbered 2, and fuse the feature map numbered 2 with the feature map numbered 1. Step 2.6: The fused 6-layer feature maps form 6 prediction feature maps of different sizes, forming a leap-layer bidirectional feature fusion network as the neck of the detection network; step2.7: Use the head of the SSD network as the head of the detection network.

Images