Tunnel flooding abnormal event detection method based on YOLO

Abstract

The invention discloses a tunnel waterlogging abnormal event detection method based on YOLO, and the method comprises the steps: firstly collecting a waterlogging video generated in a tunnel through a camera, carrying out the random picture interception of the video, selecting the data of a scene through which a vehicle passes, carrying out the marking, and building a data set; the method comprises the following steps: establishing a data set, then introducing a triple attention mechanism on the basis of a convolution module attention mechanism of YOLOv5 to improve the attention capability of the network on important features, training on a YOLOv5 model by using the established data set to obtain a target detection model, and detecting water accumulation and flooding events occurring in a tunnel based on the target detection model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence, and specifically relates to video surveillance image target detection and processing, and in particular to a tunnel flooding abnormal event detection method based on YOLO. Background Art

[0002] Detection of abnormal events of tunnel flooding is a crucial task after the tunnel is built and put into operation. Due to the special location of urban tunnels, compared with ordinary roads, it is more prone to sudden flooding and difficult to drain quickly. In order to minimize the losses that may be caused by flooding, timely detection and alarm of tunnel flooding is crucial. With the advancement of science and technology, most tunnels are currently equipped with various monitoring sensors, including water level sensors, which play an important role in the early warning and emergency response of tunnel flooding. However, the existing water detection technology still faces some challenges in the tunnel environment, resulting in certain limitations in its detection effect, including the following aspects:

[0003] At present, water accumulation detection in tunnels mainly relies on water level sensors and water pump pressure sensors. However, due to the large volume of traffic in tunnels and the many interference factors such as splashing water caused by vehicles, it is easy to cause sensor misdetection. In addition, the tunnel structure is complex, with blind spots and obstructions. Traditional sensor deployment is difficult to fully cover and is prone to missed detection. These problems make it impossible to detect water accumulation in a timely and accurate manner in the early stages, increasing the risk of water accumulation spreading and causing traffic accidents.

[0004] Traditional waterlogging detection algorithms often have low detection accuracy when faced with complex lighting and environmental conditions in tunnels. Factors such as headlights and reflective materials in tunnels can easily lead to algorithm misjudgment. At the same time, existing algorithms have limited detection capabilities for waterlogging targets and cannot accurately capture the characteristics of initial waterlogging, resulting in poor early warning effects. Summary of the invention

[0005] Purpose of the invention: In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for detecting abnormal events of tunnel flooding based on YOLO, so that abnormal events of tunnel flooding can be detected efficiently and accurately.

[0006] Technical solution: A tunnel flooding abnormal event detection method based on YOLO, including the following steps:

[0007] Step 1: Based on the monitoring system deployed in the tunnel, video and image data including water accumulation and flooding are collected, and the collected data are annotated to generate a data set;

[0008] Step 2: Build a YOLO network model, introduce a triple attention mechanism based on the convolutional module attention mechanism, design a small-scale detection layer for splash target detection based on the feature pyramid network, and introduce cross-layer connections to obtain an improved YOLO network model;

[0009] Step 3: Use the improved YOLO network model to learn and train the constructed data set to build a deep learning model that can identify water splash targets and tunnel flooding events;

[0010] Step 4: using a multi-process parallel computing method to capture and decode the real-time video streams of all surveillance cameras in the tunnel, and input them into several copies of the YOLO network model for parallel detection, thereby realizing the processing of the real-time video stream;

[0011] Step 5: Combine the target detection results of water splashes to estimate the depth of water accumulation, calculate the comprehensive confidence score, and judge whether there is an abnormal event of tunnel flooding based on the score;

[0012] Step 6: Use the multimodal large model to verify the flooding detection results, and use the large model prior knowledge for secondary verification to improve the accuracy of flooding event detection.

[0013] Furthermore, step 1 is to construct a dataset of abnormal events of tunnel flooding. Specifically, during the data collection process, the system will simultaneously record static images and dynamic videos to fully capture the real-time situation in the tunnel. In addition, the corresponding water depth measurement data is recorded. The content of these monitoring data not only includes various abnormal events of flooding, such as rising water levels, splashing water when vehicles pass by, etc., but also covers some scenes where no abnormal events occur. At the same time, in order to increase data diversity, network tools should also be used to obtain data on flooding events in some other urban tunnels and non-tunnel scenes. The data annotation process should include annotating the bounding box information of four categories of targets: splashing water when vehicles pass by, small cars, trucks and buses.

[0014] Furthermore, step 2 is to optimize the YOLO network model to improve the detection performance of water splash targets in tunnel scenes, which is divided into the following steps:

[0015] Step 2-1, based on the original CBAM module, introduces cross-dimensional interaction, combines channel attention, spatial attention and cross-layer attention, and implements a triple attention mechanism, enabling the network to more accurately capture the key features of small targets. Specifically, it includes the following steps:

[0016] (1) The introduced cross-dimensional interactive triple attention mechanism captures the dependency between the channel dimension and the spatial dimension of the input tensor through three parallel branches. The three branches specifically include:

[0017] (a) Height-width attention, calculated by the following formula:

[0018] X' hw =X⊙σ(ψ1(ZPool(X)))

[0019] Where X is the input feature, σ is the sigmoid function, ψ1 represents a two-dimensional convolution layer, and ZPool is a pooling process that compresses the input feature map into two dimensions by performing maximum pooling and average pooling in zero dimension. Specifically expressed as:

[0020] Zpool(χ)=[MaxPool 0d (χ),Avgpool 0d (x)]

[0021] (b) Channel-high attention X' ch and channel-width attention X' cw Similar to the height-width attention calculation method, just transpose the dimension of the feature before and after the calculation.

[0022] (2) The final output triple attention feature is obtained by fusing three weighted feature maps and is calculated by the following formula:

[0023]

[0024] Step 2-2, optimize the feature pyramid network design. Specifically, increase the depth of the feature pyramid and use weighted cross-layer bidirectional connections to obtain a weighted bidirectional feature pyramid module to promote the interaction between low-level feature details and high-level features, with the goal of achieving more adequate feature reuse and information transfer. The weighted bidirectional feature pyramid module includes multiple learnable weight parameters, which are used to learn and adjust the importance of different input features through a fast normalization fusion method. The fast normalization fusion method is calculated by the following formula:

[0025]

[0026] Furthermore, step 3 is to train the YOLO network model using the constructed data set and expand the data set based on the feedback, which is divided into the following steps:

[0027] Step 3-1: Train the improved YOLO network model using the data set constructed by the method.

[0028] Step 3-2: Provide manual feedback on part of the data captured in real time in the tunnel, and generate new data to expand the training data set according to the data processing method in step 1.

[0029] Furthermore, step 4 is to capture and decode the real-time video streams of all surveillance cameras in the tunnel using a parallel computing method, and input them into multiple copies of the YOLO network model for parallel detection, the purpose of which is to ensure the real-time performance of the present invention.

[0030] Furthermore, step 5 is to further post-process the detection target output in step 4 to obtain the confidence level of the predicted water depth and flooding event. Specifically, the method for estimating the water depth uses a multi-layer perceptron for regression prediction, and the specific process is as follows:

[0031] First, the following features are used for depth prediction: water splash height h, water splash width W, and vehicle speed v. The height and width of the water splash are obtained by comparing the relative size of the detection bounding box of the water splash and the nearby different types of vehicles in the target detection results. The vehicle speed is determined by calculating the displacement of the vehicle between two consecutive frames. Next, the architecture of the regression model is as follows: the first layer is the input layer, which receives the above features; the second layer is the hidden layer, which contains 64 neurons and uses the ReLU activation function; the third layer is another hidden layer, which contains 32 neurons and also uses the ReLU activation function; the last layer is the output layer, which is responsible for outputting the predicted water depth D. During the model training process, the mean square error is used as the loss function for optimization. After multiple iterative training, an effective regression model is finally obtained to accurately estimate the depth of water in the tunnel.

[0032] The confidence score of a flooding event is calculated using the following formula:

[0033]

[0034] The parameter N s is the number of detected splash targets, C s,i is the confidence of the i-th water accumulation target. Since a vehicle often splashes multiple water when passing through a flooded road, the credibility of a flooding event will greatly increase when multiple water splash targets are detected.

[0035] Furthermore, step 6 is to use multimodal large language models such as MiniCPM-V to fully verify the waterlogging detection results. This step aims to perform secondary verification on the detection results obtained by the YOLO model in the previous step by integrating the prior knowledge of the large model to ensure the accuracy and reliability of the results. Specifically, the multimodal large language model can handle a variety of data types, including images and text information. In this process, the system combines the waterlogging detection results generated by the YOLO model with relevant environmental information, historical data, and situational background, and conducts a comprehensive assessment of the waterlogging event through the reasoning ability of the large model. In this way, not only can possible false positives or false negatives be effectively identified, but also potential waterlogging risks can be identified based on existing knowledge graphs and pattern recognition capabilities.

[0036] Beneficial effects: The present invention provides a method for detecting abnormal events of tunnel flooding based on YOLO, which has the following beneficial effects:

[0037] (1) Compared with the prior art, this method proposes a method based on the YOLO network model, using the existing tunnel video monitoring system and computer vision methods to detect abnormal events of water accumulation in the tunnel in real time. By detecting the splashes of the passing vehicles, the interference caused by the reflection and fluctuation of the light on the water surface is avoided, and the detection accuracy is improved.

[0038] (2) This method optimizes the feature extraction capability of the YOLO model by introducing a triple attention mechanism and a weighted bidirectional feature pyramid network. The triple attention mechanism combines channel, spatial, and cross-layer attention to help the network capture small target features more accurately. In particular, in tunnel water detection, it can effectively detect the smaller splashes of water caused by vehicles when traffic is slow, thereby improving detection accuracy and sensitivity. At the same time, the weighted bidirectional feature pyramid network promotes the fusion of low-level detail information and high-level semantic features through a bidirectional path, and adopts a fast normalized fusion method to improve computational efficiency and reduce complexity.

[0039] (3) This method introduces a multi-layer perceptron to perform regression prediction on the depth of water accumulation. The multi-layer neural network structure of the multi-layer perceptron model can capture the complex nonlinear relationship between the input vehicle speed, water splash size, and water depth. This allows this method to not only detect whether there is an abnormal flooding event based on real-time video surveillance images, but also predict the specific depth of water accumulation. This can provide accurate water accumulation warnings and depth assessments for tunnel operation and management departments, thereby preventing water accumulation from posing a threat to traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of the method for detecting abnormal events of tunnel flooding based on YOLO according to the present invention. DETAILED DESCRIPTION

[0041] To illustrate the technical solution disclosed in the present invention in detail, further description is given below in conjunction with the accompanying drawings.

[0042] Combination Figure 1 As shown, the present invention provides a method for detecting abnormal events of tunnel flooding based on YOLO, and its specific implementation and application includes the following steps:

[0043] 1. Data Collection

[0044] First, install high-resolution surveillance cameras in the tunnel to ensure that they can cover every corner of the tunnel, especially areas prone to waterlogging. Take static images at regular intervals to ensure that various scenes in the tunnel are captured at different time periods and under different weather conditions. These static images will provide basic data for subsequent annotation. Continuously record the real-time video stream in the tunnel to capture vehicle movement, waterlogging, and its changes. Video recording should cover different time periods and different traffic conditions. During the collection process, use sensors to record additional information related to waterlogging, such as water level, rainfall, temperature, humidity, etc., for subsequent analysis. For each video or image, record metadata such as the collection time, location, and environmental conditions.

[0045] (II) Data processing

[0046] Use the professional data labeling tool LabelImg to label the collected images and videos. Select the target areas of water splashes, small cars, trucks, and buses in the image and draw bounding boxes. Organize the labeled image and video data into training sets and validation sets. The training set is used for model training, and the validation set is used for model performance evaluation. Ensure that the data set covers various complex scenarios and enhance the generalization ability of the model.

[0047] (III) Optimizing the YOLO network model

[0048] The optimization of the YOLO network model includes the following steps:

[0049] Step 2-1, based on the original CBAM module, introduces cross-dimensional interaction, combines channel attention, spatial attention and cross-layer attention, and implements a triple attention mechanism, enabling the network to more accurately capture the key features of small targets. Specifically, it includes the following steps:

[0050] (1) The introduced cross-dimensional interactive triple attention mechanism captures the dependency between the channel dimension and the spatial dimension of the input tensor through three parallel branches. The three branches specifically include:

[0051] (a) Height-width attention, calculated by the following formula:

[0052] X' hw=X⊙σ(ψ1(ZPool(X)))

[0053] Where X is the input feature, σ is the sigmoid function, ψ1 represents a two-dimensional convolution layer, and ZPool is a pooling process that compresses the input feature map into two dimensions by performing maximum pooling and average pooling in zero dimension. Specifically expressed as:

[0054] Zpool(χ)=[MaxPool 0d (χ),Avgpool 0d (x)]

[0055] (b) Channel-high attention X' ch and channel-width attention X' cw Similar to the height-width attention calculation method, just transpose the dimension of the feature before and after the calculation.

[0056] (2) The final output triple attention feature is obtained by fusing three weighted feature maps and is calculated by the following formula:

[0057]

[0058] Step 2-2, optimize the feature pyramid network design. Specifically, increase the depth of the feature pyramid and use weighted cross-layer bidirectional connections to obtain a weighted bidirectional feature pyramid module to promote the interaction between low-level feature details and high-level features, with the goal of achieving more adequate feature reuse and information transfer. The weighted bidirectional feature pyramid module includes multiple learnable weight parameters, which are used to learn and adjust the importance of different input features through a fast normalization fusion method. The fast normalization fusion method is calculated by the following formula:

[0059]

[0060] (IV) Training the improved YOLO network model:

[0061] The YOLO network model is trained using the constructed data set, and the data set is expanded based on the manual feedback after the system is deployed. The specific steps are as follows:

[0062] Step 3-1: Use the data set constructed by the method described in step (ii) to train the improved YOLO network model. It is necessary to correctly set the learning rate, batch size, number of iterations and other training hyperparameters, and regularly use the validation set to evaluate the model performance during the training process, and adjust the hyperparameters and early stopping strategy to avoid overfitting or underfitting.

[0063] Step 3-2: Provide manual feedback on some of the data captured in real time in the tunnel, mark new targets such as water splashes, generate a new data expansion training data set according to the data processing method in step 1, and use the expanded data set to iteratively train the improved YOLO network model to continuously improve the performance and robustness of the model.

[0064] (V) Training a multi-layer perceptron model for estimating water depth:

[0065] The multi-layer perceptron model is trained using meta-information such as water depth collected when constructing the data set. The specific steps are as follows:

[0066] The regression model is constructed with the following architecture: the first layer is the input layer, which receives the extracted features; the second layer is the hidden layer, which contains 64 neurons and uses the ReLU activation function; the third layer is another hidden layer, which contains 32 neurons and also uses the ReLU activation function; the last layer is the output layer, which outputs the predicted water depth D. The designed multi-layer perceptron architecture is implemented using PyTorch.

[0067] The model training uses mean square error as the loss function and uses the Adam optimizer to minimize the gap between the predicted results and the true values.

[0068] (VI) Real-time video stream processing

[0069] The surveillance cameras installed in the tunnel capture real-time video streams, decode all video streams simultaneously through parallel computing methods, and input the decoded frame data into the YOLO network model for processing. The specific steps are as follows:

[0070] a. Use the open interface to query the monitoring point information from the monitoring system platform and obtain the preview stream.

[0071] b. Use the efficient software decoding library OpenCV to decode the video stream in real time and generate image frame data.

[0072] c. Preprocess the decoded frame data, including image scaling, normalization and other operations, to generate a data format suitable for YOLO network model input.

[0073] Real-time video stream processing uses a distributed computing architecture to distribute video stream decoding and detection tasks to multiple computing nodes for parallel decoding. Each node runs multiple copies of the YOLO network model to process the assigned frame data, improving detection efficiency and real-time performance. The specific steps are as follows:

[0074] a. Divide the decoded frame data by source and distribute it to different computing nodes. A load balancing algorithm can be used to ensure balanced computing load on each node.

[0075] b. Run multiple copies of the YOLO network model simultaneously on each computing node, process the assigned frame data, and perform target detection such as splashes.

[0076] c. Merge the detection results output by each computing node and save them in the relational database to generate a complete video detection result.

[0077] (VII) Post-processing of detection targets

[0078] The targets such as water splashes and vehicles output by the YOLO network model are counted, and the number and confidence of each target are counted. The width and height of the water splashes are estimated based on the relative size and position of the detection bounding boxes of the water splashes and different types of vehicles nearby, and the vehicle speed is estimated based on the displacement of the vehicles in consecutive frames. The data of vehicle speed and water splash size are input into the multi-layer perceptron model to calculate the predicted water depth value.

[0079] The confidence score of abnormal flooding events is calculated based on the number of splash targets and the confidence level. The calculation formula is as follows:

[0080]

[0081] The parameter N s is the number of detected splash targets, C s,i is the confidence of the i-th water accumulation target.

[0082] Finally, the detected image is input into the MiniCPM-V multimodal model, and a prompt is given to introduce the location and scene recorded in the detected image. The detection results are verified again using the image and text understanding ability of the MiniCPM-V multimodal model. Finally, an alarm is issued for abnormal tunnel flooding events based on the predicted water depth, the confidence level of the flooding event, and the results of the secondary verification, and the relevant information is recorded for subsequent model iterations.

Claims

1. A tunnel flooding abnormal event detection method based on YOLO, characterized in that: The steps include: S1. Collect video and image data including water accumulation and flooding based on the monitoring system deployed in the tunnel, and annotate the collected data to generate a data set; Data annotation includes labeling the bounding box information of four categories of targets: splashes of water caused by vehicles, small cars, trucks, and buses; S2. Build a YOLO network model, introduce a triple attention mechanism based on the convolution module attention mechanism, design a small-scale detection layer for splash target detection based on the feature pyramid network, and introduce cross-layer connections to obtain an improved YOLO network model; The triple attention mechanism is used to capture the dependency between the channel dimension and the spatial dimension of the input tensor through three parallel branches, thereby realizing cross-dimensional interaction, which specifically includes the following calculation process: Calculation of height-width attention: X' hw =X⊙σ(ψ1(ZPool(X))) Where X is the input feature, σ is the sigmoid function, ψ1 represents a two-dimensional convolutional layer, and ZPool represents the pooling operation. By performing maximum pooling and average pooling in zero dimension, the input feature X map is compressed into two dimensions, which can be specifically expressed as: Zpool(χ)=[MaxPool 0d (χ),Avgpool 0d (χ)] After transposing the input feature X, the channel-height attention X' is obtained according to the height-width attention calculation method. ch and channel-width attention X' cw ; The final output triple attention feature map is obtained by fusing three weighted feature maps: S3, using the improved YOLO network model described in step S2 to learn and train the constructed data set to build a deep learning model that can identify water splash targets and tunnel flooding events; S4, using a multi-process parallel computing method to capture and decode the real-time video streams of all surveillance cameras in the tunnel, and input them into several copies of the YOLO network model for parallel detection, thereby realizing the processing of the real-time video stream; S5. Combine the target detection results of water splashes to estimate the depth of water accumulation, calculate the comprehensive confidence score, and judge whether there is an abnormal event of tunnel flooding based on the score; The calculation formula of the comprehensive confidence score is as follows: The parameter N s is the number of detected splash targets, C s,i is the confidence of the i-th water accumulation target. Since a vehicle often splashes multiple water splashes when passing through a flooded road, the confidence of a flooding event will be greatly increased when multiple water splash targets are detected; The method for estimating the depth of water accumulation uses a multi-layer perceptron to perform regression prediction, including the following process: The following features are used: water splash height h, water splash width W, and vehicle speed v. The goal is to predict the depth of water accumulation D. The height and width of the water splash are obtained by comparing the relative sizes of the detection bounding boxes of the water splash and nearby different types of vehicles in the target detection results. The vehicle speed is calculated by calculating the displacement of the vehicle between two consecutive frames. S6. Use a multimodal large model to verify the flooding detection results, and use the large model prior knowledge for secondary verification to improve the accuracy of flooding event detection.

2. The method for detecting abnormal events of tunnel flooding based on YOLO according to claim 1, characterized in that: The collected data described in step S1 includes picture and video forms, and the water depth is recorded at the same time. The content of the monitoring data includes scenes of abnormal flooding events and some scenes where no abnormal events occur.

3. The method for detecting abnormal events of tunnel flooding based on YOLO according to claim 1, characterized in that: Step S1 also includes using tools such as web crawlers to obtain water accumulation image data of other tunnels and some non-tunnel scenes to expand the training data set. The training data set also includes scene data under different rainfall amounts and different time periods to improve the generalization performance of the final YOLO network model.

4. The method for detecting abnormal events of tunnel flooding based on YOLO according to claim 1, characterized in that: In step S2, the triple attention mechanism introduces cross-dimensional interaction based on the CBAM module, combining channel attention, spatial attention and cross-layer attention, so that the network can more accurately capture the key features of small splash targets.

5. The method for detecting abnormal events of tunnel flooding based on YOLO according to claim 1, characterized in that: The feature pyramid network in step S2 is optimized by increasing the depth of the feature pyramid and adopting weighted cross-layer bidirectional connections to promote the interaction between low-level feature detail information and high-level features, thereby improving feature reuse and information transmission capabilities.

6. The method for detecting abnormal events of tunnel flooding based on YOLO according to claim 5, characterized in that: In the feature pyramid network, one or more weight parameters are set to implement a weighted bidirectional feature pyramid network, and then the importance of different input features is learned and adjusted through a fast normalization fusion method; the fast normalization fusion method is calculated by the following formula: In the formula, O is the output feature after fusion, I i is the input feature branch, w i is a learnable parameter weight, and the ReLU function ensures that w i Non-negative, ∈ = 0.0001 is a small value to prevent numerical instability.

7. The method for detecting abnormal events of tunnel flooding based on YOLO according to claim 1, characterized in that: Step S5 also includes establishing a waterlogging depth prediction regression model, and the model architecture is as follows: The first layer is the input layer, the second layer is the hidden layer, which contains 64 neurons and uses the ReLU activation function; the third layer is a hidden layer of 32 neurons, which also uses the ReLU activation function; the last layer is the output layer, which outputs the predicted water depth; The mean square error is used as the loss function for optimization, and multiple iterations of training are performed to obtain a regression model for estimating water depth.

Images