A rolling stone detection and tracking method based on video streams

Abstract

The application discloses a rolling stone detection and tracking method based on a video stream, and can realize detection and tracking of a rolling stone target in a video. In the detection stage, a method combining moving target detection and deep neural network detection is used to determine the rolling stone target and improve the accuracy of the detection method. In the tracking stage, the method firstly tracks based on the rolling stone target detection result, and then re-matches the track which is not matched at the moment by introducing the moving target detection result, so that the accuracy of the algorithm in tracking the rolling stone and the robustness of the algorithm to different scenes are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a rolling stone detection and tracking method based on video streams. Background Technology

[0002] Rockfall refers to the dynamic process by which individual rocks, due to various reasons, become unstable on the surface of slopes and cliffs and move rapidly downhill along the slope through one or more of these motions, such as falling, rebounding, jumping, rolling, or sliding, before finally coming to rest in a relatively flat area or near an obstacle. Rockfall disasters are common in mountainous areas of my country, especially in the western mountainous regions, along major transportation routes, oil (gas) pipelines, in mountain towns, near power transmission lines, and at hydropower station sites. They are widely distributed, occur suddenly, frequently, and are difficult to prevent, posing a serious threat to various structures and human activities within their affected area. Therefore, the prevention and control of rockfall disasters is a pressing and difficult problem, and the detection and tracking of rockfalls is a prerequisite for their prevention and control.

[0003] Currently, the detection of rolling stones mainly relies on cameras, radar, or lidar. High-definition cameras can obtain images with a much higher resolution than radar, giving them a unique advantage in rolling stone detection. In recent years, with the rapid development of computer vision technology, video stream-based target detection and tracking technologies have been widely used in fields such as autonomous driving and regional security. However, in the field of rolling stone disaster prevention, a mature and effective video stream-based method for rolling stone detection and tracking is still lacking.

[0004] Online multi-object tracking (MOT) algorithms based on video streams are mostly implemented using deep learning. They consist of a detector and a correlator, used for object detection and appearance feature extraction, object feature prediction, and data association, respectively. Besides deep learning methods, detectors also include moving object detection (MOD) algorithms for detecting moving objects in videos. Common methods for MOD include background subtraction, inter-frame subtraction, and optical flow, but MOD algorithms do not have the ability to identify object categories. Currently, tracking algorithm frameworks can be divided into two categories: detection-based tracking (DBT) and joint detection tracking (JDT). DBT is currently the mainstream method in deep learning-based visual multi-object tracking. However, from the perspective of deep neural network structure, sub-modules in DBT, such as feature extraction, can be integrated into the object detection network. Joint detection tracking based on the fusion of DBT sub-modules, i.e., the JDT model, using a single deep network framework to implement visual multi-object tracking, is a new trend that has emerged in the last two years. Regardless of the framework, the tracking performance of the algorithm is closely related to the detection performance of the detector.

[0005] For specific scenarios involving rockfall disasters, most algorithms perform poorly, mainly due to the following reasons: (1) Significant clutter. Rockfall disasters are common in mountainous areas of my country, with complex scenarios. When using deep neural networks to detect rockfall targets, there are problems such as small differences between the target and the background, and many interfering targets in the monitoring scene. (2) Complex and variable scenarios. Rockfall disasters are frequent. The disaster scenarios vary greatly in different locations. Even if a rock dataset is constructed and a rock detector is obtained by offline training of a deep neural network, it is difficult for the detector to achieve good detection results in different scenarios. (3) Neglecting the use of motion information. Motion information is very important when detecting and tracking rockfalls. Current detection and tracking methods mostly use deep neural networks to construct target detectors and then track based on the target detection results. In this process, the motion features of the target are only used when constructing the cost matrix. Such an algorithm process does not fully utilize the motion information of the target and is not the optimal solution for the rockfall detection and tracking problem. Therefore, there is an urgent need for a method that effectively utilizes the motion information of rockfalls and is robust to different scenarios to solve the problem of rockfall detection and tracking based on video streams. Summary of the Invention

[0006] This invention provides a video stream-based method for detecting and tracking rolling stones, enabling the detection and tracking of rolling stone targets in videos. The method can identify the rolling stone target during the detection phase, thereby reducing the computational cost during tracking, and also exhibits strong robustness across different scenarios.

[0007] The technical solution of the present invention includes the following steps:

[0008] S1. Process the scene video using the inter-frame difference method and the background difference method to obtain the moving target detection results;

[0009] S2. Process the scene video using a rock detector to obtain rock detection results;

[0010] S3. Determine the rolling stone target based on the degree of overlap between the moving target detection results and the rock detection results;

[0011] S4. For the rolling stone target detected in the current frame, obtain its motion features and use a small CNN network to obtain its appearance features. Based on the state information of the detected target in the current frame and the prediction information generated by the existing tracker, match the detected target and the existing trajectory, and update the trajectory information based on the matching result.

[0012] S5. For existing trajectories that are not matched at this time, the moving target detection result of the current frame is introduced to perform rematching. The trajectory information is updated based on the matching result to complete the tracking of the rolling stone target.

[0013] Furthermore, S1 uses inter-frame difference and background difference methods to process the scene video, specifically including the following steps:

[0014] (1) Read each frame in the video and perform grayscale processing and image filtering;

[0015] (2) Perform background subtraction on each processed frame, set an initial frame to build a background model, segment the foreground and background according to the difference between the frame to be detected and the background model, obtain a binarized image, and update the background model.

[0016] (3) Perform inter-frame difference on each processed frame, calculate the absolute difference between the frame to be detected and the previous frame or several previous frames, add the difference results, and perform binarization on the inter-frame difference image through the threshold.

[0017] (4) Perform OR operation on the binarized images obtained by the two difference methods and perform morphological processing;

[0018] (5) Calculate the connected components of the obtained image and filter out small shadows and areas by using area thresholding;

[0019] (6) Output the outer bounding box of the connected component as the moving target detection result.

[0020] Furthermore, the method for acquiring the rock detector used in S2 is as follows: rock images are collected through network acquisition and field photography, bounding boxes and category labels are marked on the rock targets in the images, a rock dataset is constructed, and the constructed convolutional neural network model is trained with the rock images as input and the specific information of the rock targets in the images as output to obtain the rock detector.

[0021] Furthermore, the method for processing the scene video using the rock detector in S2 is as follows: the rock detector is used to detect each frame in the video to obtain the detection results of the rock targets in the video frame, the detection results are read, the effective information is extracted, and the bounding box and confidence of the rock targets are output.

[0022] Furthermore, S3 specifically includes the following steps:

[0023] (1) For each frame, calculate the crossover ratio between each pair of moving targets and rock targets detected;

[0024] (2) If the crossover ratio between a moving target and a rock target is greater than the set threshold TIo U If the rock target is in motion, it is considered to be a rolling stone target.

[0025] (3) If the crossover-union ratio is less than the set threshold T IoU Then, calculate the ratios of the intersection area of ​​the two targets to the area of ​​the moving target, and the intersection area to the area of ​​the rock target, respectively. If any of these ratios is greater than a set threshold T, the calculation is performed. IoMY Similarly, the rock target was considered to be in motion and identified as a rolling stone target.

[0026] Furthermore, in S4, the trajectory information is updated based on the matching results, specifically including the following steps:

[0027] (1) Obtain the matching trajectory, unmatched target and unmatched trajectory based on the matching results;

[0028] (2) For a matched trajectory, the motion model and appearance information of the trajectory are updated using the relevant information of the detected target in the current frame associated with the trajectory;

[0029] (3) For unmatched detections, it is considered that they may represent a new trajectory. When the trajectory meets the set conditions, the trajectory status is changed to "confirmed state"; otherwise, it is "unconfirmed state".

[0030] Furthermore, S5 specifically includes the following steps:

[0031] (1) For the unmatched trajectories in S4, introduce the moving target detection results, calculate the distance between the unmatched trajectories and the moving target detection results, and construct the cost matrix;

[0032] (2) Set a distance threshold t. For elements in the cost matrix that exceed t, it is considered that the moving target corresponding to the element cannot be associated with the existing trajectory.

[0033] (3) Based on the cost matrix, the Hungarian matching method is used to associate the moving target with the unmatched trajectory to obtain the matched trajectory, the unmatched moving target and the unmatched trajectory;

[0034] (4) For the unmatched trajectory at this time, determine whether it is in the confirmed state. If it is in the "unconfirmed state", delete the trajectory directly. If it is in the "confirmed state", determine whether it meets the set conditions. If it meets the conditions, keep the trajectory; otherwise, delete the trajectory.

[0035] Beneficial effects

[0036] 1. The method of this invention uses a combination of moving target detection and a deep learning model to detect rolling stones. First, this method can determine the motion state of the target, thereby identifying the rolling stone target during the detection stage and reducing the computational cost during tracking. Second, this method can remove static backgrounds, thereby reducing interfering targets and improving the accuracy of rolling stone detection.

[0037] 2. The method of this invention first tracks the target based on the rolling stone detection results, and then uses the moving target detection results to re-match the unmatched trajectories. Firstly, this tracking method, by introducing the moving target detection results as an effective supplement to the rolling stone detection results during the tracking process, improves the overall utilization rate of target motion information in the algorithm, effectively enhancing the overall accuracy of the algorithm in tracking rolling stones. Secondly, this approach reduces the impact of the moving target detection method on the tracking results, thereby improving the overall robustness of the algorithm to different scenarios. Attached Figure Description

[0038] Figure 1 This is a flowchart of the moving target detection method of the present invention;

[0039] Figure 2 This is a flowchart of the video stream-based rolling stone detection method of the present invention;

[0040] Figure 3 This is a flowchart of the video stream rolling stone detection and tracking method of the present invention. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] This invention provides a rolling stone detection and tracking method based on video stream, the process of which is as follows: Figure 3 As shown, the specific steps include the following:

[0043] S1. Process the scene video using inter-frame difference and background subtraction methods to obtain moving target detection results. The moving target detection flowchart is as follows: Figure 1 As shown.

[0044] For inter-frame difference, if there is no moving target in the video, the image changes between consecutive frames are slight; otherwise, the changes between consecutive frames are obvious, thus enabling the detection of moving targets. For background difference, the scene background can be modeled using the initial frame. If there is no moving target in the video, the changes between the current frame and the background model are slight; otherwise, the changes between the current frame and the background model are obvious, thus enabling the detection of moving targets.

[0045] like Figure 1 As shown, the specific process for moving target detection is as follows:

[0046] S101. Read each frame from the video and perform grayscale processing and image filtering;

[0047] S102. Perform background subtraction on each processed frame to obtain a binarized image;

[0048] An initialization frame is set to build a background model. The foreground and background are segmented based on the difference between the frame to be detected and the background model to obtain a binarized image, and the background model is updated.

[0049] In this embodiment of the invention, background subtraction is implemented using a Gaussian mixture background modeling method. The number of Gaussian kernels is set to 4. The background model is initialized using the first frame of the video. In the subsequent detection process, the foreground in the video is judged and the background model is updated.

[0050] S103. Perform inter-frame difference analysis on each processed frame to obtain a binarized image;

[0051] Calculate the absolute difference between the frame to be detected and the previous frame or several previous frames, add the difference results together, and perform binarization processing on the inter-frame difference image by applying a threshold.

[0052] In this embodiment of the invention, the inter-frame difference is implemented using the three-frame difference method. The absolute difference between the current frame of the video and the previous frame and the frame before that is calculated respectively. The difference results are added together, the threshold is confirmed using the Otsu method, and the added result is binarized.

[0053] S104. Perform an OR operation on the binarized images obtained by the two differentiating methods and then perform morphological processing.

[0054] In this embodiment of the invention, morphological processing is performed on the binarized image by using a 3×3 convolution kernel to perform opening and closing operations on the binarized image, removing isolated points and bridging gaps.

[0055] S105. Calculate the connected components of the obtained image and filter out small shadows and areas by using the area threshold.

[0056] S106. Output the outer bounding box of the connected component as the moving target detection result.

[0057] S2. Use a rock detector to process the scene video and obtain the rock detection results.

[0058] Furthermore, the method for acquiring the rock detector used in S2 is as follows: rock images are collected through network acquisition and field photography, bounding boxes and category labels are marked on the rock targets in the images, a rock dataset is constructed, and the constructed convolutional neural network model is trained with the rock images as input and the specific information of the rock targets in the images as output to obtain the rock detector.

[0059] In this embodiment of the invention, the rock detector is implemented based on a YOLOX network. Specifically, obtaining the rock detector based on the YOLOX network includes:

[0060] We acquired rock images of varying sizes, scenes, quantities, and types through online acquisition and on-site photography. We then used LabelMe software to annotate the rock objects in the images, with the annotation information being the target category and the coordinates of the top-left and bottom-right points of the bounding box of the rock object, thus constructing a rock dataset.

[0061] The rock dataset was divided into training, validation, and test sets in a 6:2:2 ratio.

[0062] Using the YOLOX model, based on the pre-trained model provided by the official website, transfer learning was performed on a constructed rock dataset. Rock images were used as input, and specific information about the rock targets in the images was used as output. The training parameters were: 100 training epochs, 8 batches, and a learning rate of 0.001.

[0063] The rock detector is obtained by selecting the weights that achieve the highest detection accuracy on the validation set during training.

[0064] Furthermore, the method for processing the scene video using the rock detector in S2 is as follows: the rock detector is used to detect each frame in the video to obtain the detection results of the rock targets in the video frame, the detection results are read, the effective information is extracted, and the bounding box and confidence of the rock targets are output.

[0065] S3. Determine the rolling stone target based on the degree of overlap between the moving target detection results and the rock detection results.

[0066] Furthermore, S3 specifically includes the following steps:

[0067] S301. For each frame, calculate the crossover ratio (CROR) between detected moving targets and rock targets;

[0068] For the detection results of the two methods in a certain video frame, the cross-union ratio is first calculated according to equation (1).

[0069]

[0070] In equation (1), Let be the outer bounding box of the region where the i-th moving target is located. Let I be the bounding box of the j-th rock target, and let IoU be the cross-union ratio of the two results.

[0071] S302. If the crossover ratio between a moving target and a rock target is greater than a set threshold T. IoU If the rock target is in motion, it is considered to be a rolling stone target.

[0072] S303. If the crossover-union ratio is less than the set threshold T IoU Then, calculate the ratios of the intersection area of ​​the two targets to the area of ​​the moving target, and the intersection area to the area of ​​the rock target, respectively. If any of these ratios is greater than a set threshold T, the calculation is performed. ioMY Similarly, the rock target was considered to be in motion and identified as a rolling stone target.

[0073] For the remaining rock targets, the degree of overlap between them and the moving target detection results is calculated according to equation (2).

[0074]

[0075] In equation (2), IoM is the proportion of the overlapping area of ​​the two borders to the outer border of the moving target, and IoY is the proportion of the overlapping area of ​​the two borders to the outer border of the rock target. A threshold T is set. IoMY If either or both of the values ​​of IoM or IoY are greater than the set threshold T IoMY If the result indicates that the rock target is a rolling stone, then the rock target is determined to be a rolling stone target. The other rock targets did not move.

[0076] like Figure 2 As shown in this embodiment of the invention, when detecting rolling stones, moving target detection and rock detection based on deep neural networks are performed in parallel.

[0077] S4. For the rolling stone target detected in the current frame, obtain its motion features and use a small CNN network to obtain its appearance features. Based on the state information of the detected target in the current frame and the prediction information generated by the existing tracker, match the detected target and the existing trajectory, and update the trajectory information based on the matching result.

[0078] Furthermore, in S4, the trajectory information is updated based on the matching results, specifically including the following steps:

[0079] S401. Based on the matching results, obtain the matching trajectory, the unmatched target and the unmatched trajectory;

[0080] S402. For a matched trajectory, update the motion model and appearance information of the trajectory using relevant information of the detected target in the current frame associated with the trajectory.

[0081] S403. For unmatched detections, it is considered that they may represent a new trajectory. When the trajectory meets the set conditions, the trajectory status is changed to "confirmed state"; otherwise, it is changed to "unconfirmed state".

[0082] In this embodiment of the invention, the tracking algorithm is implemented based on the DeepSORT method, specifically as follows:

[0083] Eight-dimensional space variables are used to characterize the state of the trajectory at a certain moment, as shown in Equation (3), where (u, v) represents the center of the bounding box; γ represents the aspect ratio of the bounding box; and h represents the height of the bounding box. This represents the velocity information corresponding to (u, v, γ, h);

[0084]

[0085] To measure the correlation between the rolling stone target and the existing trajectory, a cost matrix is ​​constructed. Mahalanobis distance is used to measure the correlation between the Kalman filter prediction results and the rolling stone detection results in terms of motion features; cosine distance is used to measure the correlation between the rolling stone detection results and the historical features of the trajectory in terms of appearance.

[0086] Based on the constructed cost matrix, cascaded matching and IoU matching are used to perform data association to obtain the matched trajectory, unmatched target and unmatched trajectory;

[0087] For a matched trajectory, the motion model and appearance information of the trajectory are updated using relevant information about the rolling stone target in the current frame associated with that trajectory;

[0088] For unmatched detections, it is considered that they may represent a new trajectory, and its trajectory state is set to "unconfirmed state". When the number of consecutive detected frames F of this trajectory... D Greater than the set threshold F Dmin And the cumulative number of frames N detected as rolling stonesrf Greater than the set threshold N rfmin At that time, the trajectory state is changed to "confirmed state".

[0089] S5. For existing trajectories that are not matched at this time, the moving target detection result of the current frame is introduced to perform rematching. The trajectory information is updated based on the matching result to complete the tracking of the rolling stone target.

[0090] S5 introduces moving target detection results as an effective supplement to rolling stone detection results during tracking, thereby improving the overall robustness of the algorithm and the accuracy of tracking rolling stones.

[0091] Furthermore, S5 specifically includes the following steps:

[0092] S501. For the unmatched trajectories in S4, introduce the moving target detection results, calculate the distance between the unmatched trajectories and the moving target detection results, and construct the cost matrix;

[0093] In this embodiment of the invention, Mahalanobis distance is used to describe the correlation between the prediction results of Kalman filtering and the moving target detection results in the unmatched trajectory, as shown in Equation (4).

[0094]

[0095] Where i and j represent the sequence numbers of the tracking box and the detection box, respectively, and d j y represents the position of the j-th moving target detection box. i S represents the predicted position of the target by the Kalman filter of the i-th tracker. i This represents the covariance matrix between the detection location and the average tracking location.

[0096] S502. Set a distance threshold t. For elements in the cost matrix that exceed t, it is considered that the moving target corresponding to the element cannot be associated with the existing trajectory.

[0097] S503. Based on the cost matrix, the Hungarian matching method is used to associate moving targets with unmatched trajectories to obtain matched trajectories, unmatched moving targets, and unmatched trajectories.

[0098] S504. For the unmatched trajectory at this time, determine whether it is in the "confirmed state". If it is in the "unconfirmed state", delete the trajectory directly. If it is in the "confirmed state", determine whether it meets the set conditions. If it meets the conditions, keep the trajectory; otherwise, delete the trajectory.

[0099] In this embodiment of the invention, the determination of whether to delete a trajectory is made by measuring the proportion of frames detected as rolling stones in the trajectory and the number of consecutive unmatched frames in the trajectory. Specifically:

[0100] Calculate the ratio of the number of frames detected as rolling stones to the total length of the trajectory, and set the total length of a trajectory to N.t Frames, of which N rf If a frame is detected as a rolling stone target, the ratio R is calculated as shown in equation (5):

[0101]

[0102] The number of consecutive unmatched frames F in the statistical trajectory is counted. If F is greater than a set frame number threshold F... min Or R is less than the set proportional threshold R max If so, then delete the trajectory.

[0103] This completes the entire workflow design for the video stream-based rolling stone detection and tracking method.

[0104] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for rockfall detection and tracking based on video stream, characterized in that, Comprise the following steps: S1, using inter-frame difference method and background difference method to process scene video, obtain moving target detection result; S2, using rock detector to process scene video, obtain rock detection result; S3, according to the degree of overlap between moving target detection result and rock detection result to determine rockfall target; S4, for the rockfall target detected in the current frame, obtain its motion characteristics, at the same time, use small CNN network to obtain its appearance characteristics, based on the state information of the detected target in the current frame and the prediction information generated by the existing tracker, match the detected target and the existing track, update the track information based on the matching result; S5, for the existing track that is not matched at this time, introduce the moving target detection result of the current frame, and re-match, update the track information based on the matching result, complete the tracking of rockfall target, comprising the following steps: (1) for the unmatched track in S4, introduce the moving target detection result, calculate the distance between the unmatched track and the moving target detection result, and construct a cost matrix; (2) Set distance threshold t For cost matrix exceeding t The element is considered to be a moving target whose corresponding trajectory cannot be associated with the existing trajectory. (3) based on the cost matrix, use hungarian matching method to associate the moving target and the unmatched track, obtain the matched track, the unmatched moving target and the unmatched track; (4) for the unmatched track at this time, judge whether it is in the confirmation state, if it is "unconfirmed state", directly delete the track, if it is "confirmed state", judge whether it meets the set condition, if it meets, keep the track, otherwise, delete the track.

2. The method of claim 1, wherein, The S1 specifically comprises the following steps: (1) read each frame in the video, perform grayscale processing and image filtering; (2) perform background difference on each processed frame, set the initialization frame to establish the background model, segment the foreground and background according to the difference between the to-be-detected frame and the background model, obtain the binary image, and update the background model; (3) perform inter-frame difference on each processed frame, calculate the absolute value difference between the to-be-detected frame and the previous frame or several frames, add the difference results, and perform binaryzation processing on the inter-frame difference image through threshold value; (4) perform or operation on the binary images obtained by the two types of difference methods and perform morphological processing; (5) calculate the connected domain of the obtained image, and filter out small shadows and areas through area threshold value; (6) output the circumscribed bounding box of the connected domain as the moving target detection result.

3. The method of claim 1, wherein, The obtaining method of the rock detector used in S2 is: collecting rock images through network acquisition and field shooting means, labeling the boundary box and class label of the rock target in the image, constructing a rock data set, taking the rock image as input and the specific information of the rock target in the image as output to train the convolutional neural network model, and obtaining the rock detector.

4. The method of claim 1, wherein, The method for processing scene video by using rock detector in S2 is: detecting each frame in the video by using the rock detector to obtain the detection result of the rock target in the video frame, reading the detection result, extracting the effective information therein, and outputting the circumscribed bounding box and confidence of the rock target.

5. The method of claim 1, wherein, The S3 specifically comprises the following steps: (1) For each detected moving target and rock target in each frame, the intersection over union between each other is calculated; (2) If the intersection-over-union of a moving target and a rock target is greater than a set threshold TIoU then the rock target is considered to be in a moving state and is determined to be a rockfall target; (3) If the intersection-over-union is less than a set threshold TIoU , the ratio of the intersection area of the two targets to the moving target area and the intersection area to the rock target area is calculated, respectively. If there is a ratio greater than a set threshold TIoMY , the rock target is considered to be in motion and determined as a rockfall target.

6. The method of claim 1, wherein, The S4 updates the track information based on the matching result, and specifically includes the following steps: (1) Based on the matching result, the matching track is obtained, and the unmatching target and the unmatching track are obtained; (2) For the matching track, the motion model and the appearance information of the track are updated using the related information of the detected target in the current frame associated with the track; (3) For the unmatching detection, it is considered that it may represent a new track, when the track meets the set condition, the track state is changed to "confirmed state", otherwise, it is "unconfirmed state".

Images