Highway vehicle abnormal behavior detection method based on deep learning

Abstract

The invention discloses an expressway vehicle abnormal behavior detection method based on deep learning. The expressway vehicle abnormal behavior detection method mainly comprises vehicle detection, target tracking, abnormal behavior identification and the like. Input information of the method is a to-be-judged video sequence, and firstly, whether an input video is in a picture switching state or not is judged, namely, whether a current sequence is a picture shot when a camera moves or not is judged. And if the picture is in the switching state, not carrying out any processing, and continuing to judge the next input video sequence. And if the picture is not in the switching state, performing vehicle detection and tracking frame by frame to obtain running track information of the vehicle, and judging whether abnormal parking, track abnormity and vehicle speed abnormity exist or not according to the track. And if yes, outputting a corresponding highway vehicle abnormal behavior result. According to the method, the functions of picture switching detection, vehicle detection, target tracking, abnormal behavior recognition and early warning are integrated, and the accuracy and real-time performance of highway traffic abnormal event detection can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of highway vehicle abnormal behavior detection, and in particular to a highway vehicle abnormal behavior detection method based on deep learning. Background Art

[0003] Looking at existing research at home and abroad, abnormal behavior detection of highway vehicles has attracted widespread attention, but there are still the following problems to be solved: 1) Low detection accuracy. Traditional rule-based methods usually rely on preset rules and manual inspection. Due to the complexity of road conditions and driver behavior, relying solely on rules cannot effectively capture complex abnormal behaviors, resulting in a high incidence of missed detection and false alarms; 2) Heavy burden of manual analysis. In existing monitoring systems, video data usually requires manual analysis, especially for long-term video streams. Manual analysis is inefficient and prone to omissions. Especially in high-traffic and high-speed environments such as highways, manual analysis cannot meet real-time requirements; 3) Poor real-time performance. Although traditional video monitoring systems can collect video images in real time, data processing and behavior analysis often take a long time. Traditional methods cannot immediately judge the abnormal behavior of vehicles after data collection, resulting in a long reaction time and an increased probability of traffic accidents.

[0004] The technical differences between this application and the prior art are as follows:

[0005] Main comparative differences: This patent proposes a method for detecting abnormal behavior of highway vehicles based on deep learning, which has unique technical features and advantages compared with comparative patents 1, 2, and 3. In terms of technical solutions, this patent not only covers vehicle detection, target tracking, and abnormal behavior recognition, but also adds a special screen switching detection function, which can effectively avoid invalid vehicle detection and tracking in the picture taken by the camera moving, and improve detection efficiency and accuracy. In addition, this patent is more comprehensive in abnormal behavior judgment, taking into account various situations such as abnormal vehicle speed, abnormal parking, and abnormal trajectory, while comparative patent 1 focuses on the semantic analysis of vehicle behavior and the judgment and alarm of illegal behavior; comparative patent 2 adopts an unsupervised learning method, extracts hidden features in depth information through autoencoders for abnormal detection, and uses 3D convolution to extract deep trajectory information; comparative patent 3 focuses on lightweight design, uses improved lightweight vehicle detection algorithms and vehicle tracking algorithms, and pre-processes vehicle trajectories to establish trajectory models to detect abnormal behavior. In terms of application scenarios, this patent focuses more on improving the accuracy and real-time detection of abnormal traffic events on highways, providing comprehensive support for highway traffic management and safety assurance, while the comparative patent 1 is applicable to traffic law enforcement and safety management, the comparative patent 2 is applicable to scenarios that require real-time monitoring of traffic conditions and timely detection of abnormal situations, and the comparative patent 3 is applicable to daily traffic monitoring and emergency response on highways. Overall, this patent shows unique advantages in terms of comprehensiveness, pertinence, and adaptability to specific scenarios, and can better meet the needs of abnormal behavior detection of highway vehicles.

[0006] 1. In the technical comparison with patent CN108335489A_Highway vehicle behavior semantic analysis and abnormal behavior monitoring system and method, the main difference between this patent and comparative patent 1 is that before performing vehicle detection and tracking, this patent will first determine whether the input video is in a screen switching state. If the screen is in a switching state, no processing will be performed. This screen switching detection function can effectively avoid invalid vehicle detection and tracking in the screen shot by the camera moving, and improve detection efficiency and accuracy. Comparative patent 1 focuses on detecting and tracking moving vehicle targets through video, realizing vehicle speed detection and vehicle behavior semantic analysis, and then determining whether the vehicle has illegal behavior, and then realizing illegal behavior alarm. It focuses more on the semantic analysis of vehicle behavior and the judgment and alarm function of illegal behavior, which is obviously different from this patent in detection process and technical emphasis.

[0007] 2. In the technical comparison with patent CN113221716A_An unsupervised traffic abnormal behavior detection method based on foreground target detection, the difference between this patent and comparative patent 2 is mainly reflected in the detection method and technical means. This patent adopts a method based on deep learning to detect, track and identify abnormal behaviors of vehicles, while comparative patent 2 is an unsupervised traffic abnormal behavior detection method based on foreground target detection. It detects target vehicles on highways through the YOLO v5 model, avoiding the complex calculation amount of directly extracting features from low-level optical flow and gradient information, ensuring the robustness and accuracy of feature extraction from high-dimensional semantic information, and extracting hidden features in depth information through autoencoders for abnormality detection. At the same time, 3D convolution is used to directly extract depth trajectory information from the current frame and the previous and next consecutive frames, and then the autoencoder is used to extract hidden features in the depth information for abnormality detection. This patent comprehensively considers various situations such as abnormal vehicle speed, abnormal parking and abnormal trajectory in the judgment of abnormal behavior. In contrast, Patent 2 focuses more on detecting abnormal behavior through unsupervised learning and specific feature extraction technology. The two differ in the technical implementation path and the emphasis of abnormal behavior detection.

[0008] 3. In the technical comparison with patent CN116977946A_Lightweight vehicle abnormal behavior detection method and system for highway scenes, the main difference between this patent and comparative patent 3 lies in the difference in detection algorithm and technical architecture. This patent does not explicitly point out the use of lightweight detection algorithms, but focuses on the realization of vehicle detection, target tracking and abnormal behavior recognition through deep learning, and adds a screen switching detection link in the detection process to improve the accuracy and efficiency of detection. Comparative patent 3 discloses a lightweight vehicle abnormal behavior detection method for highway scenes, which uses a lightweight vehicle detection algorithm MG-YOLOv5s improved based on YOLOv5s to detect vehicles in monitoring video stream data, obtain the coordinate position of the vehicle target in the video, and then use the vehicle tracking algorithm BoT-SORT to track the vehicle and obtain the vehicle's motion trajectory, then pre-process the vehicle trajectory, analyze the vehicle's motion trajectory to establish the vehicle's trajectory model, and detect whether the vehicle has abnormal behaviors such as vehicle parking and vehicle reverse driving through the set judgment conditions. This patent focuses more on the overall application of deep learning and the judgment of screen switching status in terms of technical implementation, while Patent 3 focuses on the application of lightweight algorithms and the establishment of trajectory models. The two have different emphases in technical details and implementation methods. Summary of the invention

[0009] In order to solve the above technical problems, the present invention proposes a method for detecting abnormal behavior of highway vehicles based on deep learning. The method has the advantages of automated detection, efficient and accurate abnormal behavior recognition, strong real-time performance, labor cost saving and strong scalability.

[0010] To achieve the above object, the technical solution adopted by the present invention is:

[0011] A method for detecting abnormal behavior of highway vehicles based on deep learning, comprising the following steps:

[0012] Step 1, judging whether the input video is in a screen switching state according to the input video sequence information;

[0013] Step 2: If the screen is switched, no subsequent abnormal behavior judgment is performed; if the screen is not switched, vehicle detection is performed, and a convolutional neural network is used to extract and fuse features, and a prediction value is output;

[0014] Step 3, calculate the average loss function of the small batch based on the predicted value and label, and update the weight parameters in the network;

[0015] Step 4: After detecting the vehicle, use the Kalman filter method to track each vehicle and predict and update the vehicle trajectory;

[0016] Step 5: Perform abnormal behavior recognition based on the vehicle trajectory, including abnormal parking detection, trajectory abnormality detection, and vehicle speed abnormality detection.

[0017] As a further improvement of the present invention, the specific process of step 1 is as follows:

[0018] Step 11, obtain the real-time image of the camera for 6-9 seconds for specific vehicle abnormal behavior detection. First, detect whether the image in the current sequence is in a switching state. The specific steps are as follows:

[0019] Input a video sequence, all frame sequences are recorded as {f1,f2,…f n}, extract the first and last frames, i.e. f1 and f n The feature point information of the scale-invariant feature conversion is used to match the feature points of the two pictures. After excluding the vehicle area, if the matching threshold is greater than the preset threshold, it is considered that the picture has not been switched. Otherwise, it is considered that the camera is in a rotating state and the picture has been switched.

[0020] As a further improvement of the present invention, the specific process of step 2 is as follows:

[0021] Step 21: If the screen is switched, no subsequent abnormal behavior is judged;

[0022] Step 22: If the screen is not switched, continue to perform subsequent abnormal behavior judgment. Vehicle detection is the basis for abnormal behavior judgment. The real-time video sequences captured by the drone and the fixed surveillance camera are processed frame by frame. The image size of each frame input to the network is fixed pixels.

[0023] Step 23: Use data enhancement technology to randomly combine the four images into a new image. The specific process is as follows:

[0024] 1) Randomly select four different images from the dataset;

[0025] 2) Randomly scale, randomly crop, and flip each image;

[0026] 3) stitching the four processed images together to form a new composite image;

[0027] The synthetic image is divided into four quadrants, each of which is filled with a patch from another source image, and after data augmentation, it enters the network in the form of a mini-batch;

[0028] Step 24, perform feature extraction, using the backbone of the convolutional neural network CSPDarkNet53 to extract deep features in the image. The convolution calculation process is described as:

[0029]

[0030] Where F(x,y) is the output feature map; I(x+i,y+i) is the input image; K(i,j) is the convolution kernel; M and N are the sizes of the convolution kernel;

[0031] Step 25, feature fusion is performed. The top feature map is upsampled twice and fused with the bottom feature map respectively, strong semantic features are conveyed from top to bottom, and multi-scale feature fusion is performed to ensure that various types of vehicles can be detected. The formula for feature fusion is expressed as:

[0032]

[0033] In the formula, F low 、F mid 、F high Respectively represent feature maps from different levels such as bottom, middle, and top; F PAN Represents the fused feature map; Up represents the upsampling operation; Indicates feature fusion;

[0034] Step 26: perform vehicle positioning and classification, and output the predicted box boundary coordinates (x, y, w, h), category, and confidence of each vehicle. The confidence calculation formula is:

[0035] Conf=Pobj ×P class

[0036] Where P obj is the probability of detecting an object; P class is the probability of the object category, and the duplicate bounding boxes are removed by non-maximum suppression NMS;

[0037] Label smoothing reduces the error caused by modeling around incorrect labels to a certain extent, and its method can be expressed as:

[0038]

[0039] Where q is the number of categories; α is the smoothing coefficient; y r is the label before smoothing; is the label after smoothing;

[0040] Step 27, after feature extraction and feature fusion in the network, output the predicted value (including the center point, width, height and confidence of the prediction box).

[0041] As a further improvement of the present invention, the specific process of step 3 is as follows:

[0042] Step 31, calculate the average loss function of the small batch by the predicted value and the label, and update the weight parameters in the network. The loss function in the target detection task consists of two parts: the classification loss function and the regression loss function. The classification loss function is generally a binary cross entropy loss function. However, in the target detection scenario, the ratio of positive and negative samples is seriously unbalanced, and the classification loss is easily dominated by simple negative samples, resulting in model degradation. Different weights can be given to positive and negative, difficult and easy samples to make the model focus on difficult positive samples. The construction method is:

[0043]

[0044] In the formula, Loss z and Loss f are positive and negative classification loss functions respectively; β is the positive and negative coefficient, which is used to solve the imbalance problem of positive and negative samples; γ is the difficulty coefficient, which is used to solve the imbalance problem of difficult and easy samples;

[0045] Step 32, for the regression loss function, the mean square error loss function is used, the center point and width and height of the prediction box are regarded as four sets of independent variables to calculate the loss function respectively, and the CIOU loss function is used, that is, the calculation based on the intersection-over-union ratio is expressed as follows:

[0046]

[0047] In the formula, Loss Cis the CIOU loss function; IOU is the intersection-over-union ratio between the predicted box and the true box; d c is the distance between the center point of the predicted box and the real box; d d is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box; δ is a parameter to measure the consistency of aspect ratio, which can be expressed as:

[0048]

[0049] In the formula, w gt and h gt is the width and height of the real box; w and h are the width and height of the predicted box.

[0050] As a further improvement of the present invention, the specific process of step 4 is as follows:

[0051] Step 41, after detecting the vehicle, track the running trajectory of each vehicle, use the Kalman filter method to track each vehicle, predict and update the vehicle trajectory, and the state update formula of the Kalman filter is:

[0052]

[0053] In the formula, The status before the update; is the updated state; K k is the Kalman gain; z k is the measured value; H is the observation matrix, and the calculation formula of Kalman gain is:

[0054]

[0055] Where P k|k-1 is the prediction covariance, R is the measurement noise covariance;

[0056] Step 42, determine when the trajectory ends and when a new trajectory is generated. For each trajectory, record the time from the last successful match to the current moment. When the value is greater than a preset threshold, the trajectory is considered to be terminated. For those that are not successfully matched, it is considered that a new trajectory may be generated. Observe whether continuous matching is successful in the next several consecutive frames. If successful, it is determined that a new trajectory is generated.

[0057] As a further improvement of the present invention, the specific process of step 5 is as follows:

[0058] Step 51, abnormal parking detection, the running trajectory of the vehicle is determined by calculating the change of the vehicle position in adjacent frames to determine whether abnormal parking occurs. When the displacement of the vehicle in several consecutive frames is less than the set threshold, it is determined that the vehicle has abnormal parking. The displacement change can be expressed as:

[0059]

[0060] In the formula, (x t ,y t ) is the coordinate of the vehicle in the current frame; (x t-1 ,y t-1 ) is the coordinate of the vehicle in the previous frame. If Δd < threshold, the vehicle is judged to be abnormally parked;

[0061] Step 52, trajectory anomaly detection. Trajectory anomaly detection is mainly judged by the change of the vehicle's driving direction. The trajectory changes of vehicles traveling on normal trajectories are relatively smooth in a short period of time, while the driving direction of vehicles with abnormal trajectories often changes dramatically in a short period of time, which has obvious visual distinguishability. The isolation forest algorithm is used to detect abnormal trajectories. The trajectory anomaly score formula of the isolation forest algorithm is:

[0062]

[0063] Where h(x) is the average path length of the sample; c(n) is the expected path length of the node in the binary tree; s(x) is the anomaly score, the lower the score, the more abnormal the trajectory;

[0064] Step 53, abnormal speed detection. The abnormal speed of vehicles on highways includes low speed abnormality and high speed abnormality. The minimum speed limit of highways is generally 50-60km / h, and the maximum speed limit is generally 100-120km / h. Therefore, the low speed threshold is set to 50km / h, and the high speed threshold is set to 120km / h. By calculating the speed of the vehicle, if the speed in several consecutive frames is lower than the set low speed threshold, or higher than the set high speed threshold, it is determined to be abnormal speed. The speed calculation formula is:

[0065]

[0066] Where Δd is the distance change between two frames; Δt is the frame time interval.

[0067] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following advantages: First, by utilizing the deep learning method, it is possible to identify abnormal behaviors of vehicles on highways with high precision, which is more accurate than the traditional rule detection method. Secondly, the system has real-time monitoring and immediate warning functions, and can issue warnings in time at the moment when abnormal behaviors occur, helping traffic management departments or drivers to take emergency measures and reduce accidents. Third, it realizes automated detection without manual intervention, reduces the workload of traffic management departments, and improves monitoring efficiency and intelligence. In summary, the present invention not only improves the level of highway traffic safety management, but also provides technical support for the development of intelligent transportation systems, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a calculation flow chart of the highway vehicle abnormal behavior detection method of the present invention;

[0069] Figure 2 is the highway vehicle occlusion map;

[0070] Figure 3 This is the result of Mosaic data enhancement processing;

[0071] Figure 4 This is an example of a vehicle detection result.

[0072] Figure 5 This is an example diagram of the driving trajectory tracking of a single vehicle and multiple vehicles;

[0073] Figure 6 This is an example diagram of abnormal parking detection;

[0074] Figure 7 It is a schematic diagram of normal trajectory and abnormal trajectory. DETAILED DESCRIPTION

[0075] For Figure 1 As shown in FIG. 1 , a calculation flow chart of a method for detecting abnormal behavior of highway vehicles based on deep learning is shown in FIG. 1 , and the specific steps are as follows:

[0076] Step 1: determine whether the input video is in a screen switching state according to the input video sequence information. The specific process of step 1 is as follows:

[0077] Step 11, obtain the real-time camera image for 6-9 seconds for specific vehicle abnormal behavior detection. Since the image may be in a moving state due to human or other factors, and the trajectory-based abnormal behavior judgment algorithm needs to keep the reference angle consistent, it is necessary to first detect whether the image in the current sequence is in a switching state. The specific steps are as follows: Input a video sequence, and all frame sequences are recorded as {f1,f2,…f n}, extract the first and last frames, i.e. f1 and f n The feature point information of the scale-invariant feature conversion is used to match the feature points of the two images. After excluding the vehicle area, if the matching threshold is greater than the preset threshold, it is considered that the images have not been switched. Otherwise, it is considered that the camera is in a rotating state and the images have been switched.

[0078] Step 2: If the screen is switched, no subsequent abnormal behavior judgment is performed; if the screen is not switched, vehicle detection is performed, and convolutional neural networks are used for feature extraction and feature fusion to output the predicted value. The specific method is as follows:

[0079] Step 21: If the screen is switched, no subsequent abnormal behavior is judged;

[0080] Step 22: If the screen is not switched, continue to make subsequent abnormal behavior judgments. Vehicle detection is the basis for abnormal behavior judgments. The real-time video sequences captured by drones and fixed surveillance cameras are processed frame by frame, and the image size of each frame input to the network is fixed pixels;

[0081] Step 23, using data augmentation technology, randomly combine the four images into a new image, increase the diversity of the data set, and improve the generalization ability of the model. The specific process is: 1) Randomly select four different images from the data set; 2) Perform random scaling, random cropping, and flipping operations on each image; 3) Splice the four processed images together to form a new composite image. The composite image is divided into four quadrants, and each quadrant is filled with a patch from another source image. After data augmentation, it enters the network in the form of small batches;

[0082] Step 24, perform feature extraction, using the backbone of the convolutional neural network CSPDarkNet53 to extract deep features in the image. The convolution calculation process can be described as:

[0083]

[0084] In the formula, F(x,y) is the output feature map; I(x+i,y+i) is the input image; K(i,j) is the convolution kernel; M and N are the sizes of the convolution kernel. By performing convolution operations on each pixel position, the network can effectively extract vehicle features in the image;

[0085] Step 25, feature fusion is performed. During the feature extraction process, the bottom feature map has a larger size, higher resolution, stronger positioning information, and weaker semantic information, while the top feature map has a smaller size, lower resolution, weaker positioning information, and stronger semantic information. By fusing the top feature map with the bottom feature map after two upsamplings, strong semantic features are conveyed from top to bottom, and multi-scale feature fusion is performed to ensure that all types of vehicles can be detected. The formula for feature fusion is expressed as:

[0086]

[0087] In the formula, F low 、F mid 、F high Respectively represent feature maps from different levels such as bottom, middle, and top; F PAN Represents the fused feature map; Up represents the upsampling operation; Indicates feature fusion;

[0088] Step 26: Perform vehicle positioning and classification, and output the predicted box boundary coordinates (x, y, w, h), category, and confidence of each vehicle. The confidence calculation formula is:

[0089] Conf=P obj ×P class

[0090] Where P obj is the probability of detecting an object; P class is the probability of the object category. The accuracy of the detection results can be ensured by removing duplicate bounding boxes through non-maximum suppression (NMS).

[0091] In classification and target detection problems, when the amount of data is insufficient, the generalization ability of the model cannot be guaranteed, which can easily lead to overfitting. At the same time, in large datasets, there are usually wrong labels. Label smoothing can reduce the error caused by modeling around wrong labels to a certain extent. The method can be expressed as:

[0092]

[0093] Where q is the number of categories; α is the smoothing coefficient; y r is the label before smoothing; is the smoothed label. α is 0.1. Taking the one-hot label of (1,0,0) as an example, after label smoothing, it becomes (0.933,0.033,0.033).

[0094] Step 27, after feature extraction and feature fusion in the network, output the predicted value (including the center point, width, height and confidence of the prediction box).

[0095] Step 3: Calculate the average loss function of the small batch based on the predicted value and label, and update the weight parameters in the network. The specific method is as follows:

[0096] Step 31, calculate the average loss function of the small batch by the predicted value and the label, and update the weight parameters in the network. The loss function in the target detection task consists of two parts: the classification loss function and the regression loss function. For the classification loss function, it is generally a binary cross entropy loss function, but in the target detection scenario, the ratio of positive and negative samples is seriously unbalanced, and the classification loss is easily dominated by simple negative samples, resulting in model degradation. Different weights can be given to positive and negative, difficult and easy samples to make the model focus on difficult positive samples. The construction method is:

[0097]

[0098] In the formula, Loss z and Loss f are the positive and negative classification loss functions respectively; β is the positive and negative coefficient (usually 0.25), which is used to solve the imbalance problem of positive and negative samples; γ is the difficulty coefficient (usually 2), which is used to solve the imbalance problem of difficult and easy samples;

[0099] Step 32, for the regression loss function, the mean square error loss function is generally used, and the center point and width and height of the prediction box are regarded as four sets of independent variables to calculate the loss function separately. However, this method ignores the relationship between the center point coordinates and the width and height information, and cannot accurately constrain the position of the prediction box. Therefore, the use of the CIOU loss function, that is, the calculation based on the intersection-over-union ratio, can effectively improve the accuracy of the prediction box. The method is expressed as:

[0100]

[0101] In the formula, Loss C is the CIOU loss function; IOU is the intersection-over-union ratio between the predicted box and the true box; d c is the distance between the center point of the predicted box and the real box; d d is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box; δ is a parameter to measure the consistency of aspect ratio, which can be expressed as:

[0102]

[0103] In the formula, w gt and h gt is the width and height of the real box; w and h are the width and height of the predicted box.

[0104] Step 4: After detecting the vehicle, use the Kalman filter method to track each vehicle and predict and update the vehicle trajectory. The specific process is as follows:

[0105] Step 41, after detecting the vehicle, track the running trajectory of each vehicle. Use the Kalman filter method to track each vehicle and predict and update the vehicle trajectory. The state update formula of the Kalman filter is:

[0106]

[0107] In the formula, The status before the update; is the updated state; K k is the Kalman gain; z k is the measured value; H is the observation matrix. The calculation formula of Kalman gain is:

[0108]

[0109] Where P k|k-1 is the prediction covariance, R is the measurement noise covariance;

[0110] Step 42, determine when the track ends and when a new track is generated. For each track, record the time from the last successful match to the current time. When this value is greater than the threshold set in advance, the track is considered to be terminated. For those that are not successfully matched, it is considered that a new track may be generated. Observe whether the matching is successful in the next several consecutive frames. If successful, it is considered that a new track is generated.

[0111] Step 5: Perform abnormal behavior recognition based on the vehicle trajectory, including abnormal parking detection, abnormal trajectory detection, and abnormal vehicle speed detection. The specific process is as follows:

[0112] Step 51, abnormal parking detection, the vehicle's running trajectory is determined by calculating the change in the vehicle position in adjacent frames to determine whether an abnormal parking occurs. When the displacement of the vehicle in several consecutive frames is less than the set threshold, it is determined that the vehicle has an abnormal parking. The displacement change can be expressed as:

[0113]

[0114] In the formula, (x t ,y t ) is the coordinate of the vehicle in the current frame; (x t-1 ,y t-1 ) is the coordinate of the vehicle in the previous frame. If Δd < threshold, the vehicle is judged to be abnormally parked;

[0115] Step 52, trajectory anomaly detection, trajectory anomaly detection is mainly judged by the change of the vehicle's driving direction. The trajectory changes of vehicles traveling on normal trajectories are relatively smooth in a short period of time, while the driving direction of vehicles with abnormal trajectories often changes dramatically in a short period of time, with obvious visual distinguishability. However, since the specific examples of normal trajectories and abnormal trajectories are relatively diverse, they cannot be accurately fitted using mathematical formulas. Moreover, in the collected data, the vast majority are normal trajectories, while there are fewer abnormal trajectories, and the classification model cannot be trained well. To solve this problem, the present invention uses an isolation forest algorithm to detect abnormal trajectories. The trajectory anomaly score formula of the isolation forest algorithm is:

[0116]

[0117] Where h(x) is the average path length of the sample; c(n) is the expected path length of the node in the binary tree; s(x) is the anomaly score, the lower the score, the more abnormal the trajectory;

[0118] Step 53, abnormal speed detection. The abnormal speed of vehicles on highways includes low speed abnormality and high speed abnormality. The minimum speed limit on highways is generally 50-60km / h, and the maximum speed limit is generally 100-120km / h. Therefore, the low speed threshold is set to 50km / h and the high speed threshold is set to 120km / h. By calculating the speed of the vehicle, if the speed in several consecutive frames is lower than the set low speed threshold, or higher than the set high speed threshold, it is determined to be abnormal speed. The speed calculation formula is:

[0119]

[0120] Where Δd is the distance change between two frames; Δt is the frame time interval.

[0121] Example

[0122] The actual application of the present invention on the Nanjing Ring Expressway has verified its effectiveness on the expressway. Through the video stream collected by the monitoring camera, the system detects and warns of abnormal parking, abnormal trajectory and abnormal speed behavior. In the actual test, the system can process 8 video streams within 30 seconds, ensuring that the traffic management department can quickly respond to traffic abnormalities. The test environment is multiple monitoring points on the Nanjing Ring Expressway. The system processes the high-definition video stream of each monitoring point in real time and detects the operating status of the vehicle. The test time is from 7 am to 6 pm from October 5, 2023 to November 5, 2023. The monitoring video traffic is large and the traffic flow on the expressway is relatively dense. The video stream format is H.264 encoding, 1080P high-definition picture quality, and the video frame rate is 30 frames per second.

[0123] (1) First, obtain the real-time image of the camera for 6-9 seconds and input this video sequence. All frame sequences are recorded as {f1,f2,…f n}, extract the first and last frames, i.e. f1 and f n The feature point information of the two pictures is matched. After excluding the vehicle area, if the matching threshold is greater than the preset threshold, it is considered that the pictures have not been switched, and subsequent vehicle detection work is carried out.

[0124] In real high-speed traffic scenarios, one of the problems we face is vehicle occlusion. On highways, vehicles are of different sizes, and occlusion often occurs due to the angle of the surveillance camera. Figure 2As shown in the figure, in this scene, the large car in the red box blocks the small car in the green box. Before the picture is sent to the feature extraction network, Mosaic data enhancement is used to combine different images to increase data diversity. Four pictures are selected each time and placed in a large picture that is twice the width and height of the original image. The four pictures are placed in the upper left, upper right, lower left, and lower right corners respectively. Then the cut part of the small picture is pasted on the large picture. At this time, the size and position of the label box need to be readjusted. Then the integrated large picture is randomly rotated, translated, scaled, cropped, and rescaled to the original size of a picture. Through this data enhancement method, the occlusion phenomenon that occurs in real scenes is simulated, the diversity of training data is increased, and the model's detection of occluded targets is improved.

[0125] (2) The original data set can be expanded through data augmentation: 1) Input two original images; 2) Flip, scale, and change the color gamut of the two images horizontally. Create a large image that is twice the width and height of the original image, and then place the first image in the upper left corner and the second image in the upper right corner; 3) Then paste the portion of the small image onto the large image and pull it up vertically to fill the entire image. Resize and position the label box, translate, scale, and crop the integrated large image, and rescale it to the original size of the image, such as Figure 3 shown.

[0126] The CSPDarkNet53 convolutional neural network is used to extract the features of vehicles in each frame of video, and combined with the multi-scale feature fusion method to ensure that all types of vehicles can be detected. Figure 4 Then, the Kalman filter is used to track each vehicle for multiple frames and predict and update the vehicle trajectory. Examples of single-vehicle and multi-vehicle trajectory tracking are shown in Figure 5 shown.

[0127] (3) Abnormal behavior detection of highway vehicles. The system detects that some vehicles have not moved in the non-parking area for 5 consecutive seconds and outputs abnormal parking. Examples of abnormal parking detection are as follows: Figure 6 shown.

[0128] Trajectory anomaly detection is mainly based on the change in the vehicle's driving direction. The trajectory changes of vehicles traveling on normal trajectories are relatively smooth in a short period of time, while the driving direction of vehicles on abnormal trajectories will change dramatically in a short period of time, which has clear visual distinguishability. The abnormal trajectory is detected using the isolation forest algorithm. The schematic diagram of the normal and abnormal trajectories is shown in the figure below. Figure 7 shown.

[0129] The speed anomaly of vehicles on highways includes low speed anomaly and high speed anomaly. The low speed threshold is set to 50km / h and the high speed threshold is set to 120km / h. By calculating the speed of the vehicle, if the speed within 60 consecutive frames (i.e. 2 seconds) is lower than the set low speed threshold, or higher than the set high speed threshold, it is determined to be a speed anomaly.

[0130] (4) In order to verify the effect of vehicle abnormal behavior detection, four evaluation indicators, namely accuracy, recall rate, false alarm rate and missed alarm rate, are introduced. The calculation formula is:

[0131]

[0132]

[0133] In the formula, the video sequence samples with abnormal vehicle behavior are called positive samples, and the video sequence samples without abnormal vehicle behavior are called negative samples. TP means predicting positive samples as positive samples; TN means predicting negative samples as negative samples; FP means predicting negative samples as positive samples; and FN means predicting positive samples as negative samples.

[0134] During the test period from October 5, 2023 to November 5, 2023, the system continuously monitored 8 video streams of Nanjing Ring Expressway and identified abnormal behaviors of different types of vehicles, as shown in the following table:

[0135] In order to more clearly illustrate the technical solution of the embodiment of the present invention, it will be described in detail below with reference to the accompanying drawings.

[0136] like Figure 1 As shown, this Accuracy (%) Recall rate (%) False alarm rate (%) Missing rate (%) Abnormal parking degree 99 98 0 2 Abnormal trajectory 97 95 2 5 Abnormal vehicle speed 93 92 6 8

[0137] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A method for detecting abnormal behavior of highway vehicles based on deep learning, characterized by: The following steps are involved: Step 1, judging whether the input video is in a screen switching state according to the input video sequence information; Step 2: If the screen is switched, no subsequent abnormal behavior is judged; If the screen is not switched, vehicle detection is performed, and convolutional neural networks are used to extract and fuse features and output prediction values; Step 3, calculate the average loss function of the small batch based on the predicted value and label, and update the weight parameters in the network; Step 4: After detecting the vehicle, use the Kalman filter method to track each vehicle and predict and update the vehicle trajectory; Step 5: Perform abnormal behavior recognition based on the vehicle trajectory, including abnormal parking detection, trajectory abnormality detection, and vehicle speed abnormality detection.

2. According to claim 1, a method for detecting abnormal behavior of highway vehicles based on deep learning is characterized in that: The specific process of step 1 is as follows: Step 11, obtain the real-time image of the camera for 6-9 seconds for specific vehicle abnormal behavior detection. First, detect whether the image in the current sequence is in a switching state. The specific steps are as follows: Input a video sequence, all frame sequences are recorded as {f1,f2,…f n }, extract the first and last frames, i.e. f1 and f n The feature point information of the scale-invariant feature conversion is used to match the feature points of the two pictures. After excluding the vehicle area, if the matching threshold is greater than the preset threshold, it is considered that the picture has not been switched. Otherwise, it is considered that the camera is in a rotating state and the picture has been switched.

3. The method for detecting abnormal behavior of highway vehicles based on deep learning according to claim 1 is characterized in that: The specific process of step 2 is as follows: Step 21: If the screen is switched, no subsequent abnormal behavior is judged; Step 22: If the screen is not switched, continue to perform subsequent abnormal behavior judgment. Vehicle detection is the basis for abnormal behavior judgment. The real-time video sequences captured by the drone and the fixed surveillance camera are processed frame by frame. The image size of each frame input to the network is fixed pixels. Step 23: Use data enhancement technology to randomly combine the four images into a new image. The specific process is as follows: 1) Randomly select four different images from the dataset; 2) Randomly scale, randomly crop, and flip each image; 3) stitching the four processed images together to form a new composite image; The synthetic image is divided into four quadrants, each of which is filled with a patch from another source image, and after data augmentation, it enters the network in the form of a mini-batch; Step 24, perform feature extraction, using the backbone of the convolutional neural network CSPDarkNet53 to extract deep features in the image. The convolution calculation process is described as: Where F(x, y) is the output feature map; I(x+i, y+i) is the input image; K(i, j) is the convolution kernel; M and N are the sizes of the convolution kernel; Step 25, feature fusion is performed. The top feature map is upsampled twice and fused with the bottom feature map respectively, strong semantic features are conveyed from top to bottom, and multi-scale feature fusion is performed to ensure that various types of vehicles can be detected. The formula for feature fusion is expressed as: In the formula, F low 、F mid 、F high Respectively represent feature maps from different levels such as bottom, middle, and top; F PAN Represents the fused feature map; Up represents the upsampling operation; Indicates feature fusion; Step 26: perform vehicle positioning and classification, and output the predicted box boundary coordinates (x, y, w, h), category, and confidence of each vehicle. The confidence calculation formula is: Conf=P obj ×P class Where P obj is the probability of detecting an object; P class is the probability of the object category, and the duplicate bounding boxes are removed by non-maximum suppression NMS; Label smoothing reduces the error caused by modeling around incorrect labels to a certain extent, and its method can be expressed as: In the formula, q is the number of categories; α is the smoothing coefficient; y r is the label before smoothing; is the smoothed label; Step 27, after feature extraction and feature fusion in the network, output the predicted value (including the center point, width, height and confidence of the prediction box).

4. The method for detecting abnormal behavior of highway vehicles based on deep learning according to claim 1, characterized in that: The specific process of step 3 is as follows: Step 31, calculate the average loss function of the small batch by the predicted value and the label, and update the weight parameters in the network. The loss function in the target detection task consists of two parts: the classification loss function and the regression loss function. The classification loss function is generally a binary cross entropy loss function. However, in the target detection scenario, the ratio of positive and negative samples is seriously unbalanced, and the classification loss is easily dominated by simple negative samples, resulting in model degradation. Different weights can be given to positive and negative, difficult and easy samples to make the model focus on difficult positive samples. The construction method is: In the formula, Loss z and Loss f are positive and negative classification loss functions respectively; β is the positive and negative coefficient, which is used to solve the imbalance problem of positive and negative samples; γ is the difficulty coefficient, which is used to solve the imbalance problem of difficult and easy samples; Step 32, for the regression loss function, the mean square error loss function is used, the center point and width and height of the prediction box are regarded as four sets of independent variables to calculate the loss function respectively, and the CIOU loss function is used, that is, the calculation based on the intersection-over-union ratio is expressed as follows: In the formula, Loss C is the CIOU loss function; IOU is the intersection-over-union ratio between the predicted box and the true box; d c is the distance between the center point of the predicted box and the real box; d d is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box; δ is a parameter to measure the consistency of the aspect ratio, which can be expressed as: In the formula, w gt and h gt is the width and height of the real box; w and h are the width and height of the predicted box.

5. The method for detecting abnormal behavior of highway vehicles based on deep learning according to claim 1, characterized in that: The specific process of step 4 is as follows: Step 41, after detecting the vehicle, track the running trajectory of each vehicle, use the Kalman filter method to track each vehicle, predict and update the vehicle trajectory, and the state update formula of the Kalman filter is: In the formula, The status before the update; is the updated state; K k is the Kalman gain; z k is the measured value; H is the observation matrix, and the calculation formula of Kalman gain is: Where P k|k-1 is the prediction covariance, R is the measurement noise covariance; Step 42, determine when the trajectory ends and when a new trajectory is generated. For each trajectory, record the time from the last successful match to the current moment. When the value is greater than a preset threshold, the trajectory is considered to be terminated. For those that are not successfully matched, it is considered that a new trajectory may be generated. Observe whether continuous matching is successful in the next several consecutive frames. If successful, it is determined that a new trajectory is generated.

6. The method for detecting abnormal behavior of highway vehicles based on deep learning according to claim 1, characterized in that: The specific process of step 5 is as follows: Step 51, abnormal parking detection, the running trajectory of the vehicle is determined by calculating the change of the vehicle position in adjacent frames to determine whether abnormal parking occurs. When the displacement of the vehicle in several consecutive frames is less than the set threshold, it is determined that the vehicle has abnormal parking. The displacement change can be expressed as: In the formula, (x t ,y t ) is the coordinate of the vehicle in the current frame; (x t-1 ,y t-1 ) is the coordinate of the vehicle in the previous frame. If Δd < threshold, the vehicle is judged to be abnormally parked; Step 52, trajectory anomaly detection. Trajectory anomaly detection is mainly judged by the change of the vehicle's driving direction. The trajectory changes of vehicles traveling on normal trajectories are relatively smooth in a short period of time, while the driving direction of vehicles with abnormal trajectories often changes dramatically in a short period of time, which has obvious visual distinguishability. The isolation forest algorithm is used to detect abnormal trajectories. The trajectory anomaly score formula of the isolation forest algorithm is: Where h(x) is the average path length of the sample; c(n) is the expected path length of the node in the binary tree; s(x) is the anomaly score, the lower the score, the more abnormal the trajectory; Step 53, abnormal speed detection. The abnormal speed of vehicles on highways includes low speed abnormality and high speed abnormality. The minimum speed limit of highways is generally 50-60km / h, and the maximum speed limit is generally 100-120km / h. Therefore, the low speed threshold is set to 50km / h, and the high speed threshold is set to 120km / h. By calculating the speed of the vehicle, if the speed in several consecutive frames is lower than the set low speed threshold, or higher than the set high speed threshold, it is determined to be abnormal speed. The speed calculation formula is: Where Δd is the distance change between two frames; Δt is the frame time interval.

Images