A video-based method for detecting abnormal behavior in public transportation safety

Abstract

The present invention relates to a method for detecting abnormal behaviors in public transportation safety based on video. The present invention obtains the spatial position of pedestrians based on a pedestrian action recognition detection model, obtains the position of vehicles through target recognition, encodes the spatial positions of pedestrians and vehicles with corresponding times to obtain corresponding spatiotemporal sequences, inputs the spatiotemporal sequences into a pedestrian trajectory prediction model, and forms smooth and correct prediction results through a memory graph structure. The prediction results are combined with the results of road semantic segmentation to detect and predict abnormal behaviors that affect travel safety at intersections. Using a cloud-based data sharing method, abnormal behaviors are downloaded to edge servers within the corresponding range based on their spatiotemporal positions. This method can unload computing nodes from vehicles, effectively reducing the delay in abnormal behavior detection and prediction; solves the problem of occlusion of pedestrian spatial position detection by vehicles, effectively reduces the vehicle's computing power consumption, improves resource scheduling efficiency, and saves terminal task unloading costs. And the recognition accuracy is high.

Description

Technical Field

[0001] The present invention relates to the fields of trajectory prediction, pedestrian position detection, vehicle position detection, road semantic segmentation, and intelligent transportation technology, and in particular to a video-based method for detecting abnormal behavior in public transportation safety. Background Art

[0002] With the rapid development of autonomous driving, smart transportation, and smart cities, the demand for detecting unusual behavior that impacts travel safety in public environments, such as public transportation and public places, has increased significantly. Currently, there are two primary approaches to detecting unusual behavior that impacts travel safety in public environments: mobile smart devices, such as in-vehicle intelligent sensing systems and smartphones, and public place monitoring systems. The advantage of mobile smart devices is their immediate perception, enabling real-time detection of unusual pedestrian behavior within the vehicle's field of view, meeting personalized needs. However, their limitations lie in their limited computing power and limited field of view, making them unable to detect or predict unusual behavior at distant locations. Public place monitoring systems can detect unusual behavior over a wide range of areas. Combined with the widely deployed public transportation safety monitoring video network, they offload the computationally intensive task of road anomaly detection from vehicles to cloud servers or edge servers. However, their limitations lie in their inability to meet personalized needs.

[0003] Therefore, by using public place monitoring systems to promptly detect abnormal pedestrian behavior and share the results with mobile smart devices such as smart cars and smartphones via cloud-edge technology, this will expand the recognition capabilities of mobile smart devices, address their limited computing power, and achieve a unified detection method for abnormal travel safety behavior across mobile smart devices and public place monitoring systems. This will provide a new approach to reducing or preventing abnormal travel safety behavior from evolving into travel safety accidents. Summary of the Invention

[0004] The purpose of the present invention is to address the problems of the existing technology and propose a video-based public transportation safety abnormal behavior detection method using a public place monitoring system. The human skeleton points are input as the pedestrian's position into the pedestrian trajectory prediction model, and semantic segmentation and target detection are used to identify various road areas and vehicle positions. The memory graph structure is used to correct the smoothness and correctness of the trajectory prediction, and then the abnormal behavior is shared with the mobile terminal through the cloud-edge method.

[0005] Using the STGCN pedestrian action recognition model, a skeleton is created for pedestrians in the task area, with the midpoint of the line connecting the two foot bones used as the pedestrian's spatial location. Vehicle positions are acquired through object recognition, and the spatial position information of the pedestrian and vehicle is used as input to the encoder. The pedestrian trajectory prediction model uses a memory graph structure to smooth the predicted trajectory, resulting in a smooth and accurate prediction. This prediction is combined with road semantic segmentation results to detect and predict abnormal behaviors at intersections that could affect travel safety. Using cloud-based data sharing, abnormal behaviors are downloaded to edge servers within the corresponding range based on their spatiotemporal location for download and use by smart mobile devices, or broadcast to smart mobile devices.

[0006] The present invention specifically comprises the following steps:

[0007] Step 1. Build the pedestrian action recognition detection model STGCN. STGCN includes a neural network for detecting pedestrian skeleton points and a SoftMax classifier for pedestrian action classification. Drawing on the ideas of long- and short-time neural networks, the data input at each moment is not only the current frame image, but also the skeleton point positions obtained through convolution of historical frames.

[0008] Specifically, the pedestrian skeleton is created through a spatiotemporal graph convolutional neural network to obtain the pedestrian skeleton information in the video frame; the pedestrian's relative camera position is obtained based on the skeleton information, and the center point of the line connecting the positions of both feet in the skeleton result is used as the pedestrian's current spatial position.

[0009] Step 2. Based on the current spatial position of the pedestrian obtained in step 1, the position of the vehicle is obtained through the target detection algorithm. The spatial position information of the pedestrian and vehicle is used as the input of the encoder and encoded with the video frame time axis to obtain the corresponding spatiotemporal sequence.

[0010] Step 3. Obtain road semantic segmentation results through the neural network FRNet;

[0011] Step 4. Build the pedestrian trajectory prediction model, Social Memory. This model adds a memory graph structure to the Social LSTM neural network to store historical trajectory data and correct and smooth the currently predicted trajectory. The hidden states of the Long Short-Term Memory (LSTM) network are used to capture time-varying motion characteristics. To predict multiple pedestrians, the LSTM network used to predict each pedestrian shares information with surrounding LSTM networks. At each moment, each LSTM node obtains information from surrounding LSTM nodes to correct its own trajectory changes.

[0012] The spatiotemporal sequence of vehicles and pedestrians is input into the pedestrian trajectory prediction model. The historical trajectories are stored in a memory graph structure for decoding. Each frame of input obtains a corresponding predicted trajectory, which affects the predicted trajectory of subsequent frames. The predicted trajectory of the current frame is corrected by the trajectory at previous moments. Trajectory angle changes that exceed the threshold are smoothed to obtain a smooth and accurate pedestrian prediction trajectory.

[0013] Step 5. The pedestrian trajectory prediction results from Step 4 and the road semantic segmentation results from Step 3 are sent to the cloud server to detect abnormal pedestrian behavior. Using a cloud-edge data sharing approach, abnormal behavior is uploaded to the edge server within the corresponding range for download and use by smart mobile terminals, or broadcast to smart mobile terminals.

[0014] Since there are many limitations in the task of detecting the spatial position of pedestrians and the spatial position of surrounding vehicles in the video, it becomes difficult to accurately predict abnormal behavior in traffic. On the one hand, when the sensing point is a vehicle, the height of recognition will be affected by other vehicles. At the same time, considering the energy and computing power that the vehicle can provide, the computing power is lower than that of the cloud server. For example, the detection and prediction of the spatial position of pedestrians at a certain intersection are often affected by the trajectories of multiple vehicles and the positions of surrounding pedestrians. On the other hand, from the analysis of time factors, there is also a correlation between the positions of pedestrians in different time periods on a continuous time series. For example, existing trajectory prediction tasks usually use historical pedestrian positions as part of the input. The advantage of mobile smart terminals is that they can perceive in a timely manner and can perceive the abnormal behavior of pedestrians in the vehicle's field of view in real time to meet personalized needs. Compared with the existing technology, the advantages and positive effects of the present invention are:

[0015] 1. By adopting the present invention, the computing node can be unloaded from the vehicle, effectively reducing the delay of abnormal behavior detection and prediction, helping to detect and predict abnormal behaviors that affect safe travel at public urban intersections, and sharing the results of pedestrian hazard detection and prediction through cloud-edge data. The abnormal behavior is downloaded to the edge server within the corresponding range according to its spatiotemporal location so that it can be downloaded and used by smart mobile terminals, or provided to the smart mobile terminals in a broadcast manner. The abnormal detection results are used as data input for the assisted driving system to improve the driving safety of the vehicle.

[0016] 2. By transferring sensing nodes to intelligent terminals, the problem of vehicles obstructing pedestrian spatial position detection is resolved, effectively reducing vehicle computing power consumption, improving resource scheduling efficiency, and saving terminal task offloading costs. Through a reasonable task transfer strategy, the total computing power of urban pedestrian position detection and prediction is effectively reduced, reducing the resource consumption of anomaly detection on the same road section, reducing vehicle computing power consumption, and improving recognition accuracy to support the long-term stable and efficient operation of the system.

[0017] 3. Taking into account the relationship between historical trajectories and time, the internal memory graph structure in the spatiotemporal decoder is used to correct the originally non-smooth predicted trajectory into a smooth trajectory. The impact of surrounding pedestrians and vehicles on pedestrian trajectories is also taken into account. Finally, the pedestrian trajectory prediction results are combined with the semantic segmentation results of the road to detect abnormal behavior at intersections. This decouples the task of intersection anomaly detection from the vehicle, improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an architecture diagram of a video-based method for detecting abnormal behavior in public transportation safety.

[0019] Figure 2 is a schematic diagram of a method for detecting a pedestrian's position in the present invention;

[0020] Figure 3 This is a schematic diagram of target recognition results for detecting vehicle positions in the present invention;

[0021] Figure 4 Schematic diagram of the model results for intersection semantic segmentation in the present invention;

[0022] Figure 5 A schematic diagram of an algorithm for correcting pedestrian and vehicle predicted trajectories in the present invention;

[0023] Figure 6 This is an architecture diagram of the video-based public transportation safety abnormal behavior detection method. DETAILED DESCRIPTION

[0024] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0025] like Figure 1 As shown, in this embodiment, the monitoring location is the intersection, and the movement of vehicles and pedestrians at the intersection is obtained through monitoring. The computing node is the cloud terminal. Taking the intersection vehicle network scenario as an example, due to the angle limitations of the vehicle terminal equipment and the complex traffic and pedestrian flow conditions in different sections of the road, it becomes difficult for the vehicle to predict the spatial location of pedestrians in advance. If the perception angle can be made higher and wider, more information input can be obtained, and the accuracy of intersection anomaly detection can be improved. Therefore, it is very meaningful and valuable to shift the perception end from the vehicle to the monitoring of the intersection. Moving the computing node to the smart terminal can greatly reduce the vehicle's resource consumption.

[0026] A video-based method for detecting abnormal behavior in public transportation safety. This method detects abnormal intersections by identifying pedestrian and vehicle trajectories and performing semantic segmentation on intersections.

[0027] like Figure 2 As shown, the specific steps of the present invention are as follows:

[0028] Step 1: Build a pedestrian action recognition detection model STGCN (Spatio-Temporal Graph Convolutional Neural Network). STGCN includes a neural network for detecting pedestrian skeleton points and a SoftMax classifier for pedestrian action classification. Drawing on the ideas of long- and short-time neural networks, the data input at each moment is not only the current frame image, but also the skeleton point positions obtained through convolution of historical frames.

[0029] The video captured by the intersection monitoring is used as the input of the pedestrian action recognition detection model (hereinafter referred to as STGCN). The video obtained by the monitoring is decomposed into frames, and each frame is constructed with the STGCN model to construct the human skeleton points. The STGCN model can calculate the relative position of the pedestrian skeleton points in the video frame through the input of the video frame, and the output of the model is the coordinates of each skeleton point. The distribution of skeleton points is as follows Figure 3 As shown, in order to reduce the data input of the calculation nodes, only the skeleton point positions of the two soles of the feet are required, and the midpoint of the line connecting the skeleton results of the two soles of the feet is output as the relative spatial position of the pedestrian.

[0030] Step 2: Obtain vehicle location information through the fast target detection network FasterRCNN. FasterRCNN includes the network backbone, the regional preparation network RPN, the pooling layer ROI Pooling, and the classifier Classifier.

[0031] Backbone is the backbone of the entire object detection network, used for feature extraction, and typically uses networks such as VGG and ResNet. After the original image is input into the backbone, a series of feature maps are generated.

[0032] The RPN (Region Proposal Network) is used to generate candidate regions. Its input is the feature map output by the backbone. The RPN outputs a tensor of shape [2000, 4][2000, 4][2000, 4], representing 2000 proposals per image, which serves as the input to the ROI Head module. However, during training, 512 proposals are sampled, not all 2000.

[0033] The ROI Pooling layer is used to reduce the total amount of computational data without reducing features. The output of this structure is used as the input of the Classifier to obtain the corresponding category and coordinate position.

[0034] The vehicle's position result is as follows Figure 4As shown in the figure, the center point of each box is the spatial position of the vehicle. The spatial position of the vehicle and the spatial position of the pedestrian are used as the input of the encoder and encoded with the video frame time axis to obtain the corresponding spatiotemporal sequence, which serves as the input data of the subsequent pedestrian trajectory prediction model.

[0035] Step 3: Obtain the road semantic segmentation result through the neural network FRNet. FRNet takes the video frame as input, downsamples the video frame to 1 / 2 of the original image to reduce the computational burden, and extracts features from the results of the multi-layer convolutional network through the pooling layer. Through the conventional and decomposition modules, the result is divided into the input of the decoder and the input of the subsequent convolution. The subsequent convolution will continue to extract refined features, and finally the features are used as the input of the decoder again. Through the pooling layer and the normalization layer, the following is obtained: Figure 5 The road semantic segmentation results shown are used as input for the subsequent anomaly detection task.

[0036] Step 4: Build the Social Memory model for pedestrian trajectory prediction. This model adds a memory graph structure to the Social LSTM neural network to store historical trajectory data and correct and smooth the currently predicted trajectory. This memory graph predicts future spatial position information sequences based on historical position information sequences and smoothes the predicted trajectories to represent how the spatial positions of pedestrians and vehicles change over time.

[0037] Because pedestrians dynamically adjust their paths based on the movements of surrounding pedestrians and vehicles, the hidden states of the Long Short-Term Memory (LSTM) network can capture these time-varying motion characteristics. To simultaneously predict multiple pedestrians, the model shares information between the LSTM network used to predict each pedestrian and the surrounding LSTM networks. Therefore, at every moment, each LSTM node obtains information from surrounding LSTM nodes to correct its own trajectory changes.

[0038] like Figure 6 As shown, the spatiotemporal sequence of vehicles and pedestrians is input into a pedestrian trajectory prediction model. The model stores historical trajectories through a memory graph structure for decoding. Each frame of input obtains a corresponding predicted trajectory, which affects the predicted trajectory of subsequent frames. The predicted trajectory of the current frame is corrected by the trajectory at previous moments. Trajectory angle changes that exceed the threshold are smoothed to obtain a smooth and accurate pedestrian prediction trajectory. The final trajectory is smooth and does not have large angle changes. Combining the current trajectory with the previous trajectory as input can also improve the accuracy of trajectory prediction.

[0039] Step 5. Send the pedestrian prediction trajectory obtained in step 4 and the road semantic segmentation result obtained in step 3 to the cloud server to detect abnormal pedestrian behavior.

[0040] The predicted pedestrian trajectory includes the current coordinates of the pedestrian and the change sequence of the pedestrian's future coordinates and time, while the road semantic segmentation result includes the category of the road area and the relative range corresponding to each area; the pedestrian's current and future positions are used to determine the road semantic segmentation area in which they are located, and thus abnormal pedestrians are divided into the following types: pedestrians are currently in a dangerous position (for example, pedestrians walking in the lane area), pedestrians' future positions are in a dangerous position (for example, pedestrians' future positions are in the lane area), pedestrians are in a safe area, and pedestrians' future positions are in a safe area.

[0041] Using a cloud-edge-device data sharing approach, abnormal behaviors are uploaded to an edge server within range for download and use by smart mobile terminals, or broadcast to smart mobile terminals. The abnormality detection results can be used as data input for the driver assistance system to improve vehicle driving safety.

Claims

1. A video-based method for detecting abnormal behavior in public transportation safety, characterized by: The specific steps include: Step 1. Build the pedestrian action recognition detection model STGCN. STGCN includes a neural network for detecting pedestrian skeleton points and a SoftMax classifier for pedestrian action classification. Drawing on the ideas of long- and short-time neural networks, the data input at each moment is not only the current frame image, but also the skeleton point positions obtained through convolution of historical frames. Specifically: create pedestrian skeletons through spatiotemporal graph convolutional neural networks to obtain pedestrian skeleton information in video frames; Obtain the pedestrian's relative position to the camera based on the skeleton information, and use the center point of the line connecting the two soles of the feet in the skeleton result as the pedestrian's current spatial position; Step 2. Based on the current spatial position of the pedestrian obtained in step 1, the position of the vehicle is obtained through the target detection algorithm. The spatial position information of the pedestrian and vehicle is used as the input of the encoder and encoded with the video frame time axis to obtain the corresponding spatiotemporal sequence; Step 3. Obtain road semantic segmentation results through the neural network FRNet; Step 4. Build a Social Memory model for pedestrian trajectory prediction. The Social Memory model adds a memory graph structure to the Social LSTM neural network to store historical trajectory data and correct and smooth the currently predicted trajectory. It uses the hidden states of the Long Short-Term Memory (LSTM) network to capture time-varying motion characteristics. To predict multiple pedestrians, the LSTM network used to predict each pedestrian shares information with surrounding LSTM networks. At each moment, each LSTM node obtains information from surrounding LSTM nodes to correct its own trajectory changes. The spatiotemporal sequence of vehicles and pedestrians is input into the pedestrian trajectory prediction model. The historical trajectories are stored and decoded through a memory graph structure. Each frame of input generates a corresponding predicted trajectory, which influences the predicted trajectory of subsequent frames. The predicted trajectory of the current frame is corrected by the trajectory at previous moments. Any trajectory angle changes exceeding the threshold are smoothed to obtain a smooth and accurate pedestrian prediction trajectory. Step 5. Send the pedestrian trajectory prediction results obtained in step 4 and the road semantic segmentation results obtained in step 3 to the cloud server to detect abnormal pedestrian behavior; Using cloud-edge data sharing, abnormal behaviors are uploaded to edge servers within the corresponding range so that smart mobile terminals can download and use them, or provide them to smart mobile terminals in a broadcast manner.

2. The video-based public transportation safety abnormal behavior detection method according to claim 1, characterized in that: The FRNet takes video frames as input, downsamples the video frames to 1 / 2 of the original image to reduce the computational burden, and extracts features from the results of the multi-layer convolutional network through the pooling layer; Through the conventional and decomposition modules, the results are divided into the input of the decoder and the input of the subsequent convolution. The subsequent convolution will continue to extract refined features. Finally, the features are used as the input of the decoder again, and the road semantic segmentation results are obtained through the pooling layer and normalization layer.

Images