A pedestrian abnormal behavior detection method based on a bidirectional trajectory prediction network

Abstract

The application discloses a pedestrian abnormal behavior detection method based on a bidirectional trajectory prediction network, wherein a bidirectional trajectory prediction network is used to output the skeleton position of a pedestrian in each frame of a video, a loss function based on a body joint and posture is designed to train and optimize the bidirectional trajectory prediction network, and the skeleton position error between a predicted posture of the pedestrian and an actual posture is taken as an index to distinguish normal and abnormal events. The application identifies pedestrian abnormal events by evaluating the expectation of the pedestrian abnormal events in the video, outputs an abnormal score detected in each frame of the video, and supports visualization of a prediction result.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and neural networks, specifically to a method for detecting abnormal pedestrian behavior based on a bidirectional trajectory prediction network. Background Technology

[0002] Pedestrian anomaly detection refers to identifying pedestrian activities and events that do not conform to expected behavior (i.e., anomalies) from video sequences. This is a crucial task for safe autonomous driving. If a pedestrian suddenly runs in front of a car or jumps, or a child throws a ball onto the street, intelligent autonomous vehicles must be able to identify these potentially anomalous activities in order to use appropriate path planning and control strategies to avoid obstacles and ensure safety. The pedestrian anomaly detection problem is established using a training set and a test set. The training videos contain only normal walking activities (such as walking), while the test videos can contain both normal and anomalous activities (such as running, jumping, throwing bags, etc.). The goal is to learn the normal pedestrian activity patterns presented in the training data and detect anomalies in the test set.

[0003] Pedestrian anomaly events typically occur far less frequently than normal activities, making it difficult to obtain training data containing a large number of labeled anomaly events. Therefore, existing solutions to the pedestrian anomaly detection problem primarily rely on unsupervised learning methods, which typically learn feature representations of normal activities by reconstructing video frames and then calculate the reconstruction error of test instances to detect anomalies. The pixel-based features obtained through reconstruction can be unstructured and high-dimensional, potentially leading to sensitivity to noise, lack of interpretability, and redundancy. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention discloses a method for detecting abnormal pedestrian behavior based on a bidirectional trajectory prediction network.

[0005] To solve the aforementioned technical problem, the technical solution adopted by the present invention is: a method for detecting abnormal pedestrian behavior based on a bidirectional trajectory prediction network, characterized by comprising the following steps:

[0006] S01) Prepare the pedestrian anomaly detection dataset video;

[0007] S02) Use the multi-person pose estimation system AlphaPose to extract the joint positions of the pedestrian's two-dimensional skeleton, minimize the combined pose loss function, train the model, and obtain the trained pedestrian anomaly detection model. The pedestrian anomaly detection model is a bidirectional trajectory prediction network.

[0008] S03) During inference, AlphaPose is used to extract the joint positions of the pedestrian's two-dimensional skeleton in the detection video, which are then input into the bidirectional trajectory prediction network, which predicts the joint positions of the pedestrian.

[0009] S04), calculate human joint error, and determine whether there is an abnormal behavior of pedestrians according to the human joint error.

[0010] Further, the bidirectional trajectory prediction network is designed based on a CVAE model, adopts a gated recurrent unit encoder network to encode the input trajectory sequence to obtain an encoded feature vector of the input trajectory sequence, then predicts the Gaussian distribution mean and covariance of the input pedestrian sequence trajectory at the observation time step and the real time step using the CVAE, samples a latent variable from the learned distribution, and connects the encoded feature vector to predict the target position of the trajectory; the predicted target position is input into a bidirectional decoder containing a bidirectional RNN, the forward RNN is a conventional RNN decoder, except that its output is not converted to the trajectory space, the backward RNN is first initialized from the input encoder hidden state, and then the estimated target is input to backward propagation, and then the forward and backward hidden states at the same time step are connected to predict the final pose at the time step.

[0011] Further, on the 17 human body skeleton joint annotation points of the COCO dataset, the joint 18 is added as the average of the left and right shoulder joints and hip joints, and the multi-person pose estimation system AlphaPose is used to generate two-dimensional skeleton joint positions for all pedestrian sequences.

[0012] Further, the combined pose loss function introduces a combined pose constraint based on bones, joints and endpoints, and adds the physical properties of the human skeleton structure on the basis of the trajectory loss, and the combined pose loss function is:

[0013] L=L T +αL B +βL E +γL J ,

[0014] Wherein L T is the trajectory loss function, L B is the bone loss, L J is the joint loss, L E is the endpoint loss, and α, β and γ are the weights of each loss, and the calculation method of each loss is:

[0015]

[0016]

[0017] B n =J parent(n) -J n ,

[0018]

[0019] L E =δLA + δ RA + δ LF + δ RF + δ LL + δ RL ,

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] where ||·|| is the L1 norm operation, ||·||2is the L2 norm operation, represents the real pedestrian skeleton joint in the relative coordinate at time step k, represents the predicted pedestrian skeleton at time step k, and

[0027] are the final target poses of the real and predicted pedestrians at the last time step, i.e., time step t+r, respectively, D KL (P||Q) represents the KL-divergence of the mean and covariance distribution learned by the prior and recognition network in the CVAE; and J n are the predicted and real joint positions, respectively, J parent(n) represents the parent joint of the sub-joint n, B n is the real bone vector obtained by AlphaPose, is the bone vector calculated from the predicted joint, n represents the number of joints, and the end point loss L E is composed of six losses, i.e., left arm δ LA , right arm δ RA , left face δ LF , right face δ RF , left leg δ LL and right leg δ RL , the left arm loss δ LA is the joint end point on the left arm.

[0028] Further, the human joint error is the difference between the predicted pedestrian pose and the ground truth pedestrian pose, which is expressed as:

[0029] In the formula, J ktis the position of the k th bone joint at time step t obtained by AlphaPose, is the predicted value, ω kt is the confidence score obtained by AlphaPose, indicating the visibility of the k th joint at time step t.

[0030] The beneficial effects of the present application: the mainstream solution to the problem of pedestrian anomaly detection is mainly the method of reconstruction, assuming that the model learned only on normal data cannot accurately reconstruct the anomaly. But the disadvantage is that the pixel-based features obtained by reconstruction are high-dimensional non-structured signals sensitive to noise, which masks the information about the scene; and the redundant information existing in the pixel-based features increases the burden of the model trained with them to distinguish between signal and noise. The present application uses the skeleton joint vector extracted by AlphaPose as the feature input to avoid the above two problems. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is the human skeleton joint annotation point used in the present application;

[0032] Figure 2 is the flow chart of the present method. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme of the present application more clear and explicit, the following will combine the drawings to explain the present application in detail.

[0034] Example 1

[0035] The present embodiment discloses a pedestrian anomaly behavior detection method based on a bidirectional trajectory prediction network, which first trains a bidirectional trajectory prediction network, and then performs prediction, specifically including the following steps:

[0036] S01), prepare a pedestrian anomaly detection dataset video, which includes training set and test set videos, the training set only contains normal events (such as walking activities) and does not contain abnormal events. The test set contains normal and abnormal events.

[0037] S02), use the multi-person pose estimation system AlphaPose to extract the two-dimensional skeleton joint position of the pedestrian, minimize the combined pose loss function, train the model, and obtain a trained pedestrian anomaly detection model, which is a bidirectional trajectory prediction network.

[0038] In order to more fully extract the human skeleton features, on the 17 human skeleton joint annotation points of the COCO dataset, the joint node 18 is added as the average of the left and right shoulder joints and hip joints, as shown in Figure 1The two-dimensional skeleton joint positions are generated for all pedestrian sequences using the multi-person pose estimation system AlphaPose.

[0039] The combined pose loss function introduces the combined pose constraints based on bones, joints and end points. The physical properties of the human skeleton structure are added to the trajectory loss. The combined pose loss function is:

[0040] L = L T + aL B + bL E + gL J ,

[0041] wherein L T is the trajectory loss function, L B is the bone loss, L J is the joint loss, L E is the end point loss, a, b and g are the weights of the respective losses, and the respective losses are calculated as:

[0042]

[0043]

[0044] B n = J parent(n) - J n ,

[0045]

[0046] L E = d LA + d RA + d LF + d RF + d LL + d RL ,

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] wherein ||·|| is the L1 norm operation, ||·||2 is the L2 norm operation, represents the real pedestrian skeleton joint at time step k under the relative coordinates, represents the predicted human skeleton at time step k, are the final target poses of the real and predicted human at the last time step, i.e., time step t+r, respectively, D KL (P||Q) represents the KL-divergence of the learned mean and covariance distribution in the CVAE and the recognition network; n are the predicted and real joint positions, respectively, J parent(n) represents the parent joint of the sub-joint n, B Figure 2 n is the real bone vector obtained from AlphaPose, is the bone vector calculated from the predicted joints, n represents the number of joints, and the end-point loss L E is composed of six losses, i.e., left arm δ LA , right arm δ RA , left face δ LF , right face δ RF , left leg δ LL and right leg δ RL , the left arm loss δ LA is the joint end-point on the left arm.

[0054] S03), during inference, the two-dimensional skeleton joint positions of the detected human in the video are extracted using AlphaPose are input into the bidirectional trajectory prediction network, and the bidirectional trajectory prediction network predicts the human joint positions.

[0055] In this embodiment, the bidirectional trajectory prediction network is designed based on a CVAE model, a gated recurrent unit (GRU) encoder network is used to encode the input trajectory sequence to obtain an encoded feature vector of the input trajectory sequence. Then, the CVAE is used to predict the Gaussian distribution mean and covariance of the input human trajectory sequence at the observation time step and the real time step. Then, a latent variable is sampled from the learned distribution and connected with the encoded feature vector to predict the target position of the trajectory. The predicted target position is input into a bidirectional decoder containing a bidirectional RNN, the forward RNN is similar to the conventional RNN decoder, except that its output is not converted to the trajectory space. The backward RNN is first initialized from the input encoder hidden state, and the estimated target is input to propagate backward, and then the forward and backward hidden states at the same time step are connected to predict the final pose at the time step.

[0056] S04), the human joint error e t is calculated, the maximum pooling e t is calculated, and the frame-level anomaly score at time t is calculated.

[0057] ​​​The predicted poses of pedestrians performing abnormal activities deviate further from the true trajectories compared to normal activities. The bidirectional trajectory prediction network learns normal walking motions from the training data, thus the prediction error between the predicted pedestrian poses and the ground truth pedestrian poses can be used as an indicator of abnormal events. The error is first computed by the formula:

[0058]

[0059] Compute the human joint error e t , where J kt is the position of the k th bone joint at time step t obtained by AlphaPose, is the predicted value, ω kt is the confidence score obtained by AlphaPose, indicating the visibility of the k th joint at time step t.

[0060] Since the present application uses the sliding window method to process pedestrian pose sequences as the input and output of the detector, multiple predictions can be made for the same pedestrian instance. Using two error measurement methods, the summary error and the flat error, the abnormal scores of the same pedestrian are collected in multiple predictions. The summary error summarizes the skeleton joint errors of all time steps in each sequence, while the flat error averages the prediction errors of the same time step in the pedestrian sequence. After calculating the skeleton joint error of each pedestrian at each frame (time step) t, we summarize the skeleton error into a frame-level anomaly score by max pooling e t , the frame-level anomaly score at time t is calculated. The frame-level anomaly scores of all t frames are the final output of our anomaly detection model.

[0061] The above, described only the basic principles and preferred embodiments of the present application, the improvements and substitutions made by the person skilled in the art according to the present application, belong to the protection scope of the present application.

Claims

1. A pedestrian abnormal behavior detection method based on a bidirectional trajectory prediction network, characterized in that: Comprise the following steps: S01), prepare a pedestrian anomaly detection dataset video, add a joint 18 as the average of the left and right shoulder joints and hip joints on the 17 human skeleton joint annotation points of the COCO dataset, and use the multi-person pose estimation system AlphaPose to generate two-dimensional skeleton joint positions for all pedestrian sequences; S02), use the multi-person pose estimation system AlphaPose to extract pedestrian two-dimensional skeleton joint positions, minimize the combined pose loss function, train the model, and obtain a trained pedestrian anomaly detection model, the pedestrian anomaly detection model is a bidirectional trajectory prediction network; The combined pose loss function introduces a combined pose constraint based on bones, joints and endpoints, adds the physical properties of the human body skeleton structure to the trajectory loss, and calculates the combined pose constraint of bones, joints and endpoints based on L1 norm and L2 norm; The combined pose loss function is: ， wherein is a trajectory loss function, is a bone loss, is a joint loss, is an end-point loss, , and are weights for each loss, each loss being calculated as: wherein is the L1 norm operation, is the L2 norm operation, represents the real human skeleton joints in relative coordinates at time step k, represents the predicted human skeleton at time step k, and are the final target poses of the real and predicted human at the last time step, i.e., time step t+r, respectively, represents the KL-divergence of the mean and covariance distribution learned by the prior and recognition network in CVAE; and are the predicted and real joint positions, respectively, represents the parent joint of sub-joint n, is the real bone vector obtained from AlphaPose, is the bone vector calculated from the predicted joint, n represents the number of joints, end-point loss is composed of six losses, which are left arm , right arm , left face , right face , left leg , and right leg , left arm loss is the joint end-point on the left arm; S03), during inference, use AlphaPose to extract pedestrian two-dimensional skeleton joint positions in the detected video, input the bidirectional trajectory prediction network, and the bidirectional trajectory prediction network predicts the pedestrian joint position; The bidirectional trajectory prediction network is designed based on the CVAE model, uses a gated recurrent unit encoder network to encode the input trajectory sequence to obtain an encoded feature vector of the input trajectory sequence, then uses CVAE to predict the Gaussian distribution mean and covariance of the input pedestrian sequence trajectory at the observation time step and the true time step, samples a latent variable from the learned distribution, and connects it with the encoded feature vector to predict the target position of the trajectory; The predicted target position is input into a bidirectional decoder containing a bidirectional RNN, the forward RNN is a conventional RNN decoder, except that its output is not converted to the trajectory space, the backward RNN is first initialized from the input encoder hidden state, and the estimated target is input to propagate backward, then the forward and backward hidden states at the same time step are connected to predict the final pose at that time step; S04), calculate the human joint error, and determine whether there is a pedestrian abnormal behavior according to the human joint error.

2. The method of claim 1, wherein the method further comprises: The human joint error is the difference between the predicted pedestrian pose and the ground truth pedestrian pose, and the expression is: , where is the position of the k-th bone joint at time step t obtained by AlphaPose, is the predicted value, is the confidence score obtained by AlphaPose, indicating the visibility of the k-th joint at time step t.

Images