A BRT station fare evasion event detection method based on posture recognition

Abstract

The application discloses a BRT station ticket evasion event detection method based on posture recognition, which comprises the following steps: step one, making a ticket evasion behavior data set; step two, adopting a YoloV5 target detection algorithm to perform human body detection on the data set; step three, human body posture extraction, depicting the shape of the human body in the RGB image or video of the data set; step four, generating a key point or skeleton heat map stack; step five, using a 3D-CNN to perform skeleton behavior recognition; and step six, using a trained model to recognize ticket evasion actions to obtain a detection result. The application uses the posture recognition technology in computer vision, and uses a Pose3D algorithm to realize the ticket evasion detection function based on human body posture estimation and behavior recognition.

Description

TECHNICAL FIELD

[0001] The application relates to a BRT station ticket evasion event detection method based on pose recognition, and belongs to the technical field of artificial intelligence detection. BACKGROUND

[0002] Ticket evasion behavior damages the interests of the operating enterprise, interferes with the normal traffic order and public order, and has a great safety hazard. Ticket evasion behavior is relatively frequent at BRT platforms, but there is still a blank in the management and disposal of this phenomenon by public transport enterprises.

[0003] With the development of the field of computer vision, more and more tasks can be handed over to machines. Machine intelligence plays an increasingly important role in people's production and life. In recent years, with the rapid growth of graphics computing power, the field of computer vision has developed rapidly and has achieved great success in a series of basic visual tasks, so it is possible to detect BRT station ticket evasion events based on pose recognition. SUMMARY

[0004] The technical problem to be solved by the application is how to use artificial intelligence to automatically detect BRT station ticket evasion events through pose recognition.

[0005] To solve the above technical problems, the technical solution of the application provides a BRT station ticket evasion event detection method based on pose recognition, comprising the following steps:

[0006] Step 1, make a ticket evasion behavior dataset;

[0007] Step 2, use YoloV5 target detection algorithm to detect human bodies in the dataset;

[0008] Step 3, human pose extraction, depicting the shape of the human body in the RGB image or video of the dataset;

[0009] Step 4, generate key point or skeleton heat map stacking;

[0010] Step 5, use 3D-CNN for skeletal behavior recognition;

[0011] Step 6: use the trained model to recognize ticket evasion actions to obtain the detection result.

[0012] In step 1, real ticket evasion behavior videos and simulated ticket evasion behavior videos are collected, and the videos are edited to only keep the video sequences of ticket evasion related actions.

[0013] In step 2, the YoloV5 model network structure includes an input end, a backbone, a Neck and a prediction layer; the YoloV5 target detection algorithm is used to detect human bodies in the dataset, which specifically includes:

[0014] Step 2.1 input end: using Mosaic data enhancement, randomly taking four pictures from the data set, random size and distribution, stacking to obtain a new picture; setting anchor size by clustering, calculating anchor value in different training sets during training, and using adaptive picture size scaling mode during prediction;

[0015] Step 2.2 Backbone: using Focus structure, inputting the original image 416*416*3 into the Focus structure, changing it into a feature map of 208*208*12 through slicing operation, and then performing a 32 convolution kernel operation to change it into a feature map of 208*208*32;

[0016] Step 2.3 Neck: FPN structure and PAN structure are used on the neck, FPN is a top-down information transmission and fusion to obtain a predicted feature map, and PAN uses a bottom-up pyramid mode;

[0017] Step 2.4 prediction layer: using GIOU_Loss as the loss function, in the target detection prediction result processing stage, a weighted NMS operation is used for screening of the numerous target boxes to obtain the optimal target box.

[0018] In step three, a Top-Down pose estimation model using HRNet as the backbone network is used as a pose extractor, each 2D key point is stored as coordinates (x, y, score), and the score is the predicted confidence. The extracted 2D pose is stored in the format of coordinates.

[0019] In step four, based on the extracted 2D pose, T two-dimensional key point heat maps of shape KxHxW are stacked to generate a 3D heat map stack of shape KxTxHxW.

[0020] In step five, based on the bone modality and bone+RGB modality, two kinds of 3D-CNN are designed: Pose-SlowOnly and RGBPose-SlowFast. Pose-SlowOnly only uses the bone modality as input, RGBPose-SlowFast contains two branches, which process RGB and bone two modalities respectively. The RGB branch has a low frame rate and a larger network width, and the bone branch has a high frame rate and a smaller network width. There is a bidirectional connection between the two branches to promote feature fusion between modalities.

[0021] The present application uses pose recognition technology in computer vision to realize the function of ticket evasion detection based on human pose estimation and behavior recognition using Pose3D algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The FPN structure and PAN structure information transmission fusion schematic diagram is used on the Neck;

[0023] Figure 2-1 The YoloV5-based pedestrian detection result schematic diagram Figure 1 ;

[0024] Figure 2-2 The YoloV5-based pedestrian detection result schematic diagram two

[0025] Figure 3 The human body posture schematic diagram for simulating the action of escaping ticket is proposed;

[0026] Figure 4 The uniform sampling schematic diagram is used in the skeleton behavior recognition. DETAILED DESCRIPTION

[0027] In order to make the application more obvious and easy to understand, the preferred embodiments are described in detail below with the help of the accompanying drawings.

[0028] The BRT station ticket evasion event detection method based on posture recognition is provided in the embodiment. In practice, common ticket evasion actions mainly include: over-the-fence type, down-the-hole type, sideways type and follow-through type. The data set of these common ticket evasion actions is collected and simulated, and the data set of the corresponding format is made. Then, the data set collected and made is trained based on the Pose3D action recognition algorithm, and the ticket evasion event detection model is constructed.

[0029] The ticket evasion behavior detection method based on Pose3D mainly includes the following steps: data set making, human body detection, human body posture extraction, generating key point (or skeleton) heat map stacking, and action recognition based on 3D-CNN.

[0030] Step 1: making of ticket evasion behavior data set

[0031] The collected real ticket evasion behavior video and simulated ticket evasion behavior video are edited, and only the video sequence of the ticket evasion related action is reserved. Specifically, the optical camera is aimed at the passenger card gate machine area, the video is continuously recorded, and the real ticket evasion behavior video clips are manually selected; in addition, in a simulated manner, the ticket evasion behavior is simulated by a person and video recorded, and the video clips of the simulated ticket evasion behavior are obtained to expand the number of samples for training. The real ticket evasion behavior video clips and the simulated ticket evasion behavior video clips are edited, and only the video sequence of the ticket evasion related action is reserved.

[0032] The skeleton sequence of each video frame is extracted by performing human detection and pose estimation on all frames in the video sequence of the ticket evasion related action. Since Pose3D accepts two formats of skeleton data:.npy and.json, according to its suggestion, the extracted skeleton sequence of each video frame is output as a dataset file in.json format. The dataset is used to train the Pose3D model.

[0033] Step 2: human detection

[0034] Human detection, i.e. detection of pedestrians, is an important prerequisite for subsequent human pose extraction. Since ticket evasion behavior involves human detection, human pose estimation, and action recognition, it will inevitably result in longer algorithm running time, making real-time detection impossible. Therefore, it is necessary to solve the algorithm running efficiency problem at each stage. The present application uses YoloV5 target detection algorithm, which has the fastest image inference speed of 0.007s, i.e. 140 frames per second, meeting the real-time detection requirements of video images, and has a more compact structure. The YoloV5 model network structure is divided into input, backbone, neck, and prediction layer.

[0035] Step 2.1 input: Mosaic data augmentation is used, which means that four images are randomly selected from the dataset, with random size and distribution, stacked to obtain a new image, which enriches the data, increases the recognition ability of small objects, and at the same time calculates four images, which is equivalent to increasing the Mini-batch size (Mini-batch refers to the number of samples), reducing the consumption of GPU memory.

[0036] Yolov5 can also set anchor size by clustering (anchor refers to a reference box of a certain size), and then calculate the anchor value (size of reference box) in different training sets during training. Then an adaptive image size scaling mode is used during prediction, which reduces the black border and improves the prediction speed.

[0037] Step 2.2 Backbone: Focus structure and CSPNet structure are mainly used on Backbone. Focus structure does not exist in YOLOv3 and v4 versions, and its key steps are slicing operation. The original image 416*416*3 is connected to the Focus structure, which is changed to a feature map of 208*208*12 through slicing operation, and then a 32 convolution kernel operation is performed to change it to a feature map of 208*208*32.

[0038] Step 2.3 Neck: FPN structure and PAN structure are used on the Neck. FPN is top-down, using up-sampling to transfer and fuse information, obtaining the predicted feature map, as shown in Figure 1 Fig. 2(a). PAN uses a bottom-up feature pyramid mode, as shown in Figure 1 Fig. 2(b).

[0039] Step 2.4 Prediction layer: The prediction layer loss function uses GIOU_Loss. The prediction includes the bounding box loss function and non-maximum suppression (NMS). GIOU_Loss is used as the loss function in YOLOv5, effectively solving the problem when the bounding boxes do not overlap. In the target detection prediction result processing stage, for the screening of the numerous target boxes that appear, a weighted NMS operation is used to obtain the optimal target box. The calculation method of GIOU is as follows.

[0040]

[0041] The pedestrian detection result based on YoloV5 is shown in Figure 2-1 , Figure 2-2 .

[0042] Step 3: Human pose extraction

[0043] The goal of human pose extraction is to depict the shape of the human body in RGB images or videos. Considering that two-dimensional human poses have higher quality, two-dimensional human poses are chosen as input instead of three-dimensional. In the experiment, considering its good performance in the COCO keypoint recognition task, a Top-Down pose estimation model with HRNet as the backbone network is used as the pose extractor. The direct output of the model is the keypoint heat map. In practice, directly storing the keypoint heat map will consume a large amount of disk space. To improve efficiency, each 2D keypoint is stored as coordinates (x, y, score), where score is the predicted confidence. In the case of using a high-quality feature extractor, using coordinates as input, the accuracy of action recognition only has a small amount of decline (0.4%). Therefore, in subsequent work, the extracted 2D poses are stored in the format of coordinates. The extracted human poses are shown in Figure 3 .

[0044] Step 4: Generate keypoint (or skeleton) heat map stack

[0045] Based on the extracted 2D poses, T two-dimensional keypoint heat maps with a shape of KxHxW need to be stacked to generate a 3D heat map stack with a shape of KxTxHxW as input. If the 2D poses are stored in the form of coordinates in advance, the coordinates (x i , y i ) need to be converted into (x iThe Gaussian distribution is converted back to heatmap form. This process is suitable for both single-person and multi-person scenarios.

[0046] In practice, two methods are used to reduce the redundancy in 3D heatmap stack as much as possible, making it more compact. First, according to the position of the person in the video, a tightest box is found to contain all the people in all frames. After that, according to the found box, the heatmaps of each frame are cropped and rescaled to a specific size. In this way, the redundancy is reduced in space, and more information is contained in a relatively small HxW size. At the same time, uniform sampling is used to reduce the redundancy of 3D heatmap stack in the time dimension. Due to the length of the entire video, it is difficult to process, so usually a subset containing only part of the frames is selected to form a segment as the input of 3D-CNN. Based on the RGB modality method, usually only a short time window is used to sample frames to form the input of 3D-CNN (such as SlowFast, which samples frames in a time window of only 64 frames). Since this sampling method is difficult to capture the entire action, in skeletal behavior recognition, a uniform sampling method is used: when N frames need to be sampled, the entire video is first divided into N segments of the same length, and a frame is randomly selected in each segment. As shown in Figure 4 .

[0047] Step 5: Skeletal behavior recognition using 3D-CNN

[0048] Based on the skeletal modality and the skeletal + RGB modality, two 3D-CNNs, Pose-SlowOnly and RGBPose-SlowFast, are designed. Pose-SlowOnly only uses the skeletal modality as input, and the PosePathway in the table below shows its specific structure. In experiments, Pose-SlowOnly outperforms the current SOTA based on GCN on multiple datasets. Since 3D-CNN is used as the model architecture, Pose-SlowOnly can fuse features with other modalities in the early stage. Based on this, RGBPose-SlowFast is designed, which contains two branches that process the RGB and skeletal modalities respectively. The RGB branch has a low frame rate and a larger network width, and the skeletal branch has a high frame rate and a smaller network width. There is a bidirectional connection between the two branches to facilitate feature fusion between modalities. We fuse the prediction results of the two branches as the final prediction. When training, two separate loss functions are used to train the two branches respectively to avoid overfitting.

[0049] Step 6: Experimental results verification

[0050] The application uses simulation and collected ticket evasion behavior data sets to train a Pose3D action recognition algorithm, and uses the trained model to test actual ticket evasion actions and simulated ticket evasion actions respectively. The test experiment results show that each type of ticket evasion action is successfully identified by the trained model.

Claims

1. A method for detecting fare evasion at BRT stations based on pose recognition, characterized in that, Includes the following steps: Step 1: Create a dataset of fare evasion behaviors; Step 2: Perform human detection on the dataset using the YOLOv5 object detection algorithm. The YOLOv5 model network structure includes an input layer, backbone, neck, and prediction layer. The specific steps for performing human detection on the dataset using the YOLOv5 object detection algorithm are as follows: Step 2.1 Input: Mosaic data augmentation is used to randomly select four images from the dataset with random size and distribution, and stack them to obtain new images; the anchor size is set by clustering, and the anchor value of different training sets is calculated at each training session during training. An adaptive image size scaling mode is used during prediction. Step 2.2 Backbone: Using the Focus structure, the original image 416*416*3 is input into the Focus structure. Through slicing, it becomes a 208*208*12 feature map. Then, a 32-kernel convolution operation is performed to become a 208*208*32 feature map. Step 2.3 Neck: The Neck uses FPN and PAN structures. FPN is a top-down information transfer and fusion to obtain the predicted feature map, while PAN adopts a bottom-up feature pyramid mode. Step 2.4 Prediction Layer: GIOU_Loss is used as the loss function. In the object detection prediction result processing stage, a weighted NMS operation is used to filter the many target boxes to obtain the optimal target box. Step 3: Human pose extraction. The shape of the human body is depicted in the RGB images or videos of the dataset. A Top-Down pose estimation model with HRNet as the backbone network is used as the pose extractor. Each 2D keypoint is stored as coordinates (x, y, score), where score is the prediction confidence level. The extracted 2D pose is stored in coordinate format. Step 4: Generate a stack of keypoint or skeleton heatmaps. Based on the extracted 2D pose, stack T 2D keypoint heatmaps of shape K×H×W to generate a 3D heatmap stack of shape K×T×H×W. With the help of generating (x) i ,y i Centered on C i The Gaussian distribution with the maximum value converts the stored 2D pose in coordinate form back into heatmap form; To reduce redundancy in 3D heatmap stacking, a tightest bounding box is found based on the position of people in the video to include all people in all frames. The heatmap of each frame is cropped based on the found bounding box, and the cropped heatmap is rescaled to a specific size. At the same time, uniform sampling is used to reduce redundancy in the time dimension of 3D heatmap stacking. Step 5: Use 3D-CNN for skeletal behavior recognition; Step 6: Use the trained model to identify fare evasion actions and obtain the detection results.

2. The method for detecting fare evasion at BRT stations based on pose recognition as described in claim 1, characterized in that, In step one, real fare evasion videos and simulated fare evasion videos are collected, and the videos are edited to retain only the video sequences of actions related to fare evasion.

3. The method for detecting fare evasion at BRT stations based on pose recognition as described in claim 1, characterized in that, In step five, two types of 3D-CNN are designed based on the skeletal modality and the skeletal + RGB modality: Pose-SlowOnly and RGBPose-SlowFast. Pose-SlowOnly uses only the skeletal modality as input, while RGBPose-SlowFast contains two branches that process the RGB and skeletal modalities respectively. The RGB branch has a low frame rate and a larger network width, while the skeletal branch has a high frame rate and a smaller network width. There is a bidirectional connection between the two branches to promote feature fusion between modalities.

Images