A method and device for analyzing passenger behavior in subway cars based on deep learning

Abstract

The present invention relates to deep learning, computer vision and intelligent monitoring technology, and specifically to a method and device for analyzing passenger behavior in a subway car based on deep learning. The method includes the following: real-time acquisition of video data from surveillance cameras in a subway car; preprocessing of the acquired video data, and identification of the categories of human body, seat, and floor through the Yolov8 model; then using a deep convolutional neural network to extract and classify features of the preprocessed video image, and using the ST‑GCN model to accurately identify abnormal passenger behavior; analyzing and counting the recognition results, generating analysis reports, and realizing automatic and accurate analysis of subway car passenger behavior through deep learning technology, thereby improving the efficiency and accuracy of subway safety monitoring. At the same time, the device of the present invention has a simple structure, is easy to implement and maintain, and has high practical value and application prospects.

Description

Technical Field

[0001] The present invention relates to deep learning, computer vision and intelligent monitoring technology, and specifically to a method and device for analyzing passenger behavior in a subway car based on deep learning. Background Art

[0002] With the rapid development of urban rail transit, subways have become the first choice for people's public transportation due to their unique transportation advantages. As the number of people taking the subway increases, its safety issues are becoming more and more concerned. Traditional subway safety monitoring mainly relies on manual observation and analysis, which is not only inefficient but also easy to miss key information. Therefore, developing an intelligent method and device that can automatically, timely and accurately analyze the behavior of subway car passengers is of great significance to improving subway safety. Summary of the invention

[0003] The present invention provides a method and device for analyzing passenger behavior in a subway car based on deep learning, aiming to automatically, accurately and timely discover abnormal passenger behaviors, such as lying down, falling down, disputes, etc., by using deep learning technology and real-time monitoring, so as to optimize the passenger travel experience and improve the safety of subway operations. The present invention includes the following contents: A method for analyzing passenger behavior in a subway car based on deep learning, comprising the following steps:

[0004] 1) Real-time collection of video data from surveillance cameras in subway cars;

[0005] 2) Preprocess the collected video data, and identify the categories of human body, seat, and floor through the trained Yolov8 model; the training of the Yolov8 model includes extracting sample video data using the alternate frame method, and sending the sample video data to the Yolov8 model for training; the loss in the training is divided into positive and negative sample allocation strategy and loss calculation: the positive and negative sample allocation strategy is a matching strategy of the label allocation strategy, which selects positive samples according to the classification and regression scores combined with the weighted scores;

[0006] 3) Using a deep convolutional neural network to extract and classify features from the preprocessed video images; the feature extraction and classification includes using a VGG-19 convolutional network to create an input feature map for the input video sequence frames; then, through a two-branch multi-stage deep convolutional neural network, the confidence and affinity maps of the key points of the human body are obtained, wherein the first branch predicts the confidence map of the joint point position , human body local affinity domain In the second stage, the confidence map and affinity field are analyzed greedily, and the corresponding key points are generated for all human bodies in the video frame to obtain the corresponding skeleton map sequence. , Update according to the following formula:

[0007]

[0008] in, and Respectively represent The forward calculation of the two branch networks in the stage, , Respectively represent The joint confidence map and the local affinity domain of the human body at each stage, Represents the feature map extracted by the VGG-19 convolutional network;

[0009] 4) Accurately identify abnormal passenger behavior using the ST-GCN model;

[0010] 5) Analyze and compile statistics on the recognition results and generate analysis reports.

[0011] Preferably, the Loss calculation includes two branches: classification and regression;

[0012] First, calculate the loss of the classification branch. The classification branch uses BCE Loss and performs asymmetric weighting operations on positive and negative samples. The formula is as follows:

[0013]

[0014] in, Represents the overlap ratio between the predicted border and the true border ,when When , it is a positive sample; when When , it is a negative sample, Represents the calculated confidence value for each category, represents the hyperparameter, is an adjustable scale parameter; by assigning Weights are used to reduce the contribution of these samples to the loss. Finally, in order to balance the loss of positive and negative samples, an adjustable ratio parameter is added. Give negative sample loss;

[0015] Then calculate the Loss of the regression branch. The regression branch uses the distribution focus loss, and the formula is as follows:

[0016]

[0017] in, Represents the final position of the center point of the prediction box from a certain edge. Represents the distance between the center coordinate of the prediction box and a certain edge of the prediction box. Respectively represent Ceiling and flooring values;

[0018] Finally, the CIOU Loss is calculated using the following formula:

[0019]

[0020] in, Represents the overlap ratio between the predicted border and the true border, Represents the Euclidean distance between the center points of two prediction boxes, Represents the diagonal distance between the closed areas of the two prediction boxes, represents the aspect ratio of the real box, Represents the aspect ratio of the prediction box, Represents the consistency of measuring the relative proportions of the real box and the predicted box, is the weight coefficient;

[0021] The calculated Loss value of the classification branch, the Loss value of the regression branch and the CIOU Loss value are weighted and summed in a weighted ratio to obtain the final Loss value.

[0022] Preferably, the confidence map is represented as an evaluation of the visibility of each body part. If it is visible, there is a peak in the confidence map, and the current part is visible. The confidence of the pixel at the joint point is 1, and the surrounding pixels are diffused in a Gaussian distribution according to the distance. The farther the distance, the lower the confidence.

[0023] Everyone's The confidence map of each body part is generated as follows:

[0024]

[0025] in, For the Personal Confidence map of joints, is the position of the pixel, Indicates Personal The position information of the joints, is the standard deviation;

[0026] After obtaining the key point position of each body part, the affinity field is used to determine the current key point ownership and the affinity field confidence. Follow the formula below:

[0027]

[0028] in, , are the position coordinates of the two joint points, Represents affinity field, is the interpolation point of the two joint nodes, is the interpolation interval.

[0029] Preferably, accurate identification of abnormal passenger behavior using the ST-GCN model includes:

[0030] Step 1: The human behavior video uses the OpenPose human posture estimation algorithm to obtain the joint point features of all joint points in all frames, which include the spatial coordinates and confidence of the joint points;

[0031] Step 2: According to the spatial coordinates and confidence of the joint points, the relative vectors and bone lengths of each joint point in the same frame are obtained to form the spatial structural features;

[0032] Step 3: According to the spatial coordinates and confidence of the joint points, the joint point movement vectors and movement distances between different frames are obtained to form the temporal dynamic features;

[0033] Step 4: Combine the joint point feature J with all frames T and all related node numbers N to form the original skeleton information sequence;

[0034] Step 5: The spatial structure features obtained in step 2 and the temporal dynamic features obtained in step 3 are respectively combined with all frames T and all related node numbers N to form a new skeleton feature sequence;

[0035] Step 6: Splice the skeleton feature sequences together to form the human body spatiotemporal skeleton features as network input;

[0036] Step 7: Construct spatial graph convolution based on the human body spatiotemporal skeleton;

[0037] Step 8: Construct a temporal convolution module to process temporal information, and form a spatiotemporal unit ST-GCN with the spatial graph convolution in step 7;

[0038] Step 9: Input the human body spatiotemporal skeleton features into ST-GCN, and gradually generate higher-level features through multiple layers of spatiotemporal convolution;

[0039] Step 10: Construct a fully connected layer and return the predicted behavior category through the SoftMax classifier.

[0040] In order to solve the above technical problems, the present invention also provides a subway car passenger behavior analysis device based on deep learning, comprising:

[0041] Video acquisition module: used to collect video data from surveillance cameras in subway cars in real time;

[0042] Video analysis module: pre-processes the collected video data and identifies the categories of human body, seat and floor through Yolov8 model; then uses deep convolutional neural network to extract features and classify the pre-processed video images, and uses ST-GCN model to accurately identify abnormal behaviors of passengers;

[0043] Voice reminder module: used to give voice reminders to the driver and handle abnormal behaviors of passengers in a timely manner;

[0044] Analysis and statistics module: used to analyze and count the recognition results and generate analysis reports to facilitate the management and decision-making of subway operation managers.

[0045] The technical effect of the present invention is that the deep learning technology is used to realize the automatic and accurate analysis of the behavior of subway car passengers, thereby improving the efficiency and accuracy of subway safety monitoring. At the same time, the device of the present invention has a simple structure, is easy to implement and maintain, and has high practical value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, in which like reference numerals generally represent like parts.

[0047] Figure 1 This is a flowchart of the steps of the subway car passenger behavior analysis method based on deep learning of the present invention. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the technical solution of the present application, the present invention is further described in detail below in conjunction with the accompanying drawings and the best embodiments.

[0049] The present invention provides a method and device for analyzing passenger behavior in a subway car based on deep learning, aiming to automatically, accurately and timely detect abnormal passenger behaviors, such as lying down, falling, and disputes, through deep learning technology and real-time monitoring, so as to optimize passenger travel experience and improve subway operation safety.

[0050] The device of the present invention comprises: a video acquisition module, a video analysis module, a voice reminder module and an analysis and statistics module.

[0051] Video acquisition module: used to collect video data from surveillance cameras in subway cars in real time;

[0052] Video analysis module: pre-processes the collected video data and identifies the categories of human body, seat and floor through Yolov8 model; then uses deep convolutional neural network (CNN) to extract features and classify the pre-processed video images, and uses spatial temporal graph convolutional network (ST-GCN) model to accurately identify abnormal behaviors of passengers;

[0053] Voice reminder module: used to give voice reminders to the driver and handle abnormal behaviors of passengers in a timely manner;

[0054] Analysis and statistics module: used to analyze and count the recognition results and generate analysis reports to facilitate the management and decision-making of subway operation managers;

[0055] The subway car passenger behavior analysis method based on deep learning in this application includes the following steps: Figure 1 As shown:

[0056] Step S01: real-time acquisition of video data from surveillance cameras in subway carriages;

[0057] Step S02: pre-process the collected video data, and identify the categories of human body, seat, and floor through the trained Yolov8 model; the training of the Yolov8 model includes extracting sample video data using the frame-by-frame method, and sending the sample video data to the Yolov8 model for training; the Loss in the training is divided into positive and negative sample allocation strategy and Loss calculation: the positive and negative sample allocation strategy is a matching strategy of the label allocation strategy, which selects positive samples according to the classification and regression scores combined with the weighted scores;

[0058] Step S03: using a deep convolutional neural network to perform feature extraction and classification on the preprocessed video image;

[0059] Step S04: using the ST-GCN model to accurately identify abnormal passenger behavior;

[0060] Step S05: Analyze and compile statistics on the recognition results and generate an analysis report.

[0061] Further, in step S02, the real-time image of the surveillance camera in the car is obtained through the video acquisition module, the surveillance video data is extracted using the frame-by-frame method, and the surveillance video data is sent to the Yolov8 model to identify categories such as human body, seat, and floor; the training of the Yolov8 model includes extracting sample video data using the frame-by-frame method, and sending the sample video data to the Yolov8 model for training; the Loss in the training is divided into positive and negative sample allocation strategy and Loss calculation: the positive and negative sample allocation strategy is a matching strategy of the label allocation strategy, which selects positive samples based on the classification and regression scores combined with the weighted scores; the Yolov8 model is changed from the original coupling head to a decoupling head, so that classification and detection are separated, and at the same time, a detection method based on Anchor-Free is adopted, which has higher detection accuracy and faster detection speed.

[0062] Further, the specific description of the Yolov8 model is as follows:

[0063] Step 1: Data preparation: Download the surveillance video from the surveillance camera in the car, extract frames, filter and annotate the data. After filtering, annotate about 10,000 pictures as the training data set;

[0064] Step 2: Model training: Build the model structure of Yolov8, use images with a resolution of 640*640, predict (80*80+40*40+20*20)*3=25200 bounding boxes, send the prepared data set to Yolov8, and start training. During the training, Loss is divided into positive and negative sample allocation strategy and Loss calculation. The positive and negative sample allocation strategy is the matching strategy of the label allocation strategy (Task Aligned Assigner), and the positive sample is selected according to the classification and regression scores combined with the weighted scores;

[0065] Loss calculation includes two branches: classification and regression:

[0066] First, calculate the loss of the classification branch. The classification branch uses binary cross entropy loss (BCE Loss for short) to perform asymmetric weighted operations on positive and negative samples. The formula is as follows:

[0067]

[0068] in, Represents the overlap ratio between the predicted border and the true border ;

[0069] when When , it is a positive sample, The larger the value of is, the higher the quality of the positive sample is. We should focus on training these high-quality positive samples to achieve a higher average accuracy.

[0070] when When , it is a negative sample, Represents the calculated confidence value of each category. For negative samples, Weights are used to reduce the contribution of these samples to the loss. Finally, in order to balance the loss of positive and negative samples, an adjustable ratio parameter is added. Give negative sample loss.

[0071] Next, we calculate the Loss of the regression branch. The regression branch uses the distribution focal loss (Distribution FocalLoss, referred to as: ), mainly to solve the problem that the representation of the bounding box (bbox) is not flexible enough. The traditional regression method directly makes the predicted value to a discrete fixed value (label position ) to approximate, Regressing a distribution range is more effective in determining the location of the target object than approximating a discrete value. Modeling the location of the box as a general distribution allows the network to quickly focus on the label location The position distribution of the nearby range can better locate the position of the target object. The formula is as follows:

[0072]

[0073] in, Represents the final position of the center point of the prediction box from a certain edge. Represents the distance between the center coordinate of the prediction box and a certain edge of the prediction box. Respectively represent Ceiling and flooring values.

[0074] At the same time, the complete intersection over union loss (CIOU Loss) is also used. CIOU Loss is mainly used to more accurately measure the difference between the predicted border and the real border. The formula is as follows:

[0075]

[0076] in, Represents the overlap ratio between the predicted border and the true border, Represents the Euclidean distance between the center points of two prediction boxes, Represents the diagonal distance between the closed areas of the two prediction boxes, represents the aspect ratio of the real box, Represents the aspect ratio of the prediction box, Represents the consistency of measuring the relative proportions of the real box and the predicted box, is the weight coefficient. (Intersection over Union) represents the overlap rate between the predicted bounding box and the true bounding box, that is, the ratio of their intersection and union. When the two parts completely overlap, the ratio is 1. For the real border, To predict the bounding box, express and The area of ​​the intersection, express and The area of ​​the union region.

[0077] The calculation formula is as follows:

[0078]

[0079] The three losses calculated above are weighted and summed with a certain weight ratio to obtain the final loss value.

[0080] Step 3: Model reasoning: When predicting, the regression branch of the Head output is converted, and the integral form is converted into a four-dimensional regression format using Softmax and Conv calculations.

[0081] Yolov8 outputs three feature maps with scales of 80*80, 40*40 and 20*20. The Head part outputs feature maps of 6 scales for classification and regression. The category prediction branches and regression prediction branches of 3 different scales are spliced ​​and dimensionally transformed.

[0082] The classification prediction branch performs Sigmoid calculation, while the regression prediction branch needs to be decoded and restored to the original image. The output border is processed using the non-maximum suppression (NMS) algorithm, thereby selecting valid borders with a confidence level greater than 0.6 from the numerous overlapping borders output, which are used as the final recognition results.

[0083] The deep convolutional neural network (CNN) is used to extract and classify the preprocessed video images. The method includes: using OpenPose to output the key points of the human skeleton, the key points of the human skeleton are:

[0084] 0-nose, 1-neck, 2-left eye, 3-right eye, 4-left ear, 5-right ear, 6-left shoulder, 7-right shoulder, 8-left elbow, 9-right elbow, 10-left wrist, 11-right wrist, 12-left hip, 13-right hip, 14-left knee, 15-right knee, 16-left ankle, 17-right ankle.

[0085] In step S03, the OpenPose algorithm is used to first create an input feature map for the input video sequence frame using the VGG-19 convolutional network; then, after the first stage, the two-branch multi-stage CNN is used to obtain the confidence and affinity maps of the key points of the human body, and the first branch predicts the joint point position confidence map. , human body local affinity domain In the second stage, the confidence map and affinity field are parsed greedily, and the corresponding key points are generated for all human bodies in the video frame to obtain the corresponding skeleton graph sequence, each of which has 18 nodes.

[0086] Generated after the second stage , Update as follows:

[0087]

[0088] in, and Respectively represent The forward calculation of the two branch networks in the stage, , Respectively represent The joint confidence map and the local affinity domain of the human body at each stage, Represents the feature map extracted by the VGG-19 convolutional network;

[0089] The ultimate goal of the affinity domain is to obtain the probability that there is a limb between all key points. Through this probability, we can combine key points into limbs, so the final result is a probability value. The affinity domain contains two steps. The first step is to obtain the vector field of partial affinity, which contains a 2D vector field of the direction and value from key point a to key point b. By calculating the vector field, it can be determined whether other key points are in this limb; the other part is to calculate the correlation between the parts. This calculation method is to integrate the result of the first step in the interval ab, and finally solve the probability distribution of all limbs, and distinguish the limbs based on this probability distribution. So the meaning of this affinity domain is the probability that the connection from key point a to key point b belongs to a limb.

[0090] The confidence map represents an evaluation of the visibility of each body part. If it is visible, there is a peak in the confidence map, and the current part is visible. The confidence of the pixel at the joint point is 1, and the surrounding pixels are diffused in a Gaussian distribution according to the distance. The farther the distance, the lower the confidence.

[0091] Everyone's The confidence map of each body part is generated as follows:

[0092]

[0093] in, For the Personal Confidence map of joints, is the position of the pixel, Indicates Personal The position information of the joints, is the standard deviation;

[0094] After obtaining the key point position of each body part, the affinity field is used to determine the current key point ownership. The affinity field confidence is calculated according to the following formula:

[0095]

[0096] in, , are the position coordinates of the two joint points, Represents affinity field, is the interpolation point of the two joint nodes, is the interpolation interval.

[0097] For passenger lying down or falling down, it is only necessary to judge the position relationship between the key points of the human body, the shoulder and the ankle. When the ankle is higher than the shoulder or close to the shoulder, and the aspect ratio of the human body detection frame is greater than 0.6, the passenger is considered to be in a lying state, otherwise he is standing. When the passenger satisfies the lying state and is detected on the seat, the passenger is considered to be lying down, and the voice reminder module immediately gives a voice reminder to the driver. When the passenger is detected on the floor of the car, the passenger is considered to have fallen down, and the voice reminder module immediately gives a voice reminder to the driver, and transmits the analysis results to the subway operation management system, so that the subway operation management personnel can handle it in time to ensure the safety of the passengers.

[0098] Passenger dispute behavior is a continuous action that requires at least two people to participate. When the number of detected people is less than two, the behavior does not exist. When the number of detected people is greater than two, the human skeleton key points extracted above are used to identify disputes using the ST-GCN model of 100 consecutive frames.

[0099] The ST-GCN network consists of two parts: Graph Convolutional Networks (GCN) and Temporal Convolutional Networks (TCN). GCN regards the human skeleton points and their connections as a graph, and then uses the adjacency matrix and degree matrix of the graph to perform convolution operations on the spatial dimension data to obtain spatial structural features. The specific formula is as follows. TCN gradually completes the convolution of each node on all frames through the convolution kernel to obtain the temporal dynamic features of the skeleton point graph under superposition:

[0100]

[0101] in, , is the adjacency matrix, is the identity matrix, is the weight matrix.

[0102] The action video obtains the two-dimensional coordinate data of the human skeleton nodes in each frame through the posture estimation algorithm (OpenPose). According to the connection order, a spatiotemporal graph with joint coordinates is constructed as the input of ST-GCN. Multi-layer spatiotemporal graph convolution operations are performed on the input data to generate a higher-level feature map. Then it is input into the normalized exponential function (SoftMax) classifier for action classification and behavior recognition.

[0103] Further, in step S04, the specific process of the ST-GCN algorithm is as follows:

[0104] Step 1: The human behavior video uses the OpenPose human posture estimation algorithm to obtain the joint point features of all joint points in all frames, which include the spatial coordinates and confidence of the joint points;

[0105] Step 2: According to the spatial coordinates and confidence of the joint points, the relative vectors and bone lengths of each joint point in the same frame are obtained to form the spatial structural features;

[0106] Step 3: According to the spatial coordinates and confidence of the joint points, the joint point movement vectors and movement distances between different frames are obtained to form the temporal dynamic features;

[0107] Step 4: Combine the joint point feature J with all frames T and all related node numbers N to form the original skeleton information sequence;

[0108] Step 5: The spatial structure features obtained in step 2 and the temporal dynamic features obtained in step 3 are respectively combined with all frames T and all related node numbers N to form a new skeleton feature sequence;

[0109] Step 6: Splice the skeleton feature sequences together to form the human body spatiotemporal skeleton features as network input;

[0110] Step 7: Construct spatial graph convolution based on the human body spatiotemporal skeleton;

[0111] Step 8: Construct a temporal convolution module to process temporal information, and form a spatiotemporal unit ST-GCN with the spatial graph convolution in step 7;

[0112] Step 9: Input the human body spatiotemporal skeleton features into ST-GCN, and gradually generate higher-level features through multiple layers of spatiotemporal convolution;

[0113] Step 10: Construct a fully connected layer and return the predicted behavior category through the normalized exponential function (SoftMax) classifier.

[0114] The system calculates the overlap rate of the target person in the current frame for each target person in the previous frame , and calculate the target person's movement speed, analyze the target person's movement state and trajectory, and then judge whether the target person is performing fast and intense body movements by analyzing the position changes of key points. If a passenger is detected to have fast and intense body movements, the voice reminder module will immediately give a voice reminder to the driver and transmit the analysis results to the subway operation management system, so that the subway operation management personnel can handle it in time and ensure the safety of passengers.

[0115] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A method for analyzing passenger behavior in a subway car based on deep learning, characterized in that: The steps include: 1) Real-time collection of video data from surveillance cameras in subway cars; 2) Preprocessing the collected video data, and identifying the categories of human body, seat, and floor through the trained Yolov8 model; the training of the Yolov8 model includes extracting sample video data using the alternate frame method, and sending the sample video data to the Yolov8 model for training; In training, Loss is divided into positive and negative sample allocation strategy and Loss calculation: the positive and negative sample allocation strategy is a matching strategy of the label allocation strategy, which selects positive samples based on the classification and regression scores combined with the weighted scores; 3) Using a deep convolutional neural network to extract and classify features from the preprocessed video images; the feature extraction and classification includes using a VGG-19 convolutional network to create an input feature map for the input video sequence frames; then, through a two-branch multi-stage deep convolutional neural network, the confidence and affinity maps of the key points of the human body are obtained, wherein the first branch predicts the confidence map of the joint point position , human body local affinity domain In the second stage, the confidence map and affinity field are analyzed greedily, and the corresponding key points are generated for all human bodies in the video frame to obtain the corresponding skeleton map sequence. , Update according to the following formula: ； in, and Respectively represent The forward calculation of the two branch networks in the stage, , Respectively represent The joint confidence map and the local affinity domain of the human body at each stage, Represents the feature map extracted by the VGG-19 convolutional network; 4) Accurately identify abnormal passenger behavior using the ST-GCN model; 5) Analyze and compile statistics on the recognition results and generate analysis reports.

2. The method for analyzing subway car passenger behavior based on deep learning according to claim 1 is characterized in that: The Loss calculation includes two branches: classification and regression; First, calculate the loss of the classification branch. The classification branch uses BCE Loss and performs asymmetric weighting operations on positive and negative samples. The formula is as follows: ； in, Represents the overlap ratio between the predicted border and the true border ,when When , it is a positive sample; when When , it is a negative sample, Represents the calculated confidence value for each category, represents the hyperparameter, is an adjustable scale parameter; by assigning Weights are used to reduce the contribution of these samples to the loss. Finally, in order to balance the loss of positive and negative samples, an adjustable ratio parameter is added. Give negative sample loss; Then calculate the Loss of the regression branch. The regression branch uses the distribution focus loss, and the formula is as follows: ； in, Represents the final position of the center point of the prediction box from a certain edge. Represents the distance between the center coordinate of the prediction box and a certain edge of the prediction box. Respectively represent Ceiling and flooring values; Finally, the CIOU Loss is calculated using the following formula: ； in, Represents the overlap ratio between the predicted border and the true border, Represents the Euclidean distance between the center points of two prediction boxes, Represents the diagonal distance between the closed areas of the two prediction boxes, represents the aspect ratio of the real box, Represents the aspect ratio of the prediction box, Represents the consistency of measuring the relative proportions of the real box and the predicted box, is the weight coefficient; The calculated Loss value of the classification branch, the Loss value of the regression branch and the CIOU Loss value are weighted and summed in a weighted ratio to obtain the final Loss value.

3. The method for analyzing subway car passenger behavior based on deep learning according to claim 2 is characterized in that: The confidence map represents an evaluation of the visibility of each body part. If it is visible, there is a peak in the confidence map, and the current part is visible. The confidence of the pixel at the joint point is 1, and the surrounding pixels are diffused in a Gaussian distribution according to the distance. The farther the distance, the lower the confidence. Everyone's The confidence map of each body part is generated as follows: ； in, For the Personal Confidence map of joints, is the position of the pixel, Indicates Personal The position information of the joints, is the standard deviation; After obtaining the key point position of each body part, the affinity field is used to determine the current key point ownership and the affinity field confidence. Follow the formula below: ； in, , are the position coordinates of the two joint points, represents the affinity field, is the interpolation point of the two joint nodes, is the interpolation interval.

4. The method for analyzing subway car passenger behavior based on deep learning according to claim 1, characterized in that: Accurately identifying abnormal passenger behavior using the ST-GCN model includes: Step 1: The human behavior video uses the OpenPose human posture estimation algorithm to obtain the joint point features of all joint points in all frames, which include the spatial coordinates and confidence of the joint points; Step 2: According to the spatial coordinates and confidence of the joint points, the relative vectors and bone lengths of each joint point in the same frame are obtained to form the spatial structural features; Step 3: According to the spatial coordinates and confidence of the joint points, the joint point movement vectors and movement distances between different frames are obtained to form the temporal dynamic features; Step 4: Combine the joint point feature J with all frames T and all related node numbers N to form the original skeleton information sequence; Step 5: The spatial structure features obtained in step 2 and the temporal dynamic features obtained in step 3 are respectively combined with all frames T and all related node numbers N to form a new skeleton feature sequence; Step 6: Splice the skeleton feature sequences together to form the human body spatiotemporal skeleton features as network input; Step 7: Construct spatial graph convolution based on the human body spatiotemporal skeleton; Step 8: Construct a temporal convolution module to process temporal information, and form a spatiotemporal unit ST-GCN with the spatial graph convolution in step 7; Step 9: Input the human body spatiotemporal skeleton features into ST-GCN, and gradually generate higher-level features through multiple layers of spatiotemporal convolution; Step 10: Construct a fully connected layer and return the predicted behavior category through the SoftMax classifier.

5. A subway car passenger behavior analysis device based on deep learning, applying a subway car passenger behavior analysis method based on deep learning according to any one of claims 1 to 4, characterized in that: include: Video acquisition module: used to collect video data from surveillance cameras in subway cars in real time; Video analysis module: pre-processes the collected video data and identifies the categories of human body, seat and floor through Yolov8 model; then uses deep convolutional neural network to extract features and classify the pre-processed video images, and uses ST-GCN model to accurately identify abnormal behaviors of passengers; Voice reminder module: used to give voice reminders to the driver and handle abnormal behaviors of passengers in a timely manner; Analysis and statistics module: used to analyze and count the recognition results and generate analysis reports to facilitate the management and decision-making of subway operation managers.

Images