A Multi-Strategy Method for Detecting Running Behavior on Station Escalators

By using the RMNet-yolov5 model combined with Kalman filtering and the Hungarian algorithm in the escalator scenario at a train station, the problems of detecting overlapping targets and slow inference speed were solved, and real-time running behavior detection and early warning were realized on embedded devices.

CN116129517BActive Publication Date: 2026-11-17JIANGSU AEROSPACE DAWEI TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202211622780.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-16
Publication Date
2026-11-17
Estimated Expiration
2042-12-16

AI Technical Summary

Technical Problem

Existing target detection and multi-target tracking algorithms suffer from problems such as poor detection of overlapping targets, slow inference speed, and difficulty in achieving real-time detection on embedded devices with limited computing power in station escalator scenarios.

Method used

RMNet is used as the feature extraction backbone network of the YOLOv5 model. A target tracking model is constructed by combining Kalman filtering and Hungarian algorithm. The RMNet-YOLOv5 model is optimized by training through knowledge distillation. The non-maximum suppression algorithm based on DIOU is used for target detection. The ROI detection region is designed to confirm the running behavior.

Benefits of technology

It improves the performance and inference speed of target detection, reduces algorithm complexity, realizes real-time detection and early warning on embedded devices, and ensures timely identification of running behavior.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116129517B_ABST
    Figure CN116129517B_ABST
Patent Text Reader

Abstract

This invention discloses a multi-strategy method for detecting running behavior on escalators in railway stations, relating to the field of computer vision technology. The method includes: acquiring escalator surveillance video and converting it into images as an image sample set; using RMNet as the feature extraction backbone network of a YOLOv5 model to improve the inference speed of the target detection algorithm; inputting the image sample set into the improved YOLOv5 model for distillation training to obtain an RMNet-YOLOv5 model for target detection; constructing a target tracking model based on Kalman filtering and the Hungarian algorithm to track the target boxes generated by the RMNet-YOLOv5 model, significantly improving the inference speed of target tracking; and confirming whether the target exhibits running behavior based on the frame difference between the tracked target box entering and exiting a set ROI detection area. This method can accurately and quickly detect passenger running behavior on escalators, providing a good early warning effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-strategy method for detecting running behavior on escalators in train stations. Background Technology

[0002] With the rapid pace of urbanization, escalators have become an important part of public places such as shopping malls and train stations. Especially in train stations, people rushing to catch their trains may run on escalators. Running on escalators is a dangerous behavior that could cause pedestrians to fall or even cause stampedes. Therefore, station staff need to promptly detect and stop such behavior.

[0003] Running behavior detection in station escalator scenarios mainly involves object detection and multi-object tracking techniques. Existing object detection technologies include the YOLO series, with YOLOv5 being a popular algorithm. Station escalators experience peak pedestrian traffic, inevitably leading to object overlap. The existing YOLOv5 algorithm is not very effective at detecting overlapping objects, which is detrimental to subsequent object tracking tasks. Furthermore, considering real-time requirements, the inference speed of YOLOv5 needs to be further improved.

[0004] Popular multi-object tracking algorithms include DeepSort, CenterTrack, and FairMOT. These algorithms, combined with deep neural networks for feature extraction, can accurately track freely moving objects in complex scenes. However, these object tracking algorithms have complex processes and slow inference times. Coupled with the time-consuming inference of object detection, it is difficult to perform real-time detection on embedded devices with limited computing power. Summary of the Invention

[0005] To address the issues of poor real-time tracking and slow inference speed in escalator target tracking, the inventors have proposed a multi-strategy method for detecting running behavior on escalators in train stations. This method can be applied to station video surveillance and early warning systems to promptly alert staff and prevent unnecessary tragedies. The technical solution of this invention is as follows:

[0006] A multi-strategy method for detecting running behavior on station escalators includes the following steps:

[0007] Acquire escalator surveillance video and convert it into images as an image sample set;

[0008] RMNet is used as the feature extraction backbone network of the YOLOv5 model, and the image sample set is input into the improved YOLOv5 model for distillation training to obtain the RMNet-YOLOv5 model for object detection.

[0009] A target tracking model based on Kalman filtering and the Hungarian algorithm is constructed to track the target boxes generated by the RMNet-yolov5 model;

[0010] The presence of running behavior of the target is confirmed by the frame difference between the tracked target bounding box entering and exiting the set ROI detection area.

[0011] Its further technical solution is that, during the training phase, the training set images in the image sample set pass through the first and second convolutional networks in RMNet simultaneously, and then enter the Add layer together for connection, finally outputting the feature extraction image.

[0012] During the inference phase, the test set images in the image sample set are processed by the first convolutional network in RMNet to output feature extraction images;

[0013] The first convolutional network includes three convolutional layers, and the second convolutional network includes one convolutional layer; each convolutional layer and the Add layer in the first convolutional network uses ReLU as the activation function.

[0014] A further technical solution involves inputting the image sample set into the improved YOLOv5 model for distillation training, including the following methods:

[0015] The YOLOv5L model is used as the teacher model in distillation training, and the improved YOLOv5 model is used as the student model in distillation training. The loss function L is designed for distillation training, and the weights of the RMNet-YOLOv5 model are obtained by dynamically modifying the loss function of the student model during training.

[0016] Its further technical solution is that the method for constructing a target tracking model based on Kalman filtering and the Hungarian algorithm includes:

[0017] Kalman filtering is used to predict the state vector of the target box at the next moment, and then the Hungarian algorithm is used to perform cascade matching of the target box's motion trajectory and the state vector at the next moment.

[0018] A further technical solution involves determining whether a target is running based on the frame difference between the tracked target bounding box entering and exiting the set ROI detection area.

[0019] Set a small detection box within the upper and lower boundaries of the defined ROI detection area, denoted as the upper detection box and the lower detection box, respectively;

[0020] Record the frame number when the center point of the target box first enters the uplink detection box and the downlink detection box respectively, and calculate the frame difference between the two; if the frame difference is less than the first set threshold, it is confirmed that the target is running.

[0021] A further technical solution is that the first convolutional network includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence. The size of the convolutional kernels in the first convolutional layer, the third convolutional layer and the second convolutional network is 1*1, and the size of the convolutional kernels in the second convolutional layer is 3*3.

[0022] A further technical solution is that the expression for the loss function L during distillation training is designed as follows:

[0023] L = f obj (o g ,o s )+f cls (c g ,c s )+f bbox (b g ,b s )+λL distill (pred t ,pred s );

[0024] In the formula, pred t ∈{o t ,c t ,b t}, pred s ∈{o s ,c s ,b s};f obj (·), f cls (·), f bbox (·) represent the target loss function, classification loss function, and bounding box loss function of the model, respectively; s c s and b s These represent the target probability, classification probability, and bounding box regression value output by the student model, respectively, and correspond to o. g c g and b g Three label components; o t c t and b t These are the target probability, classification probability, and target detection box regression value output by the teacher model, respectively; L distill (·) represents the distillation loss function, and λ is a weighting coefficient used to control the proportion of the distillation loss function value. The distillation loss function is expressed as:

[0025] L distill (pred t ,pred s )=f obj (o t ,o s )+ot *f cls (c t ,c s )+o t *f bbox (b t ,b s ).

[0026] A further technical solution involves using Kalman filtering to predict the state vector of the target box at the next time step, including:

[0027] In the Kalman filter prediction stage, the state vector and covariance matrix at time k are estimated based on the posterior state vector at time k-1, as follows:

[0028]

[0029] In the formula, a is the state transition matrix, b is the control matrix, and Q is the model noise covariance matrix; Let be the prior state vector at time k. Let be the posterior state vector at time k-1; P is the prior covariance matrix of the prior state vectors at time k, used to describe the relationships between the target states x; k-1 Let u be the posterior covariance matrix at time k-1. k-1 This is the state control vector at time k-1;

[0030] In the Kalman filter update stage, the prior state vector and prior covariance matrix at time k are used to correct the posterior values ​​in the prediction stage, as shown below:

[0031]

[0032] In the formula, K k Let z be the Kalman filter gain matrix at time k; k Let H be the observation vector at time k, which serves as the input to the Kalman filter; H is the transformation matrix from the state vector to the observation vector, and I is the identity matrix.

[0033] A further technical solution involves using the Hungarian algorithm to perform concatenated matching of the target box's trajectory and its state vector at the next moment, including:

[0034] The sum of the Eulerian distances between the prior and posterior state vectors obtained in the Kalman filter is used as the feature cost for the target concatenated matching.

[0035]

[0036] In the formula, This represents the i-th posterior state vector predicted by the Kalman filter at time k. This represents the i-th prior state vector predicted by Kalman filtering at time k;

[0037] If the feature cost is less than the second set threshold, the match is successful. The second set threshold is a hyperparameter for target tracking and needs to be fine-tuned according to different camera angles.

[0038] A further technical solution is that the method also includes:

[0039] The RMNet-yolov5 model uses a DIOU-based nonmaximum suppression algorithm during the inference phase, where:

[0040] DIOU(A,B)=IOU(A,B)-d 2 / l 2 ;

[0041] In the formula, A is the ground truth bounding box, B is the predicted bounding box, IOU(A,B) is the intersection-union ratio of the ground truth bounding box and the predicted bounding box, d is the distance from the center point of the ground truth bounding box to the center point of the predicted bounding box, and l is the length of the diagonal of the minimum bounding rectangle of the ground truth bounding box and the predicted bounding box.

[0042] The beneficial technical effects of this invention are:

[0043] The RMNet-yolov5 model for object detection provided in this application, based on the RMNet network structure, allows for deeper model layers and the extraction of more abstract image features, thereby improving the overall performance of the detection model. Knowledge distillation is used to train the RMNet-yolov5 model, enabling the lightweight student model to achieve performance similar to the teacher model. During inference, the model employs a DIOU-based non-maximum suppression algorithm, which is more suitable for target overlap detection on escalators and train stations. A suitable target tracking model is designed for application scenarios like escalators and train stations where the target object's movement trajectory is relatively simple, reducing algorithm complexity and improving inference speed. Finally, the difference in the number of frames the tracked target box enters and exits the set ROI detection area confirms whether the target is running, thus completing the detection. Attached Figure Description

[0044] Figure 1 This is a flowchart of the multi-strategy escalator running behavior detection method provided in this application.

[0045] Figure 2 The present application provides the RMNet network structure, wherein: (a) is the RMNet network structure during the training phase, and (b) is the RMNet network structure during the inference phase.

[0046] Figure 3This is a schematic diagram of the escalator running behavior recognition provided in this application.

[0047] Figure 4 This is a comparison chart of the inference speed of the RMNet-yolov5 model provided in this application and the original yolov5 model during the object detection stage.

[0048] Figure 5 This is a comparison chart of the inference speed of the target tracking model provided in this application and the traditional DeepSort model during the target tracking stage. Detailed Implementation

[0049] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0050] Please refer to Figure 1 As shown, this embodiment provides a multi-strategy method for detecting running behavior on escalators in train stations, including the following steps:

[0051] Step 1: Obtain the escalator monitoring video and convert it into images as an image sample set.

[0052] Step 1.1: This embodiment uses self-collected surveillance videos of Wuxi Metro escalators. Multiple video segments are cut into images, with 2000 images forming the training set and 500 images forming the test set. These images are used to train the object detection model constructed below, with a training iteration count of 300.

[0053] Step 1.2: Preprocess the training and test set images. The purpose of preprocessing is to convert the original images from the surveillance video into input images suitable for the model. Bilinear interpolation is used to compress the images to a uniform size suitable for the model input, and the pixel values ​​are normalized to the range of 0 to 1. Finally, a Gaussian filter is used to smooth image noise.

[0054] Step 2: Construct a target detection model based on the YOLOv5 model.

[0055] In this embodiment, the improvements to the YOLOv5 model include the following three points, aiming to improve performance while ensuring real-time detection. Simultaneously, accurate object detection will improve the accuracy of subsequent object tracking models, specifically including:

[0056] <1> Reparameterized RMNet was used as the feature extraction backbone network for the YOLOv5 model.

[0057] like Figure 2 As shown in (a), during the training phase, the training images in the image sample set simultaneously pass through the first and second convolutional networks in RMNet, and then jointly enter the Add layer for connection, ultimately outputting the feature extraction image. Figure 2As shown in (b), during the inference phase, the test set images in the image sample set are processed by the first convolutional network in RMNet to output feature extraction images. The first convolutional network consists of a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence. The second convolutional network consists of one convolutional layer. The kernel size of the convolutional layers in the first, third, and second convolutional networks is 1*1, while the kernel size of the second convolutional layer is 3*3. ReLU is used as the activation function for each convolutional layer and the Add layer in the first convolutional network.

[0058] The RMNet network structure is equivalent to the training phase in the inference phase, but it can improve the inference speed of the model. In addition, thanks to the residual modules used by the RMNet feature extraction network in the training phase, the model can have deeper layers and extract more abstract image features, thereby improving the overall performance of the detection model.

[0059] <2> Input the image sample set into the first... <1> The improved YOLOv5 model is trained by distillation to obtain the RMNet-YOLOv5 model for object detection.

[0060] Knowledge distillation is a common method for model compression. This method utilizes the supervision information of a large, high-performance model to train a lightweight smaller model. Because it possesses the prior knowledge of the larger model, the smaller model can achieve performance similar to the larger model. To further improve model performance, this step uses distillation to optimize the YOLOv5 model that used RMNet as the feature extraction backbone in the previous step. Specifically, the YOLOv5L model is used as the teacher model in distillation training, and the improved YOLOv5 model is used as the student model. A loss function L is designed for distillation training, and the performance of the student model is improved by dynamically modifying the student model's loss function during training, resulting in the RMNet-YOLOv5 model weights.

[0061] The expression for the loss function L during distillation training is designed as follows:

[0062] L = f obj (o g ,o s )+f cls (c g ,c s )+f bbox (b g ,b s )+λL distill (pred t , pred s (1)

[0063] In the formula, pred t ∈{o t ,c t ,b t}, pred s ∈{o s ,c s ,b s};f obj (·), f cls (·), f bbox (·) represent the target loss function, classification loss function, and bounding box loss function of the model, respectively; s c s and b s These represent the target probability, classification probability, and bounding box regression value output by the student model, respectively, and correspond to o. g c g and b g Three label components; o t c t and b t These are the target probability, classification probability, and target detection box regression value output by the teacher model, respectively; L distill (·) represents the distillation loss function, and λ is a weighting coefficient used to control the proportion of the distillation loss function value. The distillation loss function is expressed as:

[0064] L distill (pred t , pred s )=f obj (o t ,o s )+o t *f cls (c t ,c s )+o t *f bbox (b t ,b s (2)

[0065] Optionally, in this embodiment, the binary cross-entropy loss function is selected as the target loss function f. obj (·), classification loss function f cls (·) is represented as:

[0066]

[0067]

[0068] in:

[0069] target box loss function f bbox (·) is represented as: fbbox (b g ,b s ) = 1 - IOU(b g ,b s ), where the specific expression of IOU(·) is consistent with equation (3).

[0070] <3> The RMNet-yolov5 model uses a nonmaximum suppression algorithm based on DIOU during the inference phase.

[0071] Non-Maximum Suppression (NMS) is a crucial post-processing step in object detection models. Traditional NMS algorithms use Intersection over Union (IOU) as the evaluation criterion. The IOU between the ground truth bounding box A and the predicted bounding box B is calculated as shown in Equation (3):

[0072] x topC =max(x topA ,x topB ), y topC =max(y topA ,y topB );

[0073] x bottomC =max(x bottomA ,x bottomB ), y bottomC =max(y bottomA ,y bottomB );

[0074] s cross =(x bottomC -x topC )*(y bottomC -y topC );

[0075] s union =(x bottomA -x topA )*(y bottomA -y topA )+(x bottomB -x topB )*(y bottomB -y topB )-s cross ;

[0076] IOU(A,B)= s cross / s union (3)

[0077] In the formula, (x topA ,y topA ), (xtopB ,y topB (x) represents the top-left corner coordinates of the ground truth bounding box A and the predicted bounding box B, respectively. bottomA ,y bottomA ), (x bottomB ,y bottomB (x) represents the lower right corner coordinates of the ground truth bounding box A and the predicted bounding box B, respectively. topC ,y topC ), (x bottomC ,y bottomC ) represent the coordinates of the upper left corner and the lower right corner of the intersection region C of A and B, respectively.

[0078] The IOU metric ignores the positional information when two detection boxes intersect. If the true target box A is large and contains the predicted target box B, then regardless of how the predicted target box B moves, their IOU value remains unchanged. This method is not suitable for detecting overlapping objects, which frequently occurs when detecting pedestrians on escalators. Therefore, this application uses DIOU as the metric for the NMS algorithm, expressed as:

[0079] DIOU(A,B)=IOU(A,B)-d 2 / l 2 (4)

[0080] In the formula, d is the distance from the center point of the true target box to the center point of the predicted target box, and l is the length of the diagonal of the minimum bounding rectangle of the true target box and the predicted target box.

[0081] Step 3: Construct a target tracking model based on Kalman filtering and the Hungarian algorithm to track the target boxes generated by the RMNet-yolov5 model.

[0082] Step 3.1: Use Kalman filtering to predict the state vector of the target box at the next time step.

[0083] Considering that the application scenario is a train station escalator, the trajectory of the target object is relatively simple and predictable, thus requiring less sophisticated target tracking algorithms. Therefore, this application uses Kalman filtering for target bounding box prediction. Kalman filtering is a classic optimal estimation method, which can be divided into a prediction stage and an update stage.

[0084] In the Kalman filter prediction stage, the state vector and covariance matrix at time k are estimated based on the posterior state vector at time k-1, as follows:

[0085]

[0086] In the formula, a is the state transition matrix, b is the control matrix, and Q is the model noise covariance matrix; Let be the prior state vector at time k. Let be the posterior state vector at time k-1; P is the prior covariance matrix of the prior state vectors at time k, used to describe the relationships between the target state vectors x; k-1 Let u be the posterior covariance matrix at time k-1. k-1 This is the state control vector at time k-1.

[0087] In the Kalman filter update stage, the prior state vector and prior covariance matrix obtained at time k are used to correct the posterior values ​​in the prediction stage, as expressed in:

[0088]

[0089] In the formula, K k Let z be the Kalman filter gain matrix at time k; k Let H be the observation vector at time k, which serves as the input to the Kalman filter; H is the transformation matrix from the state vector to the observation vector, and I is the identity matrix.

[0090] After the Kalman filter predicts the prior state vector x for the next frame, target matching needs to be performed on the predicted x, proceeding to step 3.2.

[0091] Step 3.2: Use the Hungarian algorithm to perform concatenated matching of the target box's motion trajectory and the state vector at the next moment.

[0092] In the target matching stage, this application uses the Hungarian algorithm for cascaded matching. This algorithm can effectively associate trajectories and target objects, even when the target is briefly occluded. To better ensure the real-time performance of the entire algorithm, this application does not use features extracted by a neural network for target matching like the DeepSort algorithm, but instead uses the prior state vector obtained in Kalman filtering. and posterior state vector The sum of the Eulerian distances between them is used as the feature cost for the target cascade matching:

[0093]

[0094] In the formula, This represents the i-th posterior state vector predicted by the Kalman filter at time k. Let represent the i-th prior state vector predicted by Kalman filtering at time k.

[0095] If the feature cost is less than the second set threshold, the match is successful. The second set threshold is a hyperparameter for target tracking and needs to be fine-tuned according to different camera angles.

[0096] In this embodiment, the six values ​​of the target bounding box are selected to form the state vector x in equations (5) and (6):

[0097] x=(x top ,y top ,x cen ,y cen ,ratio,h);

[0098] The above six parameters are respectively represented as: the horizontal and vertical coordinates of the upper left corner of the target box, the horizontal and vertical coordinates of the center point of the target box, the aspect ratio of the target box, and the length of the target box. The covariance matrix P represents the relationship between these six quantities. Substituting these six quantities and the covariance matrix P into equation (5), the state vector of the current frame can be predicted based on the posterior state vector of the previous frame. Substituting into equation (6) completes the update of the state vector x and the covariance matrix P. The updated state can be used to predict the state of the next frame in the form of the prior state. Then, substituting into equation (7) calculates the sum of the Eulerian distance between the prior state vector and the posterior state vector. In this application, the center point of the prior box and the posterior box is selected to calculate the Eulerian distance, that is, equation (7) is rewritten as: In addition, based on multiple tests, 90 pixels was selected as the second set threshold. That is, if the value is greater than 90 pixels, the target match will fail; otherwise, the match will succeed.

[0099] The biggest advantage of the proposed target tracking model is that it significantly improves the running speed by using Euclidean distance instead of neural networks for feature extraction. The trade-off is a decrease in target tracking accuracy. However, considering that: firstly, the trajectory of passengers in escalators is relatively simple and predictable; and secondly, a high-performance target detection algorithm can effectively assist subsequent target tracking algorithms, the impact of this target tracking algorithm on the overall algorithm performance is relatively small.

[0100] Step 4: Confirm whether the target is running based on the frame difference between the tracked target bounding box entering and exiting the set ROI detection area.

[0101] Step 4.1: As Figure 3 As shown, a small detection frame is set within the upper boundary 1 and lower boundary 2 of the defined ROI detection area, denoted as the upper detection frame 3 and the lower detection frame 4, respectively. Optionally, the ratio of the range of a single detection frame to the range of the ROI detection area is generally controlled at 1:20.

[0102] Step 4.2: Record the number of frames when the center point of the target box 5 first enters the uplink detection box 3 and the number of frames when it first enters the downlink detection box 4. This number of frames can be approximated as the time when a pedestrian enters or leaves the ROI detection area.

[0103] Step 4.3: Calculate the frame rate difference f between the two. diff If fdiff If the value is less than the first set threshold, then the target (i.e., the pedestrian) is confirmed to be running.

[0104] Table 1 shows the performance of the original YOLOv5 and the improved YOLOv5 provided in this embodiment on 500 test set images. Here, mAP@.5 represents the average AP value of the model on the test set images when the IOU threshold is greater than 0.5. The AP value is the area under the PR (precision-recall) curve. mAP@.5:.95 represents the average AP value of the model when the IOU threshold is between 0.5 and 0.95. As can be seen from the comparison in Table 1, the improved YOLOv5 provided in this embodiment outperforms the original YOLOv5.

[0105] Table 1

[0106]

[0107] In this embodiment, the above detection method is run on an NVIDIA Jetson Xavier NX embedded device. The TensorRT version running on this device is 7.1.3.1. Figure 4 and Figure 5 The graphs show a comparison of the inference speed of the detection method in the target detection and target tracking stages on the embedded device. The horizontal axis represents the frame number of the test video, and the vertical axis represents the inference speed per frame. The improved YOLOv5 target detection model provided in this embodiment has an overall inference speed that is lower than the original YOLOv5 model. Furthermore, in the target tracking stage, the target tracking model provided in this embodiment has an overall inference speed that is significantly faster than the DeepSort algorithm.

[0108] The above descriptions are merely preferred embodiments of this application, and the present invention is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.

Claims

1. A multi-strategy station escalator running behavior detection method, characterized in that, The method includes: Acquire escalator surveillance video and convert it into images as an image sample set; RMNet is used as the feature extraction backbone network of the YOLOv5 model. During the training phase, the training images in the image sample set pass through the first and second convolutional networks in RMNet and then enter the Add layer for connection to output the feature extraction image. During the inference phase, the test images in the image sample set pass through the first convolutional network of RMNet to output the feature extraction image. The first convolutional network includes three convolutional layers, and the second convolutional network includes one convolutional layer; each convolutional layer and the Add layer in the first convolutional network uses ReLU as the activation function. The image sample set is input into the improved yolov5 model for distillation training to obtain the RMNet-yolov5 model for object detection; A target tracking model based on Kalman filtering and Hungarian algorithm is constructed to track the target boxes generated by the RMNet-yolov5 model. The presence of running behavior of the target is confirmed based on the frame difference between the tracked target bounding box entering and exiting the set ROI detection area, including: Set up an uplink detection box and a downlink detection box within the upper and lower boundaries of the defined ROI detection area, respectively. Record the frame number when the center point of the target box first enters the uplink detection box and the downlink detection box, and calculate the frame difference. If the frame difference is less than the first set threshold, it is confirmed that the target is running.

2. The multi-strategy escalator running behavior detection method according to claim 1, characterized in that, The method of inputting the image sample set into the improved YOLOv5 model for distillation training includes: Using the YOLOv5L model as the teacher model in distillation training and the improved YOLOv5 model as the student model in distillation training, a loss function for distillation training is designed. L The weights of the RMNet-yolov5 model are obtained by dynamically modifying the loss function of the student model during training.

3. The multi-strategy escalator running behavior detection method according to claim 1, characterized in that, The method for constructing a target tracking model based on Kalman filtering and the Hungarian algorithm includes: Kalman filtering is used to predict the state vector of the target box at the next moment, and then the Hungarian algorithm is used to perform cascade matching of the target box's motion trajectory and the state vector at the next moment.

4. The multi-strategy method for detecting running behavior on station escalators according to claim 1, characterized in that, The first convolutional network includes a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence. The size of the convolutional kernels in the first convolutional layer, the third convolutional layer, and the second convolutional network is 1*1, and the size of the convolutional kernels in the second convolutional network is 3*3.

5. The multi-strategy method for detecting running behavior on station escalators according to claim 2, characterized in that, Loss function during distillation training L The expression is designed as follows: L = f obj ( o g , o s )+ f cls ( c g , c s )+ f bbox ( b g , b s )+ λL distill ( pred t , pred s ); In the formula, pred t ∈{ o t , c t , b t }, pred s ∈{ o s , c s , b s }; f obj (·) f cls (·) f bbox (·) represent the target loss function, classification loss function, and bounding box loss function of the model, respectively; o s , c s and b s These represent the target probability, classification probability, and bounding box regression value output by the student model, respectively, and correspond to... o g , c g and b g Three label components; o t , c t and b t These are the target probability, classification probability, and target detection box regression value output by the teacher model, respectively. L distill (·) represents the distillation loss function. λ These are weighting coefficients used to control the proportion of the distillation loss function value, which is expressed as: L distill ( pred t , pred s )= f obj ( o t , o s )+ o t * f cls ( c t , c s )+ o t * f bbox ( b t , b s )。 6. The multi-strategy escalator running behavior detection method according to claim 3, characterized in that, The method for predicting the state vector of the target box at the next time step using Kalman filtering includes: In the Kalman filter prediction stage, according to the first... k The posterior state vector at time -1 is used to estimate the 6th time step. k The state vector and covariance matrix at time step are expressed as follows: ; In the formula, a Here is the state transition matrix. b For the control matrix, Q This is the model noise covariance matrix; For the first k The prior state vector at time 1. For the first k The posterior state vector at time -1; For the first k The prior covariance matrix of the prior state vector at time t is used to describe the target state. x The relationship between them; For the first k The posterior covariance matrix at time -1 For the first k The state control vector at time -1; In the Kalman filter update phase, the first... k The prior state vector and prior covariance matrix at time step 1 are used to correct the posterior values ​​in the prediction stage, respectively, as follows: ; In the formula, K k For the first k Kalman filter gain matrix at time t; z k For the first k The observation vector at time t is used as the input to the Kalman filter; H This is the transformation matrix from the state vector to the observation vector. I It is an identity matrix.

7. The multi-strategy escalator running behavior detection method according to claim 3, characterized in that, The method for concatenating and matching the motion trajectory of the target box and the state vector at the next moment using the Hungarian algorithm includes: The sum of the Eulerian distances between the prior and posterior state vectors obtained from Kalman filtering is used as the feature for target cascade matching. cost : ; In the formula, Indicates in k The Kalman filter prediction at time 1 i A posterior state vector Indicates in k The Kalman filter prediction at time 1 i A prior state vector; If features cost If the value is less than the second set threshold, it indicates a successful match. The second set threshold is a hyperparameter for target tracking and needs to be fine-tuned according to different camera angles.

8. The multi-strategy method for detecting running behavior on station escalators according to any one of claims 1-7, characterized in that, The method further includes: The RMNet-yolov5 model uses a DIOU-based nonmaximum suppression algorithm during the inference phase, where: DIOU ( A , B )= IOU ( A , B )- d 2 / l 2 ; In the formula, A For the true target bounding box, B To predict the target bounding box, IOU ( A , B ) represents the intersection-union ratio (IoU) of the ground truth bounding boxes and the predicted bounding boxes. d The distance from the center point of the ground truth bounding box to the center point of the predicted bounding box. l It is the length of the diagonal of the minimum bounding rectangle between the true and predicted target boxes.

Citation Information

Patent Citations

  • Escalator passenger retrograde motion detection method based on deep learning

    CN111144247A

  • Road moving target detection method based on trajectory prediction

    CN111340855A

  • Target detection method and target detection terminal based on knowledge distillation

    CN113743514A