Escalator monitoring video anomaly detection method and system

Abstract

The application discloses an escalator monitoring video anomaly detection method and system, and relates to the field of escalator detection.The application obtains the key point heat map corresponding to each frame of picture in the data set by combining the trained YOLOv5 target detection model and the HR-Net key point extraction model, and obtains the key point interframe change graph corresponding to the key point heat map.The application obtains the convolutional neural network model by training the convolutional neural network with the image label containing the key point interframe change graph and the label corresponding to the key point interframe change graph.When starting detection, the picture set of the escalator monitoring video to be detected is input into the YOLOv5 target detection model frame by frame, the pedestrian target frame position corresponding to each frame of picture is obtained, and the key point heat map corresponding to each pedestrian target frame position is predicted by the HR-Net key point extraction model.The key point interframe change graph corresponding to the key point heat map is input into the convolutional neural network model to predict the label corresponding to the key point interframe change graph, and the behavior state on the escalator is obtained through the label, which realizes the intelligent detection of the escalator monitoring video anomaly.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of escalator detection, and in particular to an escalator monitoring video anomaly detection method and system. BACKGROUND

[0002] With the advancement of urbanization construction and the improvement of people's material level, urban infrastructure is constantly improving, and people also pay more attention to the safety of these facilities. In subway stations, shopping malls and other public places, escalators are widely used. However, due to equipment problems, improper behavior of users, and other reasons, escalator-related safety accidents are also on the rise. According to news reports, behaviors such as reversing, running, falling, or carrying strollers and large luggage on escalators often lead to safety accidents, causing passengers to be injured. Timely, accurate, and efficient monitoring and alarm of elevator abnormal behavior can help quickly respond to accidents, avoid casualties, and improve emergency response levels.

[0003] To improve the level of anomaly detection, scholars have made some research and exploration. Shao Haibo proposed an escalator safety monitoring system, authorized publication number CN205257749U, including a first camera group for shooting the overall image of the escalator and the passengers on the escalator, a second camera group for shooting the mechanical parts of the escalator, and a data processing device for image analysis and processing. Lu Jianzhuo et al. proposed an escalator safety monitoring device, authorized publication number CN204310668U, which separates the monitoring device from the control system and can be easily and flexibly used in different control systems, facilitating modular design of the control system. However, these inventions lack automated algorithm design (or intelligent detection design) and still focus only on information collection, without involving the joint application of YOLOv5 model, HR-Net model, and convolutional neural network model in the field of escalator monitoring video anomaly detection. The present application is designed to ensure escalator safety by combining YOLOv5 model, HR-Net model, and convolutional neural network model to achieve intelligent detection of escalator monitoring video anomalies, while greatly improving the accuracy of detection. SUMMARY

[0004] To achieve intelligent detection of escalator monitoring video anomalies and improve the accuracy of escalator monitoring video anomaly detection, the present application combines YOLOv5 model, HR-Net model, and convolutional neural network model to propose an escalator monitoring video anomaly detection method, which includes:

[0005] Obtaining monitoring video of the escalator, converting it into a picture set continuous in time points, and judging whether the current frame is a normal frame frame by frame, if yes, marking it as a positive sample, if no, marking it as a negative sample, to obtain a model data set containing positive samples and negative samples; the positive sample is specifically a picture marked with a normal label, and the negative sample is specifically a picture marked with an abnormal label;

[0006] The YOLOv5 model and the HR-Net model are trained respectively through a large-scale data set to obtain a corresponding YOLOv5 target detection model and an HR-Net key point extraction model;

[0007] The pictures in the model data set are input into the YOLOv5 target detection model frame by frame to obtain the pedestrian target frame position corresponding to each picture; the key point heat map corresponding to each pedestrian target frame position is predicted through the HR-Net key point extraction model;

[0008] For each key point heat map, the forward difference map and the backward difference map corresponding thereto are obtained, and the key point inter-frame change map corresponding to the current key point heat map is obtained by taking the sum of the forward difference map and the backward difference map;

[0009] An image label pair corresponding to each key point inter-frame change map is obtained, and a convolutional neural network is trained through the image label pair to obtain a convolutional neural network model; the image label pair includes the key point inter-frame change map and the label corresponding thereto;

[0010] The picture set of the escalator monitoring video to be detected is input into the YOLOv5 target detection model frame by frame to obtain the pedestrian target frame position corresponding to each picture, and the key point heat map corresponding to each pedestrian target frame position is predicted through the HR-Net key point extraction model, the key point inter-frame change map corresponding to each key point heat map is obtained, and the key point inter-frame change map is input into the convolutional neural network model to predict the label corresponding thereto.

[0011] Further, the key point heat map corresponding to each pedestrian target frame position is predicted through the HR-Net key point extraction model, specifically: the pedestrian target frame position is input into the HR-Net key point extraction model in sequence, so as to obtain a high-resolution feature map including human key points and human key point rectangular bounding box confidence through the HR-Net key point extraction model, and the human key point rectangular bounding box with a confidence higher than a set threshold in the high-resolution feature map is subjected to human pose estimation to obtain the pixel coordinates and the predicted confidence of the human key points, and further obtain the key point heat map corresponding to the pedestrian target frame position.

[0012] Further, the large-scale data set is the MS COCO data set; the YOLOv5 model is trained through the large-scale data set, specifically including:

[0013] The YOLOv5 model is trained through the MS COCO data set, and the loss function is used to train the YOLOv5 model during the training process The regression branch of the YOL0v5 model is trained, and the target and class branch of the YOL0v5 model is trained through the BCE loss function.

[0014] Further, the loss function The formula expression is:

[0015]

[0016] In the formula:

[0017]

[0018]

[0019] Where, Intersection(A, B) represents the intersection area of the prediction box A of the YOLOv5 model and the target box B, Union(A, B) represents the union area of the prediction box A of the YOLOv5 model and the target box B; b, b gt respectively represent the center points of the prediction box and the real box, p 2 (b, bg t ) is the Euclidean distance between the center points of the prediction box and the real box, c is the diagonal distance of the smallest closed region that can contain the prediction box and the real box, and alpha is the loss weight; represents the loss value; w gt is the width of the real box, h gt is the height of the real box, w is the width of the prediction box, and h is the height of the prediction box.

[0020] The formula expression of the BCE loss function is:

[0021]

[0022] In the formula, p is the probability that the prediction sample of the YOLOv5 model is a positive sample; is the label of the sample, which takes the value 1 when the sample belongs to a positive sample, otherwise takes the value 0, and BCELoss is the loss value.

[0023] Further, when training the YOLOv5 model through a large-scale data set, it further includes:

[0024] For the picture data in the large-scale data set, the picture input to the YOLOv5 model or the HR-Net model is randomly flipped, and Mosaic data enhancement is used, specifically: any four pictures are spliced to obtain new pictures after splicing and added to the training to expand the data set.

[0025] Further, the acquisition formula of the forward difference map is BDI k = |H k-1 -H k |;

[0026] The acquisition formula of the backward difference map is FDI k = |H k -H k+1 |;

[0027] Wherein, the value range of k is (1, n-2); in the formula, n represents the total frame number of the key point heat map, H k represents the kth key point heat map, H k-1 represents the k-1th key point heat map, H k+1 represents the k+1th key point heat map, BDI k the forward difference map corresponding to the kth key point heat map, FDI k represents the backward difference map corresponding to the kth key point heat map.

[0028] The acquisition formula of the key point interframe change map is CDI k = BDI k ∪ FDI k , in the formula, CDI k represents the key point interframe change map.

[0029] The application further provides an escalator monitoring video anomaly detection system, comprising:

[0030] A data set acquisition module is configured to acquire monitoring videos of an escalator, convert the monitoring videos into a picture set continuous in time points, and judge whether a current frame is a normal frame frame by frame, if yes, mark the current frame as a positive sample, and if not, mark the current frame as a negative sample, to obtain a model data set containing positive samples and negative samples; the positive sample is specifically a picture marked with a normal label, and the negative sample is specifically a picture marked with an abnormal label;

[0031] A first training module is configured to train a YOLOv5 model and an HR-Net model respectively through a large-scale data set to obtain a corresponding YOLOv5 target detection model and an HR-Net key point extraction model;

[0032] A key point heat map acquisition module is configured to input pictures in the model data set into the YOLOv5 target detection model frame by frame to obtain pedestrian target frame positions corresponding to the pictures; and predict key point heat maps corresponding to the pedestrian target frame positions through the HR-Net key point extraction model;

[0033] An inter-frame change map acquisition module is configured to, for each frame key point heat map, acquire a corresponding forward difference map and a backward difference map, and obtain a key point inter-frame change map corresponding to the current frame key point heat map by performing an OR operation on the forward difference map and the backward difference map.

[0034] A second training module is configured to acquire an image label pair corresponding to each key point inter-frame change map, train a convolutional neural network through the image label pair, and obtain a convolutional neural network model; the image label pair includes the key point inter-frame change map and a label corresponding to the key point inter-frame change map.

[0035] A detection module is configured to input a set of escalator monitoring video pictures to be detected into a YOLOv5 target detection model frame by frame, obtain a pedestrian target frame position corresponding to each frame picture, predict a key point heat map corresponding to each pedestrian target frame position through an HR-Net key point extraction model, acquire a key point inter-frame change map corresponding to each frame key point heat map, and input the key point inter-frame change map into the convolutional neural network model to predict a label corresponding to the key point inter-frame change map.

[0036] Further, the HR-Net key point extraction model is configured to: input each pedestrian target frame position into the HR-Net key point extraction model in sequence, obtain a high-resolution feature map including human key points and a human key point rectangular bounding box confidence through the HR-Net key point extraction model, perform human posture estimation on a human key point rectangular bounding box with a confidence higher than a set threshold in the high-resolution feature map, obtain a pixel coordinate and a predicted confidence of the human key point, and further obtain a key point heat map corresponding to the pedestrian target frame position.

[0037] Further, the large-scale data set is an MS COCO data set; and the YOLOv5 model is trained through the MS COCO data set, and specifically includes: training the YOLOv5 model through the MS COCO data set, and using a loss function The regression branch of the YOLOv5 model is configured to train a target and a category branch of the YOLOv5 model through a BCE loss function.

[0038] Further, when the YOLOv5 model is trained through the large-scale data set, the method further includes:

[0039] For picture data in the large-scale data set, a picture input into the YOLOv5 model or the HR-Net model is randomly flipped, and Mosaic data enhancement is used, specifically including: splicing any four pictures to obtain a new picture after splicing and adding the new picture to training, so as to expand the data set.

[0040] Compared with the prior art, the present application has at least the following beneficial effects:

[0041] (1) The application obtains the key point heat map corresponding to each frame of picture in the data set by jointly training the YOLOv5 target detection model and the HR-Net key point extraction model, and obtains the key point inter-frame change graph corresponding to the key point heat map, trains a convolutional neural network by using the image label containing the key point inter-frame change graph and the label corresponding to the key point inter-frame change graph, obtains a convolutional neural network model, and when starting detection, inputs the escalator monitoring video picture set to be detected into the YOLOv5 target detection model frame by frame, obtains the pedestrian target frame position corresponding to each frame of picture, and predicts the key point heat map corresponding to each pedestrian target frame position by using the HR-Net key point extraction model, inputs the key point inter-frame change graph corresponding to the key point heat map into the convolutional neural network model to predict the label corresponding to the key point inter-frame change graph, and obtains the behavior state (abnormal or normal) on the escalator through the label, which realizes the intelligent detection of the escalator monitoring video anomaly, and greatly improves the accuracy of the abnormal detection;

[0042] (2) Specifically, the application locates the pedestrian in the picture by using the YOLOv5 target detection model, extracts the human key points by using the HR-Net key point extraction model, jointly obtains the key point heat map, and obtains the key point inter-frame change graph by using the inter-frame relationship, so as to capture the abnormal behavior and reflect the inter-frame change of the human posture, which greatly improves the accuracy of the abnormal detection;

[0043] (3) The application obtains the key point inter-frame change graph corresponding to the key point heat map of the current frame by taking or of the forward difference graph and the backward difference graph, so that the inter-frame change of the human action is very quickly and conveniently obtained, which is beneficial to capture the abnormal action and speeds up the speed of the abnormal detection;

[0044] (4) When starting detection, the application only needs a small convolutional neural network model to realize the detection after obtaining the key point heat map, which greatly improves the detection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 It is a flow chart of an escalator monitoring video anomaly detection method.

[0046] Figure 2 It is a module diagram of an escalator monitoring video anomaly detection system.

[0047] Figure 3 It is a network structure diagram of the HR-Net model. DETAILED DESCRIPTION

[0048] The following is a specific embodiment of the application combined with the drawings, which further describes the technical solutions of the application, but the application is not limited to these embodiments.

[0049] Embodiment one

[0050] In order to realize intelligent detection of escalator monitoring video, as shown in the background of the present application, Figure 1 The present application provides an escalator monitoring video anomaly detection method, comprising:

[0051] Obtain the monitoring video of the escalator through a camera or other sensing equipment, convert it into a picture set continuous in time points, and judge whether the current frame is a normal frame through artificial frame-by-frame judgment. If yes, it is labeled as a positive sample, and if not, it is labeled as a negative sample, to obtain a model data set containing positive samples and negative samples. The positive sample is specifically a picture labeled with a normal label, and the negative sample is specifically a picture labeled with an abnormal label.

[0052] It should be noted that generally, reverse, falling, baby stroller, large luggage and other rare abnormal situations will be manually labeled as negative samples.

[0053] Train YOLOv5 model (specifically three detection head YOLOv5s) and HR-Net model through a large-scale data set to obtain corresponding YOLOv5 target detection model and HR-Net key point extraction model.

[0054] The large-scale data set is MS COCO data set; split the MS COCO data set to obtain a training set and a validation set, and train the YOLOv5 model through the large-scale data set (before training, the images in the data set need to be unified to a specified size), specifically:

[0055] Train the YOLOv5 model through the training set, and train the regression branch of the YOLOv5 model through the loss function during the training process, train the target and category branch of the YOLOv5 model through the BCE loss function, and verify the model obtained by training through the validation set, and take the model with the best performance on the validation set in all rounds as the yolov5 target detection model.

[0056] The formula expression of the loss function is:

[0057]

[0058] In the formula:

[0059]

[0060]

[0061] Where, Intersection(A, B) represents the intersection area of the prediction box A of the YOLOv5 model and the target box B, and Union(A, B) represents the union area of the prediction box A of the YOLOv5 model and the target box B; b, bgt respectively represent the center points of the prediction box and the real box, and ρ 2 (b, b gt ) is the Euclidean distance between the center points of the prediction box and the real box, c is the diagonal distance of the smallest closed region that can contain the prediction box and the real box, and a is the loss weight; represents the loss value; w gt is the width of the real box, and h gt is the height of the real box, w is the width of the prediction box, and h is the height of the prediction box;

[0062] The formula expression of the BCE loss function is:

[0063]

[0064] In the formula, p is the probability that the YOLOv5 model predicts the sample to be a positive sample; is the label of the sample, which is 1 when the sample belongs to a positive sample, and 0 otherwise, and BCELoss is the loss value.

[0065] It should be noted that the loss function used to train the HR-Net model is:

[0066]

[0067] wherein is the real pixel coordinate value of the sample, y is the predicted pixel coordinate value of the sample, m is the total number of pixels of the sample, and MSELoss is the loss value.

[0068] When training the YOLOv5 model through a large-scale dataset, it also includes:

[0069] For the picture data in the large-scale dataset, the picture input to the YOLOv5 model or the HR-Net model is randomly flipped, and Mosaic data enhancement is used, specifically: any four pictures are spliced to obtain a new picture after splicing and added to the training to expand the dataset.

[0070] The pictures in the model dataset are input frame by frame into the YOLOv5 target detection model to obtain the corresponding pedestrian target box position of each frame of picture; the HR-Net key point extraction model is used to predict the key point heat map corresponding to each pedestrian target box position;

[0071] The HR-Net key point extraction model is used for predicting the key point heat map corresponding to each pedestrian target frame position, specifically: each pedestrian target frame position is sequentially input into the HR-Net key point extraction model, so as to obtain a high-resolution feature map including human body key points and human body key point rectangular bounding box confidence through the HR-Net key point extraction model, and the human body key point rectangular bounding box with a confidence higher than a set threshold in the high-resolution feature map is subjected to human body posture estimation, so as to obtain pixel coordinates and a predicted confidence of the human body key point, and then the key point heat map corresponding to the pedestrian target frame position is obtained.

[0072] As shown in the detailed explanation, the HR-Net key point extraction model starts from a high-resolution subnetwork, gradually adds one by one from high to low resolution subnetworks, and then connects the multi-resolution subnetworks in parallel; in the whole process, information is exchanged in the parallel multi-resolution subnetworks again and again, so as to complete a repeated multi-scale fusion process, so as to obtain a high-resolution feature map, which avoids the loss of high-resolution information, and the predicted key point heat map is more accurate. Figure 3

[0073] For each frame key point heat map, the corresponding forward difference map and backward difference map are obtained, and the key point interframe change map corresponding to the current frame key point heat map is obtained by taking or operation on the forward difference map and the backward difference map.

[0074] The present application can obtain the key point interframe change map corresponding to the current frame key point heat map by taking or operation on the forward difference map and the backward difference map, so as to very quickly and conveniently obtain the interframe change of human body action, which is beneficial to capture abnormal action and speeds up the speed of abnormal detection.

[0075] The acquisition formula of the forward difference map is BDI k = |H k-1 -H k |;

[0076] The acquisition formula of the backward difference map is FDI k = |H k -H k+1 |;

[0077] In the formula, n represents the total number of key point heat maps, H k represents the kth frame key point heat map, H k-1 represents the (k-1)th frame key point heat map, and H k+1 represents the (k+1)th frame key point heat map, BDI k represents the forward difference map corresponding to the kth frame key point heat map, and FDI k ​BDI k represents a backward difference map corresponding to the kth frame key point heat map; since the first frame key point heat map has no forward difference map, and the last frame has no backward difference map, therefore, the value range of k is (1, n-2);

[0078] The key point inter-frame change map is obtained according to the following formula: CDI k k = BDI k k ∪ FDI k k , wherein CDI k k represents a key point inter-frame change map corresponding to the kth frame key point heat map.

[0079] An image label pair (CDI k k , Y k k ) corresponding to each key point inter-frame change map is obtained, Y k k ∈ (+1, -1), a convolutional neural network (in this embodiment, the convolutional neural network selected is ResNet10) is trained through the image label pair, and a convolutional neural network model is obtained; the image label pair includes a key point inter-frame change map and a label corresponding thereto; wherein Y k k represents a label corresponding to the key point inter-frame change map.

[0080] Real-time acquisition of the escalator monitoring video is performed, and a picture set corresponding to the detected (i.e., real-time acquired) escalator monitoring video is input frame by frame into a YOLOv5 target detection model in real time, to obtain a pedestrian target frame position corresponding to each frame picture, and a key point heat map corresponding to each pedestrian target frame position is predicted through an HR-Net key point extraction model, a key point inter-frame change map corresponding to each frame key point heat map is obtained, and the key point inter-frame change map is input into a convolutional neural network model to predict a label corresponding thereto, so that real-time detection of escalator monitoring video anomalies is realized; it should be noted that when the label is an abnormal label, the escalator can be controlled to slow down or stop or voice broadcast to remind passengers to pay attention to safety.

[0081] The present application only needs a small convolutional neural network model to realize detection after starting detection and obtaining a key point heat map, which greatly improves the detection efficiency.

[0082] The application obtains the key point heat map corresponding to each frame of picture in the data set by combining the trained YOLOv5 target detection model and the HR-Net key point extraction model, and obtains the key point inter-frame change graph corresponding to the key point heat map, trains the convolutional neural network by using the image label containing the key point inter-frame change graph and the label corresponding to the key point inter-frame change graph to obtain the convolutional neural network model, and when starting detection, the escalator monitoring video picture set to be detected is input into the YOLOv5 target detection model frame by frame to obtain the pedestrian target frame position corresponding to each frame of picture, and the HR-Net key point extraction model is used to predict the key point heat map corresponding to each pedestrian target frame position, the key point inter-frame change graph corresponding to the key point heat map is input into the convolutional neural network model to predict the label corresponding to the key point inter-frame change graph, and the behavior state (abnormal or normal) on the escalator is obtained through the label, which realizes the intelligent detection of the escalator monitoring video anomaly, and greatly improves the accuracy of the abnormal detection.

[0083] Embodiment two

[0084] As Figure 2 shown, the application further provides an escalator monitoring video anomaly detection system, comprising:

[0085] A data set acquisition module is configured to acquire the monitoring video of the escalator, convert it into a picture set continuous in time points, and judge whether the current frame is a normal frame frame by frame, if yes, mark it as a positive sample, if not, mark it as a negative sample, and obtain a model data set containing positive samples and negative samples; the positive sample is specifically a picture marked with a normal label, and the negative sample is specifically a picture marked with an abnormal label;

[0086] A first training module is configured to train the YOLOv5 model and the HR-Net model respectively by using a large-scale data set to obtain the corresponding YOLOv5 target detection model and HR-Net key point extraction model;

[0087] The large-scale data set is the MS COCO data set; the YOLOv5 model is trained by using the large-scale data set, specifically including: training the YOLOv5 model by using the MS COCO data set, and training the regression branch of the YOLOv5 model by using the loss function The target and category branches of the YOLOv5 model are trained by using the BCE loss function.

[0088] When the YOLOv5 model is trained by using the large-scale data set, it further includes:

[0089] For the picture data in the large-scale data set, the picture input into the YOLOv5 model or the HR-Net model is randomly flipped, and the Mosaic data enhancement is used, specifically including: splicing any four pictures to obtain a new picture after splicing and adding the new picture to the training to expand the data set.

[0090] The key point heat map acquisition module is configured to input pictures in a model data set into a YOLOv5 target detection model frame by frame to obtain positions of pedestrian target frames corresponding to each picture frame; and the HR-Net key point extraction model is used to predict key point heat maps corresponding to each pedestrian target frame position.

[0091] The HR-Net key point extraction model is used to predict the key point heat maps corresponding to each pedestrian target frame position, specifically, the HR-Net key point extraction model is used to input each pedestrian target frame position in sequence to obtain a high-resolution feature map including human key points and a human key point rectangular bounding box confidence through the HR-Net key point extraction model, and human posture estimation is performed on the human key point rectangular bounding box with a confidence higher than a set threshold in the high-resolution feature map to obtain pixel coordinates and a predicted confidence of the human key points, and then the key point heat map corresponding to the pedestrian target frame position is obtained.

[0092] The inter-frame change map acquisition module is configured to obtain a forward difference map and a backward difference map corresponding to each key point heat map, and obtain a key point inter-frame change map corresponding to the current key point heat map by performing an OR operation on the forward difference map and the backward difference map.

[0093] The second training module is configured to obtain an image label pair corresponding to each key point inter-frame change map, train a convolutional neural network through the image label pair, and obtain a convolutional neural network model; the image label pair includes the key point inter-frame change map and a label corresponding thereto.

[0094] The detection module is configured to input a set of escalator monitoring video pictures to be detected into the YOLOv5 target detection model frame by frame to obtain positions of pedestrian target frames corresponding to each picture frame, predict key point heat maps corresponding to each pedestrian target frame position through the HR-Net key point extraction model, obtain key point inter-frame change maps corresponding to each key point heat map, and input the key point inter-frame change maps into the convolutional neural network model to predict labels corresponding thereto.

[0095] Specifically, the YOLOv5 target detection model is used to locate pedestrians in a picture, the HR-Net key point extraction model is used to extract human key points, the key point heat map is obtained in combination, and the key point inter-frame change map is obtained by using the inter-frame relationship to capture abnormal behaviors and reflect the inter-frame change of the human posture, so that the accuracy of the abnormal detection is greatly improved.

[0096] Embodiment three

[0097] The application further provides an escalator monitoring video anomaly detection device, comprising a memory and a processor; the memory is used for storing a computer program; the processor is used for realizing the escalator monitoring video anomaly detection method as described above when executing the computer program.

[0098] It should be noted that all directional indications, such as upper, lower, left, right, front, back, etc., in the embodiments of the present application are only used to explain the relative positional relationship, movement condition, etc. between components in a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indications will also change accordingly.

[0099] In addition, the descriptions such as "first", "second", "one" and the like in the present application are only for descriptive purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0100] In the present application, unless otherwise specifically defined and limited, the terms "connection", "fixation" and the like should be understood broadly, for example, "fixation" can be fixed connection, or detachable connection, or integral; can be mechanical connection, or electrical connection; can be directly connected, or indirectly connected through intermediate medium; can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise specifically limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0101] In addition, the technical solutions of each embodiment of the present application can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection required by the present application.

Claims

1. An escalator monitoring video anomaly detection method, characterized by, The method comprises the following steps: Obtaining monitoring video of an escalator, converting the monitoring video into a picture set in time point sequence, and judging whether a current frame is a normal frame frame by frame, if yes, marking as a positive sample, if no, marking as a negative sample, and obtaining a model data set containing positive samples and negative samples; the positive sample is specifically a picture marked with a normal label, and the negative sample is specifically a picture marked with an abnormal label; Training a YOLOv5 model and an HR-Net model respectively through a large-scale data set to obtain a corresponding YOLOv5 target detection model and an HR-Net key point extraction model; Inputting pictures in the model data set frame by frame into the YOLOv5 target detection model to obtain a pedestrian target frame position corresponding to each picture; predicting a key point heat map corresponding to each pedestrian target frame position through the HR-Net key point extraction model; For each key point heat map, obtaining a forward difference map and a backward difference map corresponding thereto, and obtaining a key point inter-frame change map corresponding to the current key point heat map by performing an OR operation on the forward difference map and the backward difference map; The acquisition formula of the forward difference graph is: BDI k = |H k-1 - H k |; The acquisition formula of the backward difference graph is: FDI + = |H k - H k+1 |; wherein k is in the range of (1, n-2); in the formula, n represents the total number of frames of the key point heat map, H k represents the kth frame key point heat map, H k-1 represents the k-1th frame key point heat map, H k+1 represents the k+1th frame key point heat map, BDI k represents the forward difference map corresponding to the kth frame key point heat map, FDI k represents the backward difference map corresponding to the kth frame key point heat map; The key point inter-frame change diagram is obtained by the following formula: CDI k = BDI k ∪ FDI k , wherein CDI k represents the key point inter-frame change diagram corresponding to the kth key point heat map. Obtaining an image label pair corresponding to each key point inter-frame change map, training a convolutional neural network through the image label pair, and obtaining a convolutional neural network model; the image label pair includes a key point inter-frame change map and a label corresponding thereto; Inputting a picture set of the escalator monitoring video to be detected frame by frame into the YOLOv5 target detection model to obtain a pedestrian target frame position corresponding to each picture, predicting a key point heat map corresponding to each pedestrian target frame position through the HR-Net key point extraction model, obtaining a key point inter-frame change map corresponding to each key point heat map, and inputting the key point inter-frame change map into the convolutional neural network model to predict a label corresponding thereto.

2. The escalator monitoring video anomaly detection method of claim 1, wherein, The HR-Net key point extraction model is specifically: inputting each pedestrian target frame position into the HR-Net key point extraction model in sequence to obtain a high-resolution feature map including human key points and a human key point rectangular bounding box confidence through the HR-Net key point extraction model, performing human pose estimation on the human key point rectangular bounding box with a confidence higher than a set threshold in the high-resolution feature map, obtaining pixel coordinates and a predicted confidence of the human key points, and further obtaining a key point heat map corresponding to the pedestrian target frame position.

3. The escalator monitoring video anomaly detection method of claim 1, wherein, The large-scale data set is an MS COCO data set; The YOLOv5 model is trained through the large-scale data set, and specifically comprises the following steps: The YOLOv5 model is trained through the MS COCO dataset, and the loss function is used to train the model during the training process The regression branch of the YOLOv5 model is trained, and the target and class branch of the YOLOv5 model is trained through the BCE loss function.

4. The escalator monitoring video anomaly detection method according to claim 3, characterized in that, The loss function The formula expression is: In the formula: wherein Intersection(A, B) represents an intersection area of a prediction box A of the YOLOv5 model and a target box B, and Union(A, B) represents a union area of the prediction box A of the YOLOv5 model and the target box B; b, b gt respectively represent center points of the prediction box and the real box, and ρ 2 (b, b gt ) is an Euclidean distance between the center points of the prediction box and the real box, c is a diagonal distance of a minimum closed region capable of containing the prediction box and the real box, and a is a loss weight; represents a loss value; w gt is a width of the real box, and h gt is a height of the real box, w is a width of the prediction box, and h is a height of the prediction box. The formula expression of the BCE loss function is: In the formula, p is a probability that the YOLOv5 model predicts a sample to be a positive sample; is a label of the sample, and when the sample belongs to a positive sample, the value is 1, otherwise the value is 0, and BCELoss is a loss value.

5. The escalator monitoring video anomaly detection method of claim 4, wherein, When training the YOLOv5 model through the large-scale data set, the following steps are further included: For picture data in the large-scale data set, randomly flip the picture input into the YOLOv5 model or the HR-Net model, and use Mosaic data enhancement, specifically: splice any four pictures to obtain a new picture after splicing and add it to the training to expand the data set.

6. An escalator monitoring video anomaly detection system characterized by, The method comprises the following steps: The data set acquisition module is configured to acquire a monitoring video of the escalator, convert the monitoring video into a picture set in time points, and judge whether each frame is a normal frame, if so, mark the frame as a positive sample, and if not, mark the frame as a negative sample, to obtain a model data set containing the positive sample and the negative sample; the positive sample is specifically a picture marked with a normal label, and the negative sample is specifically a picture marked with an abnormal label; The first training module is configured to train the YOLOv5 model and the HR-Net model respectively by using the large-scale data set to obtain a YOLOv5 target detection model and an HR-Net key point extraction model; The key point heat map acquisition module is configured to input the pictures in the model data set into the YOLOv5 target detection model frame by frame to obtain the positions of the pedestrian target frames corresponding to each picture, and predict the key point heat maps corresponding to the positions of each pedestrian target frame by using the HR-Net key point extraction model; The inter-frame change map acquisition module is configured to obtain the forward difference map and the backward difference map corresponding to each key point heat map, and obtain the key point inter-frame change map corresponding to the current key point heat map by performing an OR operation on the forward difference map and the backward difference map; The acquisition formula of the forward difference graph is: BDI k = |H k-1 - H k |; The acquisition formula of the backward difference graph is: FDI k = |H k - H k+1 |; wherein k is in the range of (1, n-2); in the formula, n represents the total number of frames of the key point heat map, H k represents the kth frame key point heat map, H k-1 represents the (k-1)th frame key point heat map, H k+1 represents the (k+1)th frame key point heat map, BDI k represents the forward difference map corresponding to the kth frame key point heat map, FDI k represents the backward difference map corresponding to the kth frame key point heat map; The key point inter-frame change map is obtained according to the following formula: CDI k = BDI k ∪ FDI k , wherein CDI k represents the key point inter-frame change map corresponding to the kth key point heat map. The second training module is configured to obtain an image label pair corresponding to each key point inter-frame change map, train a convolutional neural network by using the image label pair, and obtain a convolutional neural network model; the image label pair includes the key point inter-frame change map and a label corresponding to the key point inter-frame change map; The detection module is configured to input the picture set of the escalator monitoring video to be detected into the YOLOv5 target detection model frame by frame to obtain the positions of the pedestrian target frames corresponding to each picture, predict the key point heat maps corresponding to the positions of each pedestrian target frame by using the HR-Net key point extraction model, obtain the key point inter-frame change maps corresponding to each key point heat map, and input the key point inter-frame change maps into the convolutional neural network model to predict the labels corresponding to the key point inter-frame change maps.

7. The escalator monitoring video anomaly detection system of claim 6, wherein, The HR-Net key point extraction model is configured to input the positions of the pedestrian target frames into the HR-Net key point extraction model in sequence to obtain a high-resolution feature map including human key points and a human key point rectangular bounding box confidence, perform human posture estimation on the human key point rectangular bounding box with a confidence higher than a set threshold in the high-resolution feature map, obtain pixel coordinates and a prediction confidence of the human key points, and further obtain the key point heat map corresponding to the position of the pedestrian target frame.

8. The escalator monitoring video anomaly detection system of claim 7, wherein, The large-scale data set is an MS COCO data set; The YOLOv5 model is trained through a large-scale data set, specifically comprising: training the YOLOv5 model through an MS COCO data set, and in the training process, through a loss function The regression branch of the trained YOLOv5 model is trained through a BCE loss function, and the target and category branch of the trained YOLOv5 model is trained through a CIoU loss function.

9. The escalator monitoring video anomaly detection system of claim 8, wherein, When training the YOLOv5 model by using the large-scale data set, the method further includes the following steps: For the picture data in the large-scale data set, the method further includes the following steps: randomly flipping the picture input into the YOLOv5 model or the HR-Net model, and using Mosaic data enhancement, specifically, splicing any four pictures to obtain a new picture after splicing and adding the new picture to the training to expand the data set.

Images