Detection method for intelligently monitoring abnormal state of pedestrian

Abstract

The invention provides an intelligent monitoring pedestrian abnormal state detection method for pedestrian abnormal behaviors in various monitoring scenes, and multi-target recognition of pedestrians is completed through target detection and target tracking technologies. The abnormal behavior is found, an alarm is given to prompt workers, and an abnormal picture is stored. For abnormal behaviors, a rapid identification and early warning method based on abnormal pictures of each scene is established, and workers are accurately and efficiently helped to complete safety prevention and control tasks.

Description

Technical Field

[0001] The present application relates to a method for detecting abnormal status of intelligently monitored pedestrians, and specifically to using target detection and target tracking technology to actually accurately locate pedestrians in a monitored area and continuously track them by calculating and judging the behavior and status of pedestrians. Background Art

[0002] Quickly discovering dangerous behaviors and safety hazards in the monitoring area is one of the key factors to ensure pedestrian safety. However, the monitoring equipment for each road and scene is complex, the number of monitoring videos is huge, and manual observation is inefficient, intensive, and easy to fatigue. For example, if there are ten cameras monitoring different corners of an area, the monitor needs to constantly observe between the ten monitoring screens, which will not only affect the worker's vision health, but also easily lead to missed detection, which seriously affects the ability to detect and eliminate dangers in a timely manner.

[0003] To solve this problem, modern monitoring systems use intelligent monitoring methods to detect abnormal pedestrian status. These technologies use target detection and target tracking algorithms to automatically identify pedestrians in the monitoring screen and achieve accurate positioning and continuous tracking. By calculating and judging the behavior and status of pedestrians, dangerous behavior and safety hazards can be quickly discovered.

[0004] Specifically, the intelligent monitoring system will analyze the monitoring screen in real time, detect pedestrians through the target detection algorithm, and track their movement trajectory using the target tracking algorithm. The system can also determine whether the pedestrian is in an abnormal state by calculating the pedestrian's speed, acceleration, posture, and other information. Once dangerous behavior or safety hazards are detected, the system will immediately issue an alarm to remind the monitor to take appropriate measures.

[0005] By introducing an intelligent monitoring method for detecting abnormal pedestrian conditions, monitoring efficiency and accuracy can be greatly improved. Monitors no longer need to manually observe multiple monitoring screens, but can rely on the intelligent system to automatically detect and track pedestrians, thereby discovering dangerous situations in a timely manner. This not only reduces the workload of monitors, but also avoids missed detections caused by human negligence, and improves the level of pedestrian safety.

[0006] In short, the intelligent monitoring pedestrian abnormal state detection method plays an important role in quickly discovering dangerous behaviors and safety hazards in the monitoring area. By automatically identifying and tracking pedestrians, and calculating and judging their behaviors and states, the monitoring efficiency and accuracy can be improved, thereby ensuring the safety of pedestrians. Summary of the invention

[0007] The purpose of this application is to propose an intelligent monitoring method for detecting abnormal conditions of pedestrians, which can accurately locate and continuously track pedestrians in the monitoring area by calculating and judging the behavior and status of pedestrians.

[0008] In order to achieve the above-mentioned purpose, this application combines target detection and target tracking technology to propose an intelligent monitoring pedestrian abnormal state detection method. Figure 1 As shown in the figure, the surveillance video is input, the Yolov5 neural network model is used for target detection, the coordinate position of the pedestrian in the surveillance picture is identified, and the identified coordinate position of the pedestrian is sent to the Byte Trackg target tracking algorithm for continuous tracking. The pedestrian's running trajectory is used to calculate and judge whether the pedestrian is in an abnormal state.

[0009] This application method mainly consists of 4 modules: Figure 2 As shown, it is divided into a model training module, a target detection module, a target tracking module, and an abnormal information storage module. The four modules in the method of this application are described below respectively.

[0010] (1) Model training module

[0011] The model training module mainly uses a large number of pedestrian feature images and annotation information to train a model with high accuracy and strong generalization ability. Two issues need to be considered before training. First, for surveillance videos, before risk analysis, pedestrians in the surveillance video need to be identified before they are handed over to the target tracking algorithm for pedestrian tracking. If the model is not accurate enough, false positives and missed positives will often be sent, which will further affect the accuracy of the target tracking algorithm. Because we need to provide accurate annotation information, cover a variety of scenes and situations, contain rich sample changes, and a sufficient number of samples to ensure that the selected and prepared training data have the above characteristics. Second, since most of the use is in real-time monitoring and detection scenarios, we need to consider the size of the model, detection speed, device performance, and the size of the real-time FPS value. In order to create an ideal processing environment, we need to choose a target detection algorithm with extremely high robustness, which can accurately detect various dangerous behaviors and has a fast detection speed. In this context, we are pursuing a perfect balance. We need an efficient target detection algorithm that can quickly and accurately detect dangerous behaviors in real-time monitoring. At the same time, we need to consider the performance of the device to ensure that the algorithm can run on resource-constrained devices and maintain a stable real-time FPS value. By choosing a suitable target detection algorithm, we can achieve this balance. We need an algorithm with high robustness and the ability to adapt to various scenarios and conditions to ensure accurate detection of dangerous behaviors. At the same time, we also need an algorithm with high efficiency, which can quickly process large amounts of data in real-time monitoring and maintain a high detection speed. Therefore, in this application method, we use Yolov5 as the target detection algorithm network structure diagram as shown below Figure 3Compared with other target detection algorithms, Yolov5 has higher detection accuracy and faster detection speed. It adopts a new detection architecture, uses a deeper network, and also adopts a series of optimization strategies, including data enhancement, network structure optimization, learning rate adjustment, etc., to improve the performance of the model.

[0012] The Yolov5 model training module mainly includes the following steps:

[0013] 1. Data preparation: First, you need to prepare a training dataset. The dataset should contain annotated images and the corresponding target bounding box information. You can use annotation tools such as Labelimg or Labelbox to manually annotate the target bounding box. Then, save the image and annotation information in a specific format, such as Yolo format (.txt file) or COCO format (.Json file.

[0014] 2. Model configuration: Next, you need to configure the parameters and hyperparameters of the model. You can choose different model architectures according to your actual needs, such as Yolov5s, Yolov5m, Yolov5l, or Yolov5x. You can also adjust hyperparameters such as the size of the input image, learning rate, batch size, etc. In addition, you can choose whether to use pre-trained weights to initialize the model.

[0015] 3. Model training: After data preparation and model configuration are completed, model training can begin. Use the training data set to train the model and use the back-propagation algorithm to update the model weights. You can choose different optimizers, such as SGD, Adam, etc., to optimize the model's loss function. During the training process, you can monitor the model's training loss and validation loss to evaluate the model's performance.

[0016] 4. Model evaluation: After the model training is completed, the model can be evaluated using the test data set. The performance of the model can be evaluated by calculating the model's precision, recall rate, F1 value and other indicators on the test data set. The model can be adjusted and optimized based on the evaluation results to further improve the accuracy and robustness of the model.

[0017] (2) Model prediction module

[0018] The model prediction module is the process of applying the trained model to new video data to achieve target detection, which mainly includes the following steps:

[0019] 1. Model loading: First, you need to load the trained model into memory. You can use the functions or interfaces provided by the framework to load the model's weights and configuration files. After loading the model, you can get the model's input and output information by calling the corresponding functions.

[0020] 2. Input data preprocessing: Before model inference, the input data needs to be preprocessed. The preprocessing steps include image scaling, normalization, channel order adjustment, etc. to make it meet the input requirements of the model. You can use image processing libraries such as OpenCV or PLT to implement preprocessing operations.

[0021] 3. Model reasoning: After preprocessing the input data, the preprocessed data can be input into the model for reasoning. The process of model reasoning is to forward propagate the input data and predict the bounding box and category of the target through the network structure and weights of the model. You can use the functions or interfaces provided by the framework to call the model's reasoning method.

[0022] 4. Output results: After the model inference is completed, the target detection results will be obtained. The results usually include the coordinates of the detected target bounding box, category label, and confidence level. The results can be post-processed as needed, such as non-maximum suppression (NMS), to filter overlapping bounding boxes and improve detection accuracy. Finally, the output results are sent to the target tracking module.

[0023] (3) Target Tracking Module

[0024] The main function of the target tracking module is to continuously track one or more targets and obtain the desired information from the tracking process.

[0025] Before using the algorithm we need to solve the following problems:

[0026] 1. Morphological changes: Posture changes are a common interference problem in target tracking. When a moving target changes its posture, its features and appearance model will change, which can easily lead to tracking failure. For example: athletes in sports games, pedestrians on the road.

[0027] 2. Scale change: Scale adaptation is also a key issue in target tracking. When the target scale is reduced, the tracking frame cannot track adaptively, and a lot of background information will be included, resulting in incorrect update of the target model. When the target scale is increased, the tracking frame cannot completely include the target, and the target information in the tracking frame is incomplete, which will also lead to incorrect update of the target model. Therefore, it is very necessary to implement scale adaptive tracking.

[0028] 3. Occlusion and disappearance: The target may be occluded or disappear temporarily during the movement. When this happens, the tracking frame will easily include the occluder and background information in the tracking frame, causing the tracking target in subsequent frames to drift onto the occluder. If the target is completely occluded, the corresponding model of the target cannot be found, resulting in tracking failure.

[0029] 4. Image Blur: Situations such as changes in light intensity, rapid movement of the target, and low resolution can lead to image models, especially when the moving target is similar to the background. Therefore, it is very necessary to select effective features to distinguish the target from the background.

[0030] To solve these problems, we need to select an object tracking algorithm that can handle these four problems. The Byte Track object tracking algorithm perfectly meets our requirements. Byte Track has the following advantages:

[0031] 1. High Efficiency: Byte Track uses a pixel-level tracking method. Compared with traditional feature-based tracking algorithms, it can process images faster and has high real-time performance.

[0032] 2. Robustness: The Byte Track algorithm has strong robustness and can accurately track the target even in complex environments such as target occlusion and light changes.

[0033] 3. Precision: Byte Track can accurately track the target at the pixel level, capture the subtle movements and changes of the target, and thus performs well in some applications that require high-precision tracking.

[0034] The operation process of the Byte Track object tracking algorithm is as Figure 4 , we need to save the third matching result of the Byte Track object tracking algorithm. When the saved result meets the condition of 30 consecutive frames of tracking, we need to compare and calculate the result of the last frame with the saved results of the previous 30 frames. We judge whether the target is in an abnormal state based on the result after comparison and calculation.

[0035] (4) Abnormal Information Storage Module

[0036] The main function of the abnormal information storage module is to save the data sent by the object tracking module. However, the data sent by the object tracking module is not saved immediately. We need to save the data to a list of key concerns first. In this list, we will record the identification number, the abnormal state, the time of occurrence of the abnormality, the duration of the abnormal state, and the storage location of the abnormal event video of the pedestrians in the abnormal state. While continuously updating the list of key concerns for data, we also retain the past data. If the duration of the abnormal state of a pedestrian in the list of key concerns exceeds a certain threshold, we need to give a warning prompt to the pedestrians in the abnormal state in the monitoring screen, which is manifested as a red flashing box drawn around the human contour. If the abnormal state time of the pedestrian does not meet the threshold, we can choose to ignore the abnormal state and not perform any processing. This is applicable to situations where the frequency of abnormal states is low or the impact on the system is small.

[0037] When it is determined that the abnormal behavior of the pedestrian in an abnormal state has ended or has walked out of the monitoring screen, we need to save the abnormal data of the pedestrian. The saved data includes: identification number, abnormal state, abnormal event address, abnormal event time, and abnormal event end time. Before saving this data, we need a: easy to use, open source and free, cross-platform support, high performance, scalability, and support for multiple storage engines. Therefore, we choose MySQL database and create a database format such as Figure 5 In addition, the time period when the abnormal event occurred is saved as a video in MP4 format. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of the operation of the abnormal state detection method for intelligent monitoring pedestrians

[0039] Figure 2 Module structure diagram of the abnormal state detection method for intelligent monitoring pedestrians

[0040] Figure 3 Yolov5 target detection algorithm structure flow chart

[0041] Figure 4 Byte Track target tracking algorithm structure flow chart

[0042] Figure 5 Create a format for Mysql database Specific implementation methods

[0044] In order to better describe the intelligent monitoring pedestrian abnormal state detection method, the specific implementation method of this application is given below.

[0045] This project trained an abnormal state detection and recognition model based on YOLOv5 target detection in Windows system. After opening the project, select the detect.py file to select the input video path or call the camera for real-time detection. An alarm is issued for the detection of pedestrians in abnormal state, and the event segment of the abnormal event is saved in the format of video. In order to realize the intelligent monitoring of pedestrian abnormal state detection, this project sets up four modules: model training module, target detection module, target tracking module, and abnormal information storage module.

[0046] (1) Model training module

[0047] Collect and annotate the training dataset. Make sure the dataset contains the objects that need to be detected, and use the annotation tool to annotate the bounding box and category label for each object. The dataset should contain a training set, a validation set, and a test set. Convert the annotated data to the format required by the Yolov5 model. The data format used by Yolov5 can be Yolo format (.txt file) or COCO format (.json file). You can use scripts or tools to convert the annotated data to the required format and configure the parameters of the Yolov5 model according to actual needs. You can adjust the network structure, input size, training hyperparameters, etc. of the model. The Yolov5 model is usually initialized with pre-trained weights. Pre-trained weights for different model versions can be downloaded from the Yolov5 official GitHub repository. Use the prepared dataset and configured model parameters for training. You can use the command line or script to run the training command, specifying the training set, validation set, model parameters, optimizer, etc. During the training process, you can periodically evaluate the performance of the model on the validation set. You can use predefined evaluation metrics (such as mAP) to measure the accuracy and recall of the model. Based on the evaluation results, you can adjust the model's hyperparameters, data augmentation strategy, learning rate, etc. to improve model performance. After training is completed, the model can be tested using the test set to evaluate its performance in actual scenarios.

[0048] (2) Object Detection Module

[0049] After completing the model training, configure the detect.py file parameters: use the best weights during the training process, that is, the best.pt file, debug the confidence threshold and IOU threshold, set the input path to the camera or video path, and finally run the file to complete the model prediction. After connecting the camera, this project uses the camera's real-time monitoring screen as input, detects pedestrian targets in real time, and sends the detected pedestrian data to the target tracking module.

[0050] (3) Target Tracking Module

[0051] The data sent by the target detection module is received, and the information in the data is:

[0052] 1. Coordinates of the detection box: usually expressed in the form of (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the detection box, w is the width of the detection box, and h is the height of the detection box.

[0053] 2. Confidence of the detection box: Indicates the model’s confidence in the category to which the detection box belongs.

[0054] 3. Detection box category: indicates the object category predicted by the model.

[0055] The pedestrian data are recorded and tracked, and it is determined whether the tracked pedestrian data is in an abnormal state, and the pedestrian data in an abnormal state is sent to the abnormal information storage module.

[0056] (4) Abnormal information storage module

[0057] Receive the target tracking module data and save it in the focus list. When the duration of abnormal state in the focus list exceeds a certain threshold, it will automatically give a warning and store it in the created MySQL.

Claims

1. Intelligent monitoring method for detecting abnormal status of pedestrians, Features: The method of identifying, tracking and analyzing abnormal behavior of pedestrians in abnormal state in the monitored images can accurately and efficiently complete the risk analysis task for pedestrians in abnormal state. The specific steps are as follows: Step 1) Use the deep learning target algorithm based on Yolov5 to detect pedestrians and other targets and locate their coordinates; Step 2) Use the Byte Trackg target tracking algorithm to continuously track the detected pedestrian; Step 3) extracting and calculating information such as walking speed, walking direction, and walking trajectory of the continuously tracked pedestrian; Step 4) The extracted information is judged as abnormal, and pedestrians with abnormal conditions are placed in a key attention list; Step 5) Pedestrians who are continuously in an abnormal state are put into an abnormal event list, and information such as the time of occurrence of the abnormal event is recorded. Step 6) Save the time period when the abnormal event occurs, and store the data in the abnormal event list in the MySQL database.

2. According to the intelligent monitoring pedestrian abnormal state detection method described in claim 1, it is characterized in that in step 1), a large number of pedestrian images are first labeled, and the model is trained using the deep learning target algorithm of Yolov5, and then the deep learning model of Yolov5 is used to extract features of the monitored images and locate the coordinates of the pedestrians.

3. According to the intelligent monitoring pedestrian abnormal state detection method of claim 1, it is characterized in that in step 2), the positioned coordinate position is sent to the Byte Trackg target tracking algorithm, Byte Trackg uses Kalman filtering to predict the position of the tracking trajectory of the current frame in the next frame, and the Iou between the predicted coordinate position and the actual coordinate position is used as the similarity between the two matches, and the matching is completed by the Hungarian algorithm to determine the tracking object.

4. The intelligent monitoring pedestrian abnormal state detection method according to claim 1 is characterized in that in step 3), for the pedestrian to be tracked, the coordinate position of the current frame and the coordinate position of the previous 30 frames are used as the position to calculate the walking speed, walking direction, walking trajectory and other information.

5. According to the intelligent monitoring pedestrian abnormal state detection method of claim 1, it is characterized in that step 4) uses the collected walking speed, walking direction, and walking trajectory information to judge the abnormal behavior of pedestrians, including sudden acceleration, sudden pause, frequent change of direction, etc. We will determine it as an abnormal state to be confirmed, and put the pedestrians in the abnormal state to be confirmed into the key attention list.

6. According to the intelligent monitoring pedestrian abnormal state detection method of claim 1, it is characterized in that in step 5), if the pedestrian is in an abnormal state to be confirmed for a long time, we regard it as an abnormal event, and record the occurrence time, end time, cause of the event, risk level and other information of the abnormal event and put it into the abnormal event list.

7. The method for detecting abnormal status of intelligently monitored pedestrians according to claim 1 is characterized in that step 6) uses the occurrence time and end time in the abnormal event list to intercept the frequency band where the abnormal event occurs, and stores all the data in the abnormal event list into a Mysql database for viewing and use.

Images