AI visual passenger flow volume statistical system and method based on YOLOv7

Abstract

The invention discloses an AI visual passenger flow volume statistics and analysis system based on YOLOv7, and belongs to the field of computer vision and data statistical analysis. Through multi-model collaborative optimization, the system realizes the functions of passenger flow statistics, personnel attribute analysis, abnormal behavior detection and multi-dimensional data mining. The method specifically comprises the steps of real-time passenger flow counting, peak period distribution and hot spot area analysis based on YOLOv7 and SORT algorithms; utilizing a PaddleClas module to extract the gender, age and clothes attributes of the pedestrian so as to construct a user portrait; and early warning is carried out on abnormal behaviors such as running, gathering and object throwing in combination with an ASTNet network. The system adopts a mixed data set training model, hardware resource consumption is reduced through a multi-task parallel processing architecture, and the real-time performance is improved while the detection precision is guaranteed. The multi-dimensional data analysis capability can provide accurate support for commercial operation optimization and public safety early warning, and the method is suitable for intelligent management of retail, scenic spots, transportation hubs and other scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of detection and data statistical analysis, and relates to an AI visual passenger flow statistics system and method based on YOLOv7. Background Art

[0002] With the intelligent upgrade of retail, transportation hubs, smart parks, and public security scenarios, passenger flow statistics and in-depth behavior analysis have become core requirements for optimizing operational efficiency and ensuring public safety. However, current passenger flow statistics and behavior analysis solutions still have the following shortcomings:

[0003] 1. Traditional solutions mostly focus on basic counting functions, relying on manual statistics, infrared sensors, or early vision algorithms (such as background subtraction and optical flow). They have problems such as limited functionality, poor environmental adaptability, single data dimension, and inability to perform intelligent analysis.

[0004] 2. Traditional algorithms are not robust enough to target occlusion and dense scenes, and lack the ability to deeply explore pedestrian attributes and behavioral characteristics, making it difficult to meet advanced needs such as accurate user profiling and abnormal event warnings.

[0005] In recent years, multi-task vision systems based on deep learning have gradually become a research hotspot.

[0006] YOLOv7 is a single-stage real-time object detector. It was introduced to the YOLO family in July 2022. Object detection models such as YOLOv7, with their efficient feature fusion architecture and real-time inference performance, provide fundamental support for passenger flow counting in complex scenarios. However, a single detection model cannot meet the following high-level requirements:

[0007] 1. Passenger flow data statistics: In addition to counting the total passenger flow of the total number of people entering and leaving a certain area within a specific time period, it is also necessary to count the time periods of peak and trough passenger flow and analyze the passenger flow trend in different time periods.

[0008] 2. Analysis of tourists’ regular behavior: statistics on the average length of stay of tourists in a certain area; statistics on the areas where tourists most often stay and move; analysis of the group size of tourists, whether they appear in the form of individuals or groups.

[0009] 3. Multi-dimensional feature perception: Detecting pedestrian attributes such as gender, age, and clothing supports refined data analysis, enabling user profiling and precision marketing in commercial scenarios.

[0010] 4. Ensure public safety: To ensure public safety, the system needs to have abnormal behavior detection functions to detect abnormal behaviors such as running, throwing objects, and gathering; and monitor crowd density to prevent stampede accidents.

[0011] 4. Multi-task and multi-model collaborative optimization: The YOLOv7 model, target tracking, attribute detection, and behavior recognition modules must collaborate efficiently to avoid redundant computing power and information conflicts.

[0012] Intelligent passenger flow counting systems are crucial for improving the operational efficiency and ensuring public safety in public places like scenic spots. Multi-dimensional data statistics and intelligent analysis enable managers to formulate more effective development strategies based on statistical analysis results. They also provide early warnings to mitigate public safety risks, ensuring safe, stable, and efficient operation of the entire area. Summary of the Invention

[0013] This paper proposes an AI visual passenger flow statistics system and method based on YOLOv7. By integrating the sort library, the pedestrian attribute detection module of the PaddlePaddle deep learning framework, and the abnormal behavior recognition module driven by the ASTNet spatiotemporal network, it realizes the full-link analysis of "detection-attribute-behavior".

[0014] The technical solution adopted in the present invention is as follows:

[0015] An AI visual passenger flow counting system based on YOLOv7, including:

[0016] Pedestrian attribute recognition module, trained using pa100K;

[0017] ASTNet spatiotemporal network-driven abnormal behavior recognition module, trained using ShanghaiTech;

[0018] YOLOv7 model trained on BDD100K and selfie datasets.

[0019] An AI passenger flow counting method based on YOLOv7 includes the following steps:

[0020] The first step is dataset preparation. We use the public datasets BDD100K (a large-scale, diverse driving video dataset released by the Berkeley AI Lab), pa100K (currently the largest dataset for pedestrian attribute recognition, available for download from the path provided in the paddleCLA project on GitHub), ShanghaiTech (a large-scale crowd counting dataset released by ShanghaiTech University), and self-filmed datasets as training sets. The BDD100K and self-filmed datasets train the YOLOv7 model, pa100K trains the pedestrian attribute recognition module, and ShanghaiTech trains the ASTNet spatiotemporal network-driven abnormal behavior recognition module. The BDD100K and self-filmed datasets undergo preprocessing before entering the network model. The specific steps are as follows:

[0021] Step 11: Prepare the BDD100K dataset, reorganize the label files, remove the images without objects and unclear images, retain the labels of vehicles and pedestrians, and unify the annotation files into txt format.

[0022] Step 12: To increase the proportion of Asian data, we collected online and self-photographed pedestrian images as training data and added them to the BDD100K dataset. The format and training set, validation set, and test set ratios are the same as those of BDD100K.

[0023] The second step is to train the YOLOv7 model and pedestrian attribute recognition module. The specific steps are as follows:

[0024] Step 21: Use the organized dataset to train the YOLOv7 model using Python code. After the training is completed, the YOLOv7 model training weights are obtained and the model results are evaluated using the mean average precision (mAP). (0.5) ) is used as the evaluation indicator. The average precision is calculated as follows:

[0025]

[0026] Among them, IoU represents the intersection-over-union ratio of the predicted box and the actual box, bbox gt Represents the true bounding box, bbox pre Represents the predicted bounding box, mAP (0.5) It means the average precision of the intersection-over-union ratio is greater than or equal to 0.5; i represents the target of the i-th category; TP represents the correctly detected positive sample target, TN represents the correctly detected negative sample target, FP represents the incorrectly detected positive sample target, that is, false positive; Q represents the total number of detection categories involved in the entire task.

[0027] Step 22: Use the dataset pa100K to train the open source model paddleclas. The mean average precision (mAP) is also used as the evaluation metric. Detection boxes are not involved here, so only the correctness of the classification results needs to be considered.

[0028] Step 23: Use the ShanghaiTech dataset to train ASTNet, and use the area under the ROC curve (AUC) in the field of abnormal behavior recognition as the evaluation metric. The ROC curve is a graphical tool used to represent the performance of a classification model. It uses the true positive rate (TP) and the false positive rate (FP) as the horizontal and vertical coordinates to depict the performance of the classifier at different thresholds.

[0029] The third step is to build a system to implement the full-link analysis of "detection-attribute-behavior". The specific steps are as follows:

[0030] Step 31: There are two data input methods for the system: local media file input and real-time camera data input.

[0031] Step 32: The video data to be inspected will be read frame by frame, and each frame will be converted into a tensor and fed into the following two models in parallel: the detection model YOLOv7 and the abnormal behavior recognition module driven by the ASTNet spatiotemporal network.

[0032] Step 33: YOLOv7 will identify people and vehicles appearing in the image and process the detection results through the sort algorithm to track each detection result in the video stream; ASTNet will analyze the behavior in the video stream, mainly responding to fast-moving objects, such as running people or thrown foreign objects.

[0033] Step 34: The detection results of YOLOv7 and sort library will be divided into the following three branches for processing: (1). Enter the passenger flow data statistical analysis module. Through processing, we will obtain the total passenger flow, the time distribution of peak and trough changes in passenger flow, the time each person stays in the screen and the average stay time statistics, hot spot area statistics and other results; (2). Combined with the ASTNet detection results, it will issue warnings and responses to behaviors such as running, falling, staying, and gathering; (3). The detected person frame will be returned to the original image and cut out and sent to the paddleclas module for attribute recognition, and the characteristics of the person such as gender, clothing, and age group will be counted.

[0034] The YOLOv7-based AI visual passenger flow statistics and analysis system of the present invention integrates the YOLOv7 model, the sort tracking algorithm, the paddleclass module, and the abnormal behavior recognition module driven by the ASTNet spatiotemporal network to achieve a full-link analysis of passenger flow from "detection-attribute-behavior" while occupying little video memory, requiring relatively low hardware, achieving high detection speed, and strong real-time performance. Compared to most current passenger flow detection and analysis algorithms, it is more versatile, with multi-dimensional data statistics and intelligent analysis, allowing managers to conduct more comprehensive and objective analysis and formulate more accurate and effective development strategies. It can also provide early warning to avoid public safety risks, ensuring the safety, stability, and efficiency of the entire region. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a system overall framework diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The specific embodiments of the present invention are described in detail below with reference to examples.

[0037] The following embodiment provides an AI passenger flow statistical analysis method based on YOLOv7, which is applied to scenic spots and includes the following implementation steps:

[0038] Part 01: Dataset Preparation. We used the public datasets BDD100K, pa100K, and ShanghaiTech, as well as a self-photographed dataset, as training sets. The BDD100K and self-photographed datasets were used to train the YOLOv7 model, the pa100K dataset was used to train the pedestrian attribute recognition module, and the ShanghaiTech dataset was used to train the ASTNet spatiotemporal network-driven abnormal behavior recognition module. The BDD100K and self-photographed datasets were preprocessed before entering the network model. The specific steps are as follows:

[0039] Step 11: Prepare the BDD100K dataset. After downloading, reorganize the label file, remove the targetless and unclear images, retain the vehicle and pedestrian labels, and unify the annotation file into txt format.

[0040] Step 12: To increase the proportion of Asian data, we collected online and self-photographed pedestrian images as training data and added them to the BDD100K dataset. The format and training set, validation set, and test set ratios are the same as those of BDD100K.

[0041] This example saves the BDD100K and self-photographed dataset images and annotation files in the following directory structure to facilitate YOLOv7 reading:

[0042]

[0043] After downloading the remaining data sets, rewrite the corresponding paths to directly enter the corresponding model training.

[0044] In part 02, configure the operating environment suitable for YOLOv7, paddleclas, and ASTNet, and then use Python code for training;

[0045] This example uses the following code to train the dataset:

[0046] Step 21: Use python train.py --img 640 --batch 32 --epoch 100 --data data / mydata.yaml --cfg cfg / training / yolov7.yaml --weights yolov7.pt --device 0. The training progress will be displayed during the training process. The progress information includes indicators such as loss function value, mAP, recall rate, etc. After completion, the weight file and indicator results will be saved in the folder. Use average precision (mAP (0.5) ) is used as the evaluation indicator. The average precision is calculated as follows:

[0047]

[0048] Among them, IoU represents the intersection-over-union ratio of the predicted box and the actual box, bbox gt Represents the true bounding box, bbox pre Represents the predicted bounding box, mAP (0.5) It means the average precision of the intersection-over-union ratio is greater than or equal to 0.5; i represents the target of the i-th category; TP represents the correctly detected positive sample target, TN represents the correctly detected negative sample target, FP represents the incorrectly detected positive sample target, that is, false positive; Q represents the total number of detection categories involved in the entire task.

[0049] YOLOv7 is the most advanced algorithm in the YOLO series, surpassing many existing object detectors in both speed and accuracy. Subsequent versions 8, 9, and 10 have not seen significant improvements. The YOLOv7 network architecture primarily consists of four components: the input layer, the backbone network, the neck, and the detection head. In this architecture, after image preprocessing at the input layer, the image is fed into the backbone network for feature extraction. The neck then fuses the features before the detection head outputs the final result. Its high speed, efficiency, and accuracy make it well-suited for use in passenger counting systems.

[0050] To verify the advantages of the YOLOv7 network, the indicator results of each method under BDD100K are shown in Table 1:

[0051] Table 1 Results of various detection algorithms

[0052]

[0053] YOLOv7 has the fastest detection speed and higher accuracy.

[0054] Step 22: Using the Code:

[0055]

[0056]

[0057] The open source model paddleclas is trained using the dataset pa100K. The mean average precision (mAP) is also used as the evaluation metric. Detection boxes are not involved here, so only the correctness of the classification results needs to be considered.

[0058] Step 23: Exploit Code:

[0059] python train.py\

[0060] --cfg config / ped2_wresnet.yaml

[0061] The ShanghaiTech dataset was used to train ASTNet, using the area under the ROC curve (AUC) in the field of abnormal behavior recognition as the evaluation metric. The ROC curve is a graphical tool used to represent the performance of a classification model. It uses the true positive rate (TP) and the false positive rate (FP) as the horizontal and vertical coordinates to depict the performance of the classifier at different thresholds.

[0062] Part 03: Build a system to implement full-link analysis of "detection-attribute-behavior" and real-time detection and analysis:

[0063] Model deployment is the process of integrating multiple trained models and applying them to real-world environments. This system can be deployed on multiple platforms, including local computers, cloud servers, and embedded devices.

[0064] Step 31: There are two data input methods for the system: local media file input and real-time camera data input.

[0065] Step 32: The video data to be inspected will be read frame by frame through the cv2.VideoCapture method of the OpenCV library, and each frame of the image will be converted into a tensor and entered into the following two models in parallel: the detection model YOLOv7 and the abnormal behavior recognition module driven by the ASTNet spatiotemporal network. YOLOv7 will output the information of the prediction box, including the prediction results for people and various types of vehicles. ASTNet will output a vector containing classification information, mainly for classification of whether there is an abnormal situation (running or throwing objects).

[0066] Step 33: YOLOv7 will identify people and vehicles appearing in the image and process the detection results through the sort algorithm to track each detection result in the video stream; ASTNet will analyze the behavior in the video stream, mainly responding to fast-moving objects, such as running people or thrown foreign objects.

[0067] Step 34: The detection results of YOLOv7 and sort library will be divided into the following three branches for processing: (1). Enter the passenger flow data statistical analysis module. Through processing, we will obtain the total passenger flow, the time distribution of peak and trough changes in passenger flow, the time each person stays in the screen and the average stay time statistics, hot spot area statistics and other results; (2). Combined with the ASTNet detection results, it will issue warnings and responses to behaviors such as running, falling, staying, and gathering; (3). The detected person frame will be returned to the original image and cut out and sent to the paddleclas module for attribute recognition, and the characteristics of the person such as gender, clothing, and age group will be counted.

[0068] Step 35: The code for each function is shown below:

[0069] If you need to analyze the total passenger flow, passenger flow trend over time, and other results, run the following code:

[0070] python detect_and_track_basic.py --weights yolov7.pt --source file path --classes 0

[0071] The result is a txt file that records statistical data and a line chart that analyzes the trend of changes.

[0072] To count the average length of stay of tourists, hot spots, walking path analysis, and points of interest, run the following code:

[0073] python task2.py --weights yolov7.pt --source file path --classes 0

[0074] The result is a txt file recording statistical data, a heat map corresponding to the original image size, and a picture corresponding to the original image showing the walking paths of each tourist.

[0075] To count the age, gender, clothing, and other attributes of tourists, run the code:

[0076] python detect_and_track_pedestrain_attr.py --weights yolov7.pt --source file path --classes 0

[0077] The result is a txt file that records statistical data.

[0078] To issue an alert for abnormal behavior (running, gathering, throwing objects from high places), run the following code:

[0079] python task_safe.py --custom-data file path

[0080] The result is a video stream, and an alarm is issued in case of emergency.

Claims

1. An AI visual passenger flow counting system based on YOLOv7, characterized by: include: Pedestrian attribute recognition module, trained using pa100K; ASTNet spatiotemporal network-driven abnormal behavior recognition module, trained using ShanghaiTech; YOLOv7 model trained on BDD100K and selfie datasets.

2. An AI visual passenger flow statistics method based on YOLOv7, characterized in that: The following steps are involved: The first step is to collect data Step 11: Prepare the BDD100K dataset, reorganize the label files, remove the images without objects and unclear images, retain the labels of vehicles and pedestrians, and unify the annotation files into txt format; Step 12: To increase the proportion of Asian data, we used online and self-photographed pedestrian images as training data and added them to the BDD100K dataset. The format and training set, validation set, and test set ratios were the same as those of BDD100K. The second step is to train the YOLOv7 model and pedestrian attribute recognition module Step 21: Use the dataset to train the YOLOv7 model using Python code. After the training is completed, the YOLOv7 model training weights are obtained and the model results are evaluated using the mean average precision (mAP). (0.5) ) as the evaluation indicator; the average precision calculation formula is as follows: Among them, IoU represents the intersection-over-union ratio of the predicted box and the actual box, bbox gt Represents the true bounding box, bbox pre Represents the predicted bounding box, mAP (0.5) It means the average precision of the intersection-over-union ratio is greater than or equal to 0.5; i represents the target of the i-th category; TP represents the correctly detected positive sample target, TN represents the correctly detected negative sample target, and FP represents the incorrectly detected positive sample target, that is, false positive; Q represents the total number of detection categories involved in the entire task, and q represents each category in Q; Step 22: Use the pa100K dataset to train the open source model paddleclas. The mean average precision (mAP) is also used as the evaluation metric. Detection boxes are not involved here, so only the correctness of the classification results needs to be considered. Step 23: Use the ShanghaiTech dataset to train ASTNet, and use the area under the ROC curve (AUC) in the field of abnormal behavior recognition as the evaluation metric. The ROC curve is a graphical tool used to represent the performance of a classification model. It uses the true positive rate (TP) and the false positive rate (FP) as the horizontal and vertical coordinates to depict the performance of the classifier at different thresholds. The third step is to build a system to implement the full-link analysis of "detection-attribute-behavior" Step 31: There are two data input methods for the system: local media file input and real-time camera data input. Step 32: The video data to be inspected is read frame by frame, and each frame is converted into a tensor and fed into the following two models in parallel: the detection model YOLOv7 and the abnormal behavior recognition module driven by the ASTNet spatiotemporal network; Step 33: YOLOv7 will identify people and vehicles in the image and process the detection results through the sort algorithm to track each detection result in the video stream. ASTNet will analyze the behavior in the video stream, mainly responding to fast-moving objects, such as running people or thrown foreign objects. Step 34: The detection results of YOLOv7 and sort library will be divided into the following three branches for processing: (1) Enter the passenger flow data statistical analysis module. Through processing, we will obtain the total passenger flow, the time distribution of peak and trough changes in passenger flow, the time each person stays in the screen and the average stay time statistics, hot spot area statistics and other results; (2) Combined with ASTNet detection results, it will provide early warning and response to behaviors such as running, falling, staying, and gathering; (3) The detected person frame will be returned to the original image and cut out and sent to the paddleclas module for attribute recognition, and the person's gender, clothing, age group and other characteristics will be counted.

Images