Bus passenger flow statistical method and electronic equipment
By setting up a target detection model and detection line in bus passenger flow statistics, combined with the id status judgment method, the accuracy of passenger flow statistics in complex scenarios is solved, and more accurate passenger flow statistics are achieved.
Patent Information
- Application Number
- CN202311499697.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-13
- Publication Date
- 2025-05-13
AI Technical Summary
In the statistics of bus passenger flow, it is difficult to accurately count passenger flow in complex scenarios, especially when the installation position of the monitoring equipment is limited, the angle is tilted, and the passengers are crowded to get on and off the bus, it is easy to have missed inspections or repeated counting problems.
A bus passenger flow statistics method is adopted. By setting a target detection model and detection line, it is determined whether the passengers corresponding to the head detection box are on or off the bus. Through the id status judgment method, the initial state and the disappearing state are counted, and the changes in the intermediate state are ignored to reduce false alarms and repeated counting.
It improves the accuracy of passenger flow statistics, reduces missed inspections and repeated counts in complex scenarios, and ensures the accuracy of passenger flow statistics.
Smart Images
Figure CN119992518A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of image recognition, and in particular to a method for counting public transport passenger flow and electronic equipment. Background Art
[0002] Passenger flow counting algorithms have been widely used in the field of public transportation. By counting the number of people getting on and off at each station and the total number of people in the car, the passenger flow of the line at each time can be obtained, which is convenient for the reasonable arrangement of the bus. With the development and widespread application of video surveillance technology, image recognition technology and target tracking technology, monocular cameras have gradually replaced high-cost binocular cameras for public transportation passenger flow detection. Existing technologies have made a lot of optimizations for target detection and target tracking, making the ID tracking technology for getting on and off passengers mature, and the monitoring equipment is usually installed overhead, so that the occlusion problem rarely occurs. However, ID tracking is only the preparatory work before monocular passenger flow counting. In the actual algorithm implementation process, how to accurately count passenger flow is still a relatively difficult problem, especially in complex scenarios, such as due to the limited installation position of the monitoring equipment, the tilted angle, the crowded phenomenon of passengers getting on and off the bus, and the passengers giving way and getting on and off the bus repeatedly. Simply not counting them is likely to result in missed detection, and counting every time a passenger passes the trip wire will result in repeated counting, resulting in errors in passenger flow statistics. Summary of the invention
[0003] In order to solve the above technical problems, the present invention proposes a method and electronic device for counting public transportation passenger flow. The purpose of the present invention is achieved through the following technical solutions:
[0004] A method for counting public transport passenger flow comprises the following steps:
[0005] S1. Set the target detection model and detection line, obtain the monitoring video of the bus door opening and closing process, input the target detection model to obtain the head detection frame in each image frame and set the corresponding ID;
[0006] S2. Determine whether the passenger corresponding to the head detection frame is on or off the bus based on the positional relationship between the coordinates of the center point at the bottom of the head detection frame and the detection line:
[0007] S3. Establish a tracking list for each head detection frame in the monitoring video according to the ID, and only take the initial state, disappearance or end state of the tracking list; if the head detection frame corresponding to the ID is initially on the bus and the state when it disappears or ends is under the bus, the number of people getting off the bus will increase by one; if the initial state is under the bus and the state when it disappears or ends is on the bus, the number of people getting on the bus will increase by one; if the initial state and the state when it disappears or ends are the same, it will not be counted.
[0008] As a further improvement, in step S1, the detection line is set at the outermost edge line at the bottom of the bus passenger boarding and alighting area.
[0009] As a further improvement, in step S2, the method for determining whether the passenger corresponding to the head detection frame is on the bus or off the bus is as follows:
[0010] S2.1 For the head detection frames in the first three image frames and the last image frame of the real-time video, if some of the following conditions are met, it is judged that the person is on the bus, otherwise it is judged that the person is under the bus. The remaining image frames are judged by step S2.2:
[0011]
[0012] Among them, h c is the height of the head detection frame, d is the vertical distance between the center point of the bottom of the head detection frame and the detection line, and y is the height of the head detection frame. ll is the y coordinate of the left endpoint of the detection line, y lr is the y coordinate of the right endpoint of the detection line, y c is the y coordinate of the bottom center point of the head detection frame; abs() means taking the absolute value;
[0013] S2.2 For the remaining image frames, if the center point of the head detection frame is above the detection line, it is judged to be under the bus, otherwise it is judged to be on the bus.
[0014] As a further improvement, in step S1, the target detection model accesses the real-time video when receiving the door opening signal, and stores the IDs of all head detection frames that appear in the door opening and closing process when receiving the door closing signal.
[0015] As a further improvement, the target detection model is a yolov5 head target detection network.
[0016] As a further improvement, in step S3, when establishing the tracking chain list, for the id that has completed the status judgment, if the id has been recorded in the tracking storage chain list, the status of the image frame corresponding to the id is stored in the tracking chain of the id; if it has not been recorded, the tracking chain of this id is initialized in the chain list.
[0017] As a further improvement, the ID whose status changes whether the head detection box is on or off the bus is marked, and the restriction on the disappearance threshold of the ID is lifted, so that the corresponding ID is still recorded by the tracker after disappearing from the screen. When performing DeepSort feature cascade matching, after the tracks with time_since_update=0 are matched, the tracks with changed status are matched first, and then the remaining tracks are matched in order from small to large according to time_since_update; where tracks represents the set of historical tracking chain tracks that have not been deleted in DeepSort, and the time_since_update of each track represents the time from the last update. When a track matches a target in the latest frame, it is updated and the time_since_update will be set to 0. If it fails to match any target, the time_since_update will be increased by one, so the track with a larger time_since_update has not been matched for a longer time, and the priority of matching with the target of the new frame is lower.
[0018] As a further improvement, in step S3, a DeepSort target tracker is used to establish a tracking list.
[0019] As a further improvement, the DeepSort target tracker initializes the ID tracking storage list upon receiving a door opening signal.
[0020] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.
[0021] The beneficial effects of the present invention are:
[0022] First: Use the ID initial and final state judgment method, which only judges by combining the state when each ID is generated and the state when it disappears, ignoring intermediate states such as whether the ID trips the line, reducing false alarms caused by the target detection box moving back and forth near the detection line when queuing to get on the bus, and discarding the counts of passengers temporarily asking for directions and giving way.
[0023] Second: judge the relationship between the head frame and the detection line within a short period of time after the door is opened, determine the initial state of the passengers near the detection line, judge the relationship between the head frame and the detection line within a short period of time after the door is closed, determine the final state of the passengers near the detection line, and solve the problem of getting on the bus but not passing the detection line.
[0024] Third: When tracking the target, mark the ID whose status has changed, extend the disappearance time threshold until the door closing signal is received, and prioritize matching during cascade matching to solve the problem of repeated accumulation caused by the same passenger getting on and off the bus multiple times at the same station and disappearing in the screen during the switching of IDs. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.
[0026] Figure 1 It is a schematic diagram of the overall process of the present invention;
[0027] Figure 2 A picture of passengers queuing up to board the bus;
[0028] Figure 3 This is a picture of passengers passing through the detection line;
[0029] Figure 4 A picture of passengers lingering near the inspection line due to waiting;
[0030] Figure 5 This is a picture of passengers who have not passed the detection line but have boarded the bus;
[0031] Figure 6 A diagram for passengers preparing to get off the bus but not passing through the detection line, but still needing to be counted;
[0032] Figure 7 It is a flow chart of the passenger flow statistics method of the present invention;
[0033] Figure 8 The distance d between the target with ID 327 and the detection line is less than the height h of its detection box, and the state is judged to be on the car;
[0034] Fig. 9 The distance d between the target with ID 1 and the straight line is greater than the height h of its detection box, and the state is judged to be under the car;
[0035] Fig.10 This is the original DeepSort matching partial flow chart;
[0036] Fig.11 This is the optimized matching flow chart of the present invention. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solutions and advantages of the invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.
[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0039] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0040] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or description". Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.
[0041] like Figure 1 The present invention proposes a complex scene monocular passenger flow counting method based on image deep learning. The principle of the present invention is as follows Figure 1 As shown, the specific working method includes the following steps:
[0042] Step 1: Target detection and tracking. After receiving the door opening signal, access the real-time video, obtain the image frame, pre-process the image frame, and input it into a head target detection network (yolov5s) to obtain the head detection frame. Input the head detection frame between the left and right points of the detection line into the target tracker (DeepSort) to obtain the id of each head detection frame. When the target tracker (DeepSort) receives the door opening signal, it initializes the id tracking storage list and stores the status of the head detection frame corresponding to all ids that appear in the door opening and closing process in each frame (on the car or under the car).
[0043] Among them, Yolov5 is an object detection method based on deep learning, which has the characteristics of high efficiency and high precision. Among them, Yolov5s is relatively computationally small and fast. Yolov5s is an object detection algorithm based on deep convolutional neural network (DCNN). Its network structure consists of a series of convolutional layers and some additional auxiliary layers. YOLOv5s consists of three parts, namely backbone, neck, and head. Backbone: The backbone network, most of the time, refers to the network for extracting features. Its function is to extract information from the picture for use by the subsequent network. The backbone network can directly load the official trained model parameters. In addition, after it, add the network you need to make the custom model more suitable for actual use. Neck: Placed between the backbone and the head, further use the features extracted by the backbone to improve the robustness of the model. Head backbone: Get the network output, and the head uses the previously extracted features to make predictions.
[0044] Compared with traditional object detection algorithms such as RCNN, Fast RCNN, Faster RCNN and SSD, Yolov5 has the following advantages:
[0045] 1. Efficient
[0046] Due to the use of some acceleration technologies based on new convolution modules, such as the CSP (Cross Stage Partial) module, Yolov5s is faster and less computationally intensive than traditional object detection algorithms.
[0047] 2. High accuracy
[0048] The backbone of Yolov5s uses a relatively novel SPP (Spatial Pyramid Pooling) module. Compared with the backbones of other object detection algorithms (such as ResNet, DarkNet, etc.), SPP can combine feature maps of various sizes to generate a global feature description, so it is more robust to small objects, occlusion, lighting and other problems. In addition, Yolov5s also uses a multi-scale training method to perform appropriate processing on objects of different scales, thereby improving the accuracy of classification and positioning.
[0049] 3. Clear network structure
[0050] The network structure of Yolov5s is very clear and highly modularized, making it easy to adjust and design the corresponding model. In addition, there are many relatively novel convolutional models in Yolov5s, such as the CSP module, which can provide reference for subsequent model design.
[0051] DeepSort Object Tracker:
[0052] The main idea of DeepSORT is to combine the two tasks of target detection and target tracking. First, the target detection algorithm (Faster R-CNN, etc.) is used to detect the position and bounding box of the target object in each frame. Then, the feature representation of the target is extracted through a deep learning model (such as CNN), and each target is matched with the tracked target in the previous frame. Factors such as the feature similarity and motion consistency of the target are considered in the matching process to determine the identity and trajectory of the target. One of the key contributions of DeepSORT is the use of a powerful appearance feature descriptor that can accurately distinguish the similarities between different targets. DeepSORT also supports long-term tracking by handling complex situations such as the disappearance and reappearance of the target. The main technical features include cascade matching, ReID network, Mahalanobis distance and cosine distance.
[0053] For cascade matching: When a target is obscured for a long time, the uncertainty of the Kalman filter prediction will greatly increase and the observability in the state space will be greatly reduced.
[0054] If two trackers compete for the matching right of the same detection result at this time, the Mahalanobis distance of the track with a longer occlusion time is often smaller, making the detection result more likely to be associated with the track with a longer occlusion time. This undesirable effect often destroys the continuity of tracking. Understand it this way. Assuming that the covariance matrix is a normal distribution, continuous prediction without updating will cause the variance of this normal distribution to become larger and larger. Then the points far from the mean Euclidean distance may obtain the same Mahalanobis distance value as the points closer in the previous distribution. Therefore, DeepSORT uses cascade matching to give priority to more frequently occurring targets. Its matching process is a cycle. That is, from the track with missing age = 0, each frame is matched, and the track with no missing age = 30, the track with the maximum time of 30 frames of missing track is matched one by one with the detection result. In other words, the track that has not been lost is given priority matching rights, and the track with the longest loss is matched last. Here, 30 frames are lost and there is still a chance to match, which is to track the occluded target again. The distance metric of cascade matching is a fusion of Mahalanobis distance and cosine distance. Two distance thresholds are set and filtered first. Then the fused distance is used for Hungarian algorithm matching, which is the matching of detection box and prediction box. And the matching has priority. Those trajectories that have been matched in history are matched first, and those trajectories that have not been matched for a long time in history are matched last. In other words, the possibility of occlusion is relatively small, and most of the unmatched trajectories may be that the target has disappeared.
[0055] For the ReID network, DeepSORT uses a simple CNN to extract the appearance features of the detected object. After each detection and tracking, the appearance features of the object are extracted and saved, and a maximum of 100 frames are saved. Each subsequent step is to perform a similarity calculation between the appearance features of the detected object in the current frame and the previously stored appearance features. This similarity will serve as an important basis for discrimination. Deep Sort uses the Cosine deep feature network trained on a large-scale person re-identification dataset, which contains more than 1,100,000 images of 1,261 pedestrians, making it very suitable for deep metric learning in a person tracking environment. The Cosine deep feature network uses a wide residual network with 2 convolutional layers and 6 residual blocks. The L2 normalization layer can calculate the similarity between different pedestrians to be compatible with the cosine appearance metric. By calculating the cosine distance between pedestrians, the smaller the cosine distance, the more similar the two pedestrian images are.
[0056] DeepSort uses Mahalanobis distance to represent the distance from the detection box to the trajectory. Mahalanobis distance is an "enhanced version of Euclidean distance". Mahalanobis distance avoids the risk of different variances of data features in Euclidean distance, and adds a covariance matrix in the calculation. Its purpose is to normalize the variance, so that the so-called "distance" is more in line with the data features and practical significance. Mahalanobis distance is the Euclidean distance scaled by rotation transformation. It incorporates the covariance matrix of the sample into the distance metric calculation, which is equivalent to the correction of Euclidean distance. Mahalanobis distance completes orthogonality, solves the problem of correlation between features, and contains standardization, which solves the problem of inconsistent scales between features. Mahalanobis distance can be used to determine the distance from a point to a distribution. Mahalanobis distance is a method of representing the covariance distance of data and calculating the similarity of two unknown sample sets. Formula: Mahalanobis distance is the Euclidean distance scaled by rotation transformation, so the calculation formula of Mahalanobis distance can be derived from Euclidean distance.
[0057] Since the Mahalanobis distance has some problems in measuring after occlusion, DeepSort adds the image feature similarity to make a comprehensive judgment. Cosine distance is a similarity measurement method. Mahalanobis distance distinguishes based on position, while cosine distance distinguishes based on direction and features. Cosine similarity is a commonly used method to measure the similarity between vectors. It can be used to calculate the cosine value of the angle between two vectors. In image similarity calculation, the image can be converted into a feature vector. In DeepSort, the ReID network is used, and then cosine similarity is used to compare the similarity of these feature vectors.
[0058] Step 2: Determine the ID status of a single image frame. The specific steps are:
[0059] If the image frame is within three frames after receiving the door opening signal (three frames are required for the tracker to form an ID determination state), or the image frame is a frame that receives a signal, the ID whose bottom edge center point is above the detection line is judged as being on or off the vehicle. If the detection frame satisfies the following formula, it means that its actual state is on the vehicle. If not, it means that it is under the vehicle:
[0060]
[0061]
[0062] Among them, h c is the height of the detection frame, d represents the distance between the center point at the bottom of the detection frame and the detection line, and y ll is the y coordinate of the left endpoint of the detection line, y lr is the y coordinate of the right endpoint of the detection line, y c is the y coordinate of the center point at the bottom of the detection frame. This means that if the distance from the center point at the bottom of the detection frame to the straight line is less than the height of the detection frame, the state is determined to be on the car (e.g. Figure 5 shown), otherwise under the vehicle.
[0063] 2) If the image frame does not meet the conditions shown in 1), directly determine the relationship between the center point of the bottom edge of the detection frame and the straight line. If it is above the straight line, the frame with the ID is under the car. If it is below the straight line, the frame with the ID is on the car.
[0064] For the ID that has completed status judgment in steps 1) and 2), if the ID has been recorded in the tracking storage linked list, the status of the frame of the ID is stored in the tracking chain of the ID. If it has not been recorded, the tracking chain of this ID is initialized in the linked list.
[0065] Through the above steps, the present invention solves the problem that the vehicle actually gets on the bus but does not pass the detection line, thereby making passenger flow statistics more accurate.
[0066] Step 3: Target tracking optimization. In DeepSort, for all the tracking chains in the determined state, the detection frame of the current frame is cascade matched according to the time from the last update of the tracking chain to the largest. And when the unmatched tracking chain has an unupdated time that exceeds the maximum limit, the tracking chain will be deleted (such as Fig.10 As shown). The present invention optimizes this cascade matching method, marks the ID whose state changes in step 2, removes the limit of its ID disappearance threshold, so that it can still be recorded by the tracker after disappearing from the screen, and will not be deleted until the door is closed. When performing DeepSort feature cascade matching, after the tracking chain matched on the previous frame is matched (the priority is always kept first), the tracking chain whose state changes in step 2 is matched first, and then the remaining tracking chains from small to large are matched according to the update time, so that the tracker can better retrieve this ID (such as Fig.11 shown).
[0067] Step 4: ID initial and final state judgment method. After receiving the door closing signal, the tracking list is counted. For each ID recorded in the tracking table, only the initial state, the disappearance state or the end state are taken. If the initial state is on the bus and the end state is under the bus, the number of people getting off the bus will increase by one; if the initial state is under the bus and the end state is on the bus, the number of people getting on the bus will increase by one; if the initial state and the end state are the same, no count will be made. That is, the present invention adopts the ID initial and final state judgment method, which only judges by combining the state when each ID is generated and the state when it disappears, ignoring the intermediate states such as whether the ID is tripping the line, reducing the false alarms caused by the target detection frame moving back and forth near the detection line when queuing to get on the bus, and discarding the counts of passengers temporarily asking for directions and giving way, thereby making passenger flow statistics more accurate.
[0068] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for counting public transport passenger flow, characterized in that: The steps include: S1. Set the target detection model and detection line, obtain the monitoring video of the bus door opening and closing process, input the target detection model to obtain the head detection frame in each image frame and set the corresponding ID; S2. Determine whether the passenger corresponding to the head detection frame is on or off the bus based on the positional relationship between the coordinates of the center point at the bottom of the head detection frame and the detection line: S3. Establish a tracking list for each head detection frame in the monitoring video according to the ID, and only take the initial state, disappearance or end state of the tracking list; if the head detection frame corresponding to the ID is initially on the bus and the state when it disappears or ends is under the bus, the number of people getting off the bus will increase by one; if the initial state is under the bus and the state when it disappears or ends is on the bus, the number of people getting on the bus will increase by one; if the initial state and the state when it disappears or ends are the same, it will not be counted.
2. The public transportation passenger flow statistics method according to claim 1, characterized in that: In step S1, the detection line is set at the outermost edge of the bottom of the bus passenger boarding and alighting area.
3. The public transportation passenger flow statistics method according to claim 2, characterized in that: In step S2, the method for determining whether the passenger corresponding to the head detection frame is on the bus or off the bus is as follows: S2.1 For the head detection frames in the first three and last image frames of the real-time video, if the following conditions are met, it is judged that the person is on the bus, otherwise it is judged that the person is under the bus. The remaining image frames are judged by step S2.2: Among them, h c is the height of the head detection frame, d is the vertical distance between the center point of the bottom of the head detection frame and the detection line, and y is the height of the head detection frame. ll is the y coordinate of the left endpoint of the detection line, y lr is the y coordinate of the right endpoint of the detection line, y c is the y coordinate of the bottom center point of the head detection frame; abs() means taking the absolute value; S2.2 For the remaining image frames, if the center point of the head detection frame is above the detection line, it is judged to be under the bus, otherwise it is judged to be on the bus.
4. The public transportation passenger flow statistics method according to claim 1, characterized in that: In step S1, the monitoring video starts when the bus door opens and ends when the bus door closes.
5. The public transportation passenger flow statistics method according to claim 1, characterized in that: The target detection model is the yolov5 head target detection network.
6. The public transportation passenger flow statistics method according to claim 1, characterized in that: In step S3, when establishing the tracking chain list, for the id that has completed the status judgment, if the id has been recorded in the tracking storage chain list, the status of the image frame corresponding to the id is stored in the tracking chain of the id. If it has not been recorded, the tracking chain of this id is initialized in the chain list.
7. The public transportation passenger flow statistics method according to claim 6, characterized in that: Mark the IDs whose status of the head detection frame changes from being on or off the bus, and remove the corresponding ID disappearance threshold restriction so that the corresponding ID is still recorded by the tracker after disappearing from the screen. When performing DeepSort feature cascade matching, after matching the tracks with time_since_update=0, first match the tracks with changed status, and then match the remaining tracks in the order of time_since_update from small to large; Tracks represents the set of historical tracking chain tracks that have not been deleted in DeepSort, and the time_since_update of each track represents the time since its last update.
8. The public transportation passenger flow statistics method according to claim 1, characterized in that: In step S3, a DeepSort target tracker is used to establish a tracking list.
9. The method for counting public transportation passenger flow according to claim 8, characterized in that: The DeepSort target tracker initializes the ID tracking storage list upon receiving the door opening signal.
10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.