Passenger flow statistical method, device and equipment and computer readable storage medium

Abstract

The invention discloses a passenger flow statistical method, and the method comprises the steps: carrying out the detection of a current frame image through a target detection model, and obtaining a detection frame corresponding to each target in the current frame image; obtaining a preset number of historical image frames corresponding to the current frame image; determining a tracking frame corresponding to each detection frame from each historical image frame through feature matching and detection frame distance matching; according to the detection frames and the tracking frames, tracks corresponding to the targets are updated; and performing passenger flow statistics according to each track. By applying the passenger flow statistical method provided by the invention, the counting efficiency is improved, the influence of factors such as light and shielding is reduced, and the user privacy is effectively protected. The invention further discloses a passenger flow statistics device and equipment and a storage medium, which have corresponding technical effects.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a passenger flow counting method, device, equipment and computer-readable storage medium. Background Art

[0002] Passenger counting refers to the process of real-time monitoring and counting the number of people entering or passing through a specific area in a specific place, such as a shopping mall, station, airport, museum, etc.

[0003] The current passenger flow counting methods mainly include: manual counting, mechanical counting, optical counting, and statistics using computer vision technology and image processing algorithms. Manual counting refers to manually recording the number of people entering or leaving within a specific time period; mechanical counting refers to counting using mechanical devices, usually with a simple entry counter installed at the door; optical counting technology uses infrared transmitters and receivers to detect the entry and exit of people, and when someone passes through the infrared, the device will record a count; using computer vision technology and image processing algorithms to count people, the system can automatically identify, classify and track people through cameras monitoring the entry and exit areas.

[0004] However, the above passenger counting methods all have certain disadvantages. First, manual counting is inefficient and is greatly affected by human factors. Second, the mechanical structure of mechanical counting is complex and inconvenient to maintain. Third, optical counting is easily affected by factors such as light and occlusion. Finally, when using computer vision technology and image processing algorithms to count people, the complexity of the environment may lead to a decrease in counting accuracy, and occlusion may affect the accuracy of counting. When using video surveillance and face recognition technology, it will damage user privacy.

[0005] To sum up, how to effectively solve the problems of low manual counting efficiency, large human factors, complex mechanical structure, inconvenient maintenance, susceptibility to factors such as light and occlusion, and damage to user privacy is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0006] The purpose of this application is to provide a passenger flow counting method, which improves counting efficiency, reduces the impact of factors such as light and occlusion, and effectively protects user privacy; another purpose of this application is to provide a passenger flow counting device, equipment and computer-readable storage medium.

[0007] In order to solve the above technical problems, this application provides the following technical solutions:

[0008] A passenger flow counting method, comprising:

[0009] Detecting the current frame image using the target detection model to obtain detection frames corresponding to the targets contained in the current frame image;

[0010] Acquire a preset number of historical image frames corresponding to the current frame image;

[0011] Determine the tracking frames corresponding to the respective detection frames from each historical image frame through feature matching and detection frame distance matching;

[0012] Update the trajectories corresponding to each target according to each detection frame and each tracking frame;

[0013] Passenger flow statistics are performed based on each trajectory.

[0014] In a specific implementation of the present application, detecting the current frame image using a target detection model includes:

[0015] Use the head and shoulder detection model to detect the current frame image.

[0016] In a specific implementation of the present application, after obtaining the detection frames corresponding to the respective targets contained in the current frame image, and before determining the tracking frames corresponding to the respective detection frames from the historical image frames by feature matching and detection frame distance matching, the method further includes:

[0017] Get the aspect ratio of each detection frame;

[0018] Determine the detection frame whose aspect ratio is greater than a preset value as an invalid detection frame;

[0019] Accordingly, the tracking frames corresponding to the respective detection frames are determined from the historical image frames through feature matching and detection frame distance matching, including:

[0020] The tracking frames corresponding to the detection frames except the invalid detection frames are determined from each historical image frame through feature matching and detection frame distance matching.

[0021] In a specific implementation of the present application, determining the tracking frames corresponding to the respective detection frames from the respective historical image frames by feature matching and detection frame distance matching includes:

[0022] Match the historical detection frame sets corresponding to each detection frame from each historical image frame through the feature extraction model;

[0023] For each detection frame, calculating the cosine distance between the detection frame and each historical detection frame in the corresponding historical detection frame set;

[0024] Determine the minimum cosine distance corresponding to each detection frame;

[0025] Determine the historical detection frame corresponding to the minimum cosine distance less than or equal to the preset distance value as the tracking frame of the corresponding detection frame;

[0026] For a detection frame whose minimum cosine distance is greater than the preset distance value, the Hungarian algorithm is used to match the detection frame with each historical detection frame in the corresponding historical detection frame set, and the successfully matched historical detection frame is determined as the tracking frame of the corresponding detection frame;

[0027] For the detection frame that fails to be matched using the Hungarian algorithm, calculate the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set;

[0028] The tracking frame corresponding to the detection frame is determined according to each Euclidean distance.

[0029] In a specific implementation of the present application, calculating the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set includes:

[0030] Acquire a nearest neighbor historical detection frame in a nearest neighbor image frame of the current frame image from a set of historical detection frames corresponding to the detection frame;

[0031] When it is determined that the length change and / or width change of the detection frame compared to the nearest neighbor historical detection frame exceeds a preset change threshold, filtering out the detection frame;

[0032] For the detection frames remaining after filtering, determining the track disappearance time according to the detection frames and the corresponding nearest neighbor historical detection frames;

[0033] Acquire the moving speed of the detection frame, and calculate the product of the moving speed and the track disappearance time to obtain the predicted distance;

[0034] Obtaining the actual distance between the detection frame and the corresponding nearest neighbor historical detection frame;

[0035] When it is determined that the actual distance is less than the predicted distance, determining that the target corresponding to the detection frame leaves;

[0036] When it is determined that the actual distance is greater than or equal to the predicted distance, the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set is calculated.

[0037] In a specific implementation of the present application, passenger flow statistics are performed according to each track, including:

[0038] Get the preset quadrilateral passenger flow statistics area;

[0039] Passenger flow statistics are performed according to the relationship between each trajectory and the quadrilateral passenger flow statistics area.

[0040] In a specific implementation of the present application, performing passenger flow statistics according to the relationship between each trajectory and the quadrilateral passenger flow counting area includes:

[0041] When the trajectory passes through any two store-passing lines in the quadrilateral passenger flow statistical area, determining that the target corresponding to the trajectory is a store-passing line;

[0042] When the trajectory is generated from the quadrilateral passenger flow statistical area and passes through any store-passing line in the quadrilateral passenger flow statistical area, judging whether the target corresponding to the trajectory is leaving the store or passing through the store according to the trajectory;

[0043] When the trajectory passes through the store entry and exit line in the quadrilateral customer flow statistical area, and the movement direction corresponding to the trajectory is from the inside of the quadrilateral customer flow statistical area to the outside of the quadrilateral customer flow statistical area, it is determined that the target corresponding to the trajectory is entering the store;

[0044] When the trajectory passes through the store entry and exit line in the quadrilateral customer flow statistical area, and the movement direction corresponding to the trajectory is from the outside of the quadrilateral customer flow statistical area to the inside of the quadrilateral customer flow statistical area, it is determined that the target corresponding to the trajectory is exiting the store.

[0045] A passenger flow counting device, comprising:

[0046] A detection frame acquisition module is used to detect the current frame image using the target detection model to obtain detection frames corresponding to the targets contained in the current frame image;

[0047] A historical image frame acquisition module, used to acquire a preset number of historical image frames corresponding to the current frame image;

[0048] A tracking frame determination module is used to determine the tracking frames corresponding to each detection frame from each historical image frame through feature matching and detection frame distance matching;

[0049] A trajectory update module is used to update the trajectory corresponding to each target according to each detection frame and each tracking frame;

[0050] The passenger flow statistics module is used to perform passenger flow statistics based on each trajectory.

[0051] A passenger flow counting device, comprising:

[0052] Memory for storing computer programs;

[0053] A processor is used to implement the steps of the passenger flow counting method as described above when executing the computer program.

[0054] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the passenger flow counting method described above are implemented.

[0055] The passenger flow counting method provided in the present application detects the current frame image using a target detection model to obtain detection frames corresponding to each target contained in the current frame image; obtains a preset number of historical image frames corresponding to the current frame image; determines the tracking frames corresponding to each detection frame from each historical image frame through feature matching and detection frame distance matching; updates the trajectories corresponding to each target according to each detection frame and each tracking frame; and performs passenger flow counting according to each trajectory.

[0056] It can be seen from the above technical solution that by obtaining the detection frames corresponding to each target contained in the current frame image, and determining the tracking frames corresponding to each detection frame from the historical image frame, the trajectories corresponding to each target in each detection frame and each tracking frame are updated to achieve passenger flow statistics according to the updated trajectories. Compared with manual counting, it greatly improves the counting efficiency, avoids the influence of human factors, does not require complex mechanical structures, greatly reduces the influence of factors such as light and occlusion, does not require face recognition, and effectively protects user privacy.

[0057] Correspondingly, the present application also provides a passenger flow counting device, equipment and computer-readable storage medium corresponding to the above-mentioned passenger flow counting method, which have the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 This is a flow chart of an implementation of the passenger flow statistics method in the embodiment of the present application;

[0060] Figure 2 Another implementation flow chart of the passenger flow statistics method in the embodiment of the present application;

[0061] Figure 3 This is a schematic diagram of performing passenger flow counting based on a quadrilateral passenger flow statistical area in an embodiment of the present application;

[0062] Figure 4 This is a structural block diagram of a passenger flow counting device in an embodiment of the present application;

[0063] Figure 5 This is a structural block diagram of a passenger flow counting device in an embodiment of the present application;

[0064] Figure 6 A schematic diagram of the specific structure of a passenger flow counting device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] Currently, more and more public places and commercial organizations need to accurately understand the passenger flow in order to effectively manage resources and make decisions. Therefore, passenger counting technology came into being and gradually became an indispensable part of modern commerce and urban management.

[0066] The application of passenger counting has a wide range of significance. For example, in the retail industry, merchants can optimize store layout, adjust product display, formulate promotion strategies and arrange staff work schedules by analyzing passenger flow. In transportation hubs, passenger counting can help management departments better understand passenger mobility, reasonably arrange trains and improve service quality. In addition, urban planning and public safety management are also inseparable from the support of passenger counting technology.

[0067] Passenger flow counting technology has gone through a development process from traditional manual counting to modern intelligent counting. The current passenger flow counting methods have problems such as low manual counting efficiency, large human factors, complex mechanical structure, inconvenient maintenance, easy to be affected by factors such as light and occlusion, and damage to user privacy.

[0068] To this end, the passenger flow counting method provided in this application improves counting efficiency, reduces the impact of factors such as light and occlusion, and effectively protects user privacy.

[0069] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0070] See also Figure 1 , Figure 1 This is a flow chart of an implementation of the passenger flow counting method in the embodiment of the present application. The method may include the following steps:

[0071] S101: Detect the current frame image using the target detection model to obtain detection frames corresponding to the targets contained in the current frame image.

[0072] A target detection model is obtained by pre-training, and the current frame image is detected using the target detection model to obtain detection frames corresponding to each target contained in the current frame image.

[0073] Object detection model is an important technology in the field of computer vision, which aims to identify and locate specific objects from images or videos. This technology is widely used in many fields such as autonomous driving, security monitoring, face recognition, medical image analysis, etc.

[0074] Object detection not only needs to identify the object category in the image, but also needs to determine the location of the object in the image. Bounding boxes are usually used to represent these locations. Therefore, object detection mainly consists of two parts: classification task and positioning task. Object detection usually requires a large number of annotated data sets. Common data sets include COCO, Pascal VOC, etc. The annotation method uses bounding boxes for annotation, and each bounding box is associated with a corresponding category label.

[0075] Detection models can be divided into two categories: one is a two-stage detection model; the other is a one-stage detection model. Two-stage detection models usually include traditional R-CNN, Fast R-CNN, and Faster R-CNN, which first generate candidate regions and then classify and regress these regions. They usually have high accuracy but are relatively slow; one-stage detection models usually include detection frameworks such as YOLO and SSD. These models integrate the target detection process into a forward propagation, which is fast and suitable for real-time applications, but generally have slightly lower accuracy than two-stage models.

[0076] In addition, the latest detection models also use the Transformers framework. For example, DETR is a major innovation in the field of target detection, which completely changes the traditional target detection paradigm. DETR transforms the target detection model from a region proposal-based method to a framework that directly predicts bounding boxes and applies NMS, simplifying the target detection process. Some researchers have begun to try to introduce dynamic convolutions in convolutional neural networks to dynamically adjust convolution parameters according to input features to improve model flexibility and performance. Some researchers have begun to explore target detection methods based on graph neural networks, which use graph structures to model the relationship between targets to improve detection accuracy. With the continuous improvement of data sets, algorithms, and hardware performance, the accuracy and real-time performance of detection models are also constantly improving.

[0077] S102: Obtain a preset number of historical image frames corresponding to the current frame image.

[0078] After detecting and obtaining the detection frames corresponding to the respective targets contained in the current frame image, a preset number of historical image frames corresponding to the current frame image are obtained.

[0079] It should be noted that the preset number can be set and adjusted according to actual conditions, and the embodiment of the present application does not limit this. For example, it can be set to 100.

[0080] S103: Determine the tracking frames corresponding to the detection frames from the historical image frames through feature matching and detection frame distance matching.

[0081] After obtaining the detection frames corresponding to the targets contained in the current frame image and acquiring the historical image frames, the tracking frames corresponding to the detection frames are determined from the historical image frames through feature matching and detection frame distance matching. Generally, the image frame closest to the current frame image is selected from the historical image frames containing the unobstructed historical detection frames corresponding to the detection frames, and the historical detection frame corresponding to the detection frame in the selected historical image frames is determined as the tracking frame corresponding to the detection frame. By combining feature matching with detection frame distance matching for tracking, the possibility of losing the target is reduced when frame loss occurs, the probability of successful matching can be increased, and the situation of mismatching can be reduced.

[0082] You can also pre-classify each detection frame. If the detection confidence is higher than 0.5, the detection frame is classified as a high-confidence group detection frame, and the confidence between 0.3 and 0.5 is a low-confidence group detection frame. Only the high-confidence group detection frame is tracked and matched.

[0083] S104: Update the trajectories corresponding to the respective targets according to the respective detection frames and the respective tracking frames.

[0084] After determining the tracking frames corresponding to the detection frames, the trajectories corresponding to the targets are updated according to the detection frames and the tracking frames. For example, based on the principle of smooth connection, a smooth connection can be established between the detection frames and the tracking frames according to the trajectories corresponding to the detection frames.

[0085] S105: Perform passenger flow statistics according to each trajectory.

[0086] After updating the trajectories corresponding to the respective targets, passenger flow statistics are performed according to the trajectories. By performing passenger flow statistics once after acquiring an image frame, the real-time performance of passenger flow statistics is improved.

[0087] It can be seen from the above technical solution that by obtaining the detection frames corresponding to each target contained in the current frame image, and determining the tracking frames corresponding to each detection frame from the historical image frame, the trajectories corresponding to each target in each detection frame and each tracking frame are updated to achieve passenger flow statistics according to the updated trajectories. Compared with manual counting, it greatly improves the counting efficiency, avoids the influence of human factors, does not require complex mechanical structures, greatly reduces the influence of factors such as light and occlusion, does not require face recognition, and effectively protects user privacy.

[0088] It should be noted that, based on the above embodiments, the embodiments of the present application also provide corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.

[0089] In a specific implementation of the present application, step S103 may include the following steps:

[0090] Step 1: Match the historical detection frame set corresponding to each detection frame from each historical image frame through the feature extraction model;

[0091] Step 2: For each detection frame, calculate the cosine distance between the detection frame and each historical detection frame in the corresponding historical detection frame set;

[0092] Step 3: Determine the minimum cosine distance corresponding to each detection frame;

[0093] Step 4: Determine the historical detection frame corresponding to the minimum cosine distance that is less than or equal to the preset distance value as the tracking frame of the corresponding detection frame;

[0094] Step 5: For the detection frame whose minimum cosine distance is greater than the preset distance value, the Hungarian algorithm is used to match the detection frame with each historical detection frame in the corresponding historical detection frame set, and the successfully matched historical detection frame is determined as the tracking frame of the corresponding detection frame;

[0095] Step 6: For the detection frame that fails to be matched using the Hungarian algorithm, calculate the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set;

[0096] Step 7: Determine the tracking frame corresponding to the detection frame according to each Euclidean distance.

[0097] For the convenience of description, the above seven steps can be combined for explanation.

[0098] When determining the tracking frame corresponding to each detection frame, the historical detection frame set corresponding to each detection frame is matched from each historical image frame through the feature extraction model. For each detection frame, the cosine distance between the detection frame and each historical detection frame in the corresponding historical detection frame set is first calculated to determine the minimum cosine distance corresponding to each detection frame. A distance threshold is set in advance, and the historical detection frame corresponding to the minimum cosine distance less than or equal to the preset distance value is determined as the tracking frame of the corresponding detection frame. By using cosine distance calculation to determine the tracking frame, the tracking frame can be quickly determined.

[0099] The tracking frame may be located in the previous frame of the current frame, or in other historical image frames except the previous frame. For example, when the target is occluded, there is overlap between the detection frames. The degree of overlap is used to determine whether the target is severely occluded. The features extracted from severely occluded targets are highly redundant and cannot accurately represent the feature information of the target. That is, the tracking frame determined at this time is invalid.

[0100] For the detection frame whose minimum cosine distance is greater than the preset distance value, the Hungarian algorithm is used to match the detection frame with each historical detection frame in the corresponding historical detection frame set, and the successfully matched historical detection frame is determined as the tracking frame of the corresponding detection frame. By using the Hungarian algorithm to further match the tracking frame for the detection frame whose minimum cosine distance is greater than the preset distance value, the accuracy of the matched tracking frame is improved.

[0101] For the detection frame that failed to match using the Hungarian algorithm, the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set is calculated, and the tracking frame of the corresponding detection frame is determined according to each Euclidean distance. By using the Euclidean distance again to determine the corresponding tracking frame for the detection frame that failed to match using the Hungarian algorithm, the intuitiveness of the Euclidean distance is fully utilized.

[0102] It should be noted that the preset distance value can be set and adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0103] In a specific implementation of the present application, calculating the Euclidean distance between a detection frame and each historical detection frame in a corresponding historical detection frame set may include the following steps:

[0104] Step 1: Obtain the nearest historical detection frame in the nearest neighbor image frame of the current frame image from the historical detection frame set corresponding to the detection frame;

[0105] Step 2: When it is determined that the length change and / or width change of the detection frame compared to the nearest neighbor historical detection frame exceeds a preset change threshold, the detection frame is filtered out;

[0106] Step 3: For the remaining detection frames after filtering, determine the track disappearance time based on the detection frame and the corresponding nearest neighbor historical detection frame;

[0107] Step 4: Get the moving speed of the detection frame, and calculate the product of the moving speed and the track disappearance time to get the predicted distance;

[0108] Step 5: Get the actual distance between the detection box and the corresponding nearest neighbor historical detection box;

[0109] Step 6: When it is determined that the actual distance is less than the predicted distance, the target corresponding to the detection frame is determined to leave;

[0110] Step 7: When it is determined that the actual distance is greater than or equal to the predicted distance, the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set is calculated.

[0111] For the convenience of description, the above seven steps can be combined for explanation.

[0112] For the detection frames that failed to match using the Hungarian algorithm, the nearest neighbor historical detection frames in the nearest neighbor image frame of the current frame image are obtained from the historical detection frame set corresponding to the detection frame. Preset the change thresholds for the length and width of the detection frame. When it is determined that the length change of the detection frame compared with the nearest neighbor historical detection frame exceeds the preset change threshold, or the width change exceeds the preset change threshold, or the length change and width change exceed the preset change threshold, the detection frame is filtered out. By filtering out the detection frames whose length change, width change, or both length change and width change exceed the preset change threshold according to the length and width changes of the detection frame, the detection frames whose length and width changes suddenly change are effectively identified, and further trajectory tracking of the detection frames whose length and width changes suddenly change is avoided.

[0113] For the remaining detection frames after filtering, the track disappearance time is determined based on the detection frame and the corresponding nearest neighbor historical detection frame, that is, the track disappearance time is obtained by subtracting the last update time of the target from the current time, and the moving speed of the detection frame is obtained. The product between the moving speed and the track disappearance time is calculated to obtain the predicted distance, and the actual distance between the detection frame and the corresponding nearest neighbor historical detection frame is obtained. When it is determined that the actual distance is less than the predicted distance, it is determined that the target corresponding to the detection frame has left, so as to stop further tracking of the target that has left. When it is determined that the actual distance is greater than or equal to the predicted distance, the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set is calculated. This achieves effective screening of detection frames that require Euclidean distance calculation and improves resource utilization.

[0114] For each detection frame, the nearest neighbor image frame is the historical detection frame that contains the detection frame and is closest to the current image frame where the detection frame is located in time sequence. The nearest neighbor historical detection frame is the historical detection frame corresponding to the detection frame in the nearest neighbor image frame.

[0115] See also Figure 2 , Figure 2 This is another implementation flow chart of the passenger flow counting method in the embodiment of the present application. The method may include the following steps:

[0116] S201: Detect the current frame image using a head and shoulder detection model to obtain detection frames corresponding to the respective targets contained in the current frame image.

[0117] The head and shoulder detection model is selected in advance as the target detection model. Based on the general target detection model, the labeled pedestrian segmentation dataset is used for training. Based on the pre-trained weights, the table detection model is obtained by transfer learning training. The target detection model outputs an image with detection frames marked for pedestrians. The head and shoulder detection model is used to detect the current frame image, and the detection frames corresponding to each target contained in the current frame image are obtained. By selecting the head and shoulder detection model as the target detection model, the situation where pedestrian tracking matching errors caused by pedestrian occlusion are avoided, and the passenger flow counting effect is improved.

[0118] The target detection model can be obtained by multi-target detection model transfer learning training, and the target detection model can be flexibly selected according to the actual scenario. Detection models of traditional computer vision methods, such as feature-based detection, sliding window methods, etc. Methods based on deep learning include convolutional neural networks and end-to-end learning neural networks. The embodiment of the present application can use a single-stage detection YOLO series model to achieve real-time detection of targets. If you want to obtain a detection with higher detection accuracy, a two-stage detection model can be used. The detection model collects store pedestrian data sets for annotation, and the model finally outputs information such as the location, category, and confidence of the detection box.

[0119] S202: Obtain the aspect ratio of each detection frame.

[0120] After detecting and obtaining the detection frames corresponding to the respective targets contained in the current frame image, the aspect ratio of each detection frame is obtained.

[0121] S203: Determine a detection frame whose aspect ratio is greater than a preset value as an invalid detection frame.

[0122] The aspect ratio threshold is set in advance. After obtaining the aspect ratio of each detection frame, the aspect ratio of each detection frame is compared with the preset value, and the detection frame with an aspect ratio greater than the preset value is determined as an invalid detection frame. By determining the detection frame with an aspect ratio greater than the preset value as an invalid detection frame, the detection frame is filtered and the tracking of invalid detection frames is avoided.

[0123] It should be noted that the preset value can be set and adjusted according to actual conditions, and the embodiment of the present application does not limit this. For example, it can be set to 1.5.

[0124] S204: Obtain a preset number of historical image frames corresponding to the current frame image.

[0125] S205: Determine the tracking frames corresponding to the detection frames except the invalid detection frames from the historical image frames through feature matching and detection frame distance matching.

[0126] After acquiring each historical image frame, the tracking frames corresponding to each detection frame except the invalid detection frame are determined from each historical image frame through feature matching and detection frame distance matching. For example, during feature matching, the feature extraction model can be used to extract the deep features of pedestrians, and feature matching can be performed based on the deep features. The motion of the target can be estimated based on the motion filter model to achieve detection frame distance matching. By only performing tracking frame matching on each detection frame except the invalid detection frame, the trajectory tracking of the target corresponding to the infinite detection frame is avoided, the task volume is reduced, and resource utilization is improved.

[0127] Feature extraction models can be used for feature matching. Feature extraction models are key tools used in machine learning and deep learning to extract useful information from raw data. The goal of feature extraction is to convert data into a form that is more suitable for algorithm processing to improve model building efficiency and prediction accuracy. The purpose of feature extraction is dimensionality reduction, which reduces the computational complexity by removing redundant or irrelevant features and reducing the dimensions of the data. Improve the performance of the model by extracting more discriminative features to improve the accuracy and robustness of the model. Convert raw data into a format that is more suitable for model learning to enhance data usability.

[0128] Feature extraction can be divided into manual feature extraction and automatic feature extraction. Manual feature extraction can include statistical features, such as mean, variance, skewness, kurtosis, etc., and can also include frequency domain features, which extract the frequency characteristics of the signal through Fourier transform, or morphological features, which are commonly used in image processing, such as edges, corners, etc. Automatic feature extraction includes deep learning models, such as convolutional neural networks (CNNs), which automatically extract image features through multiple levels of convolution operations. Embedding features in natural language processing (NLP), such as word embedding and sentence embedding, automatically learn feature representations from text. Feature extraction models are an important part of the machine learning process. Through effective feature extraction, the performance and efficiency of the model can be significantly improved. With the development of deep learning, automatic feature extraction methods have received more and more attention, and many complex tasks can achieve superior performance through learned feature representations.

[0129] The feature extraction model extracts the apparent feature information of the target from the detection frame, and can effectively identify the same target even under different viewing angles, lighting and background conditions. Feature extraction mostly adopts the ReID model, which mainly uses deep learning for feature extraction and similarity measurement, mainly including feature extraction, metric learning and similarity measurement steps. For targets with similar color features, the features are often very similar. Feature extraction can also combine information from different modalities such as RGB (Red Green Blue) images, depth information, infrared images, etc. to improve recognition accuracy.

[0130] Multi-target tracking is an important task in the field of computer vision, which aims to track multiple moving targets in real time in a video sequence. It is widely used in autonomous driving, video surveillance, motion analysis and human-computer interaction scenarios. The multi-target tracking task is to establish the association of the same target between multiple frames, that is, to determine the matching relationship between the target detected in the current frame and the target in the previous frame, including distance-based methods, such as the Hungarian algorithm, maximum flow, etc.; appearance feature-based methods, such as using deep learning to extract target appearance features for matching; using Kalman filtering or particle filtering to predict the position of the target. At present, multi-target tracking methods based on deep learning technology are gradually becoming mainstream. These methods improve the accuracy of detection and feature extraction through deep neural networks, thereby optimizing data association and tracking effects.

[0131] The embodiment of the present application can use the ResNet network as the feature extraction model to output a 128-dimensional feature vector. Compared with the usual 512-dimensional, the embodiment of the present application changes to 128-dimensional, which has a smaller dimension, is conducive to reducing the end-side storage and faster calculation. The training set can use Market1501, and the model has a map value of 0.942, a Rank-1 value of 0.923, and a Rank-5 value of 0.997 on the test set.

[0132] S206: Update the trajectories corresponding to the respective targets according to the respective detection frames and the respective tracking frames.

[0133] Motion filtering is an algorithm or model used to process motion in dynamic systems. It is widely used in computer vision, robot navigation, virtual reality and other fields. Its main purpose is to estimate the true motion state of an object from noisy observation data. The motion filtering model describes the dynamic system through state equations and observation equations. The state equation defines the evolution of the system state over time, while the observation equation describes how to obtain noisy observation data from the true state.

[0134] The pedestrian motion filter model is a mathematical model mainly used to estimate pedestrian motion. Its basic idea is to filter the observed pedestrian trajectory to eliminate the influence of system noise interference and accidental errors, so as to obtain more accurate pedestrian motion estimation results. The pedestrian motion filter model mainly includes state model, observation model, motion model and measurement model. The pedestrian motion filter model can use different filtering algorithms to estimate the motion state of pedestrians. Common filtering algorithms include Kalman filtering, particle filtering and interactive filtering. In practical applications, appropriate algorithms are selected according to different motion characteristics. By adopting the filtering model, the target tracking trajectory will be smoother and the occurrence of accidental errors will be reduced. The trajectory tracking results can be used to update the mean and variance in the filter model, and the detection box features at the current moment are used as the latest features of the tracker for subsequent tracking and matching.

[0135] The embodiment of the present application uses a Kalman filter to model pedestrian motion. The core of the Kalman filter is to fuse information and continuously update the model, where the state equation is:

[0136] ;

[0137] in, is the motion state of the filter system at the previous moment, is the state transition matrix; is the control input of the system (the input that the system can add, usually defaults to 0), all the external input quantities of the system, and B is the coefficient; Represents a random variable, represents the uncertainty of the state, and conforms to the normal distribution, and the system assumes that the target motion is linear.

[0138] The state equation is used to estimate the state of the system at a certain moment. The establishment of the motion model requires prior knowledge. However, the model is often only a mathematical approximation and is not completely consistent with the actual observed state value. The system observation equation is as follows:

[0139] ;

[0140] in, Indicates the observation state, is the transfer matrix, is noise and is assumed to obey Gaussian distribution.

[0141] S207: Obtain a preset quadrilateral passenger flow counting area.

[0142] See also Figure 3 , Figure 3This is a schematic diagram of a method of counting customers according to a quadrilateral customer flow counting area in an embodiment of the present application. A quadrilateral customer flow counting area is preset. In the ABCD area, line D is the store entry and exit line, A, B, and C are store passing lines, line D needs to cover all areas of the store entrance, and line B needs to be located outside the store.

[0143] S208: Perform passenger flow statistics based on the relationship between each trajectory and the quadrilateral passenger flow statistics area.

[0144] After obtaining the preset quadrilateral passenger flow statistics area, passenger flow statistics are performed according to the relationship between each trajectory and the quadrilateral passenger flow statistics area. For example, the running direction of the target is determined according to the trajectory, and passenger flow statistics are performed according to the running direction and the intersection of the trajectory with each side of the quadrilateral passenger flow statistics area, so as to record the number of people passing through the store, the number of people entering the store, and the number of people leaving the store. By using quadrilaterals to analyze and count, it is more accurate than the single-sided crossing line counting method. Real-time detection and tracking of multiple pedestrians is achieved, and the tracking results of pedestrians are output in real time while reading the video stream. The passenger flow counting method provided in the embodiment of the present application can be deployed on edge devices, which can count passenger flow in real time, help enterprises and institutions make fact-based decisions, and better understand customer needs, improve service quality, and gain an advantage in market competition.

[0145] The trajectories can also be filtered before passenger flow statistics are performed, and only the trajectories that use the current frame image as a reference frame and match the tracking frame for three consecutive frames are filtered out, and passenger flow statistics are performed on the filtered trajectories.

[0146] In a specific implementation of the present application, step S208 may include the following steps:

[0147] Step 1: When the trajectory passes through any two store-passing lines in the quadrilateral passenger flow statistical area, the target corresponding to the trajectory is determined to be a store-passing line;

[0148] Step 2: When the trajectory is generated from the quadrilateral passenger flow statistical area and passes through any store-passing line in the quadrilateral passenger flow statistical area, determine whether the target corresponding to the trajectory is exiting the store or passing through the store according to the trajectory;

[0149] Step 3: When the trajectory passes through the store entry and exit line in the quadrilateral customer flow statistical area, and the movement direction corresponding to the trajectory is from the inside of the quadrilateral customer flow statistical area to the outside of the quadrilateral customer flow statistical area, it is determined that the target corresponding to the trajectory is entering the store;

[0150] Step 4: When the trajectory passes through the store entry and exit line in the quadrilateral customer flow statistical area, and the movement direction corresponding to the trajectory is from the outside of the quadrilateral customer flow statistical area to the inside of the quadrilateral customer flow statistical area, it is determined that the target corresponding to the trajectory is to exit the store.

[0151] For the convenience of description, the above four steps can be combined for explanation.

[0152] When the trajectory passes through any two store-passing lines in the quadrilateral passenger flow statistical area, the target corresponding to the trajectory is determined to be a store-passing. When the trajectory is generated from the quadrilateral passenger flow statistical area and passes through any store-passing line in the quadrilateral passenger flow statistical area, the target corresponding to the trajectory is determined to be a store-exit or a store-passing according to the trajectory. For example, when the generation position of the trajectory is closer to the store-entry and store-exit lines, the target corresponding to the trajectory is determined to be a store-exit; when the generation position of the trajectory is farther away from the store-entry and store-exit lines, the target corresponding to the trajectory is determined to be a store-passing.

[0153] When the trajectory passes through the store entry and exit line in the quadrilateral passenger flow statistical area, and the movement direction corresponding to the trajectory is from the inside of the quadrilateral passenger flow statistical area to the outside of the quadrilateral passenger flow statistical area, it is determined that the target corresponding to the trajectory is entering the store. When the trajectory passes through the store entry and exit line in the quadrilateral passenger flow statistical area, and the movement direction corresponding to the trajectory is from the outside of the quadrilateral passenger flow statistical area to the inside of the quadrilateral passenger flow statistical area, it is determined that the target corresponding to the trajectory is leaving the store. By using quadrilaterals to analyze and count, the accuracy of passenger flow counting is greatly improved compared to the unilateral crossing line counting method.

[0154] Corresponding to the above method embodiment, the present application further provides a passenger flow counting device, and the passenger flow counting device described below and the passenger flow counting method described above can be referred to each other.

[0155] See also Figure 4 , Figure 4 This is a structural block diagram of a passenger flow counting device in an embodiment of the present application, and the device may include:

[0156] The detection frame obtaining module 41 is used to detect the current frame image using the target detection model to obtain the detection frames corresponding to the targets contained in the current frame image;

[0157] A historical image frame acquisition module 42 is used to acquire a preset number of historical image frames corresponding to the current frame image;

[0158] A tracking frame determination module 43, used to determine the tracking frames corresponding to the detection frames from each historical image frame by feature matching and detection frame distance matching;

[0159] A trajectory updating module 44 is used to update the trajectory corresponding to each target according to each detection frame and each tracking frame;

[0160] The passenger flow statistics module 45 is used to perform passenger flow statistics according to each trajectory.

[0161] It can be seen from the above technical solution that by obtaining the detection frames corresponding to each target contained in the current frame image, and determining the tracking frames corresponding to each detection frame from the historical image frame, the trajectories corresponding to each target in each detection frame and each tracking frame are updated to achieve passenger flow statistics according to the updated trajectories. Compared with manual counting, it greatly improves the counting efficiency, avoids the influence of human factors, does not require complex mechanical structures, greatly reduces the influence of factors such as light and occlusion, does not require face recognition, and effectively protects user privacy.

[0162] In a specific implementation of the present application, the detection frame obtaining module 41 is specifically a module that detects the current frame image using a head and shoulder detection model.

[0163] In a specific embodiment of the present application, the device may further include:

[0164] An aspect ratio acquisition module is used to obtain the aspect ratio of each detection frame after obtaining the detection frames corresponding to each target contained in the current frame image and before determining the tracking frames corresponding to each detection frame from each historical image frame through feature matching and detection frame distance matching;

[0165] An invalid detection frame determination module, used to determine a detection frame whose aspect ratio is greater than a preset value as an invalid detection frame;

[0166] The tracking frame determination module 43 is specifically a module for determining the tracking frames corresponding to the detection frames except the invalid detection frames from each historical image frame through feature matching and detection frame distance matching.

[0167] In a specific implementation of the present application, the tracking frame determination module 43 includes:

[0168] A historical detection frame set matching submodule is used to match the historical detection frame set corresponding to each detection frame from each historical image frame through a feature extraction model;

[0169] A cosine distance calculation submodule is used to calculate, for each detection frame, the cosine distance between the detection frame and each historical detection frame in the corresponding historical detection frame set;

[0170] The minimum cosine distance determination submodule is used to determine the minimum cosine distance corresponding to each detection frame;

[0171] The first tracking frame determination submodule is used to match the detection frame with each historical detection frame in the corresponding historical detection frame set using the Hungarian algorithm for the detection frame whose minimum cosine distance is greater than a preset distance value, and determine the successfully matched historical detection frame as the tracking frame of the corresponding detection frame;

[0172] The Euclidean distance calculation submodule is used to calculate the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set for the detection frame that failed to be matched using the Hungarian algorithm;

[0173] The second tracking frame determination submodule is used to determine the tracking frame corresponding to the detection frame according to each Euclidean distance.

[0174] In a specific implementation of the present application, the Euclidean distance calculation submodule includes:

[0175] A nearest neighbor historical detection frame acquisition unit, used to acquire a nearest neighbor historical detection frame in a nearest neighbor image frame of a current frame image from a set of historical detection frames corresponding to the detection frame;

[0176] A detection frame filtering unit, configured to filter out the detection frame when it is determined that a change in length and / or a change in width of the detection frame compared to the nearest neighbor historical detection frame exceeds a preset change threshold;

[0177] A track disappearance time determination unit, for determining the track disappearance time according to the detection frame and the corresponding nearest neighbor historical detection frame for the detection frame remaining after filtering;

[0178] A predicted distance obtaining unit, used to obtain the moving speed of the detection frame, and calculate the product between the moving speed and the track disappearance time to obtain the predicted distance;

[0179] An actual distance acquisition unit, used to acquire the actual distance between the detection frame and the corresponding nearest neighbor historical detection frame;

[0180] A target departure determination unit, used to determine that the target corresponding to the detection frame has left when it is determined that the actual distance is less than the predicted distance;

[0181] The Euclidean distance calculation unit is used to calculate the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set when it is determined that the actual distance is greater than or equal to the predicted distance.

[0182] In a specific implementation of the present application, the passenger flow statistics module 45 includes:

[0183] A quadrilateral passenger flow statistics area acquisition submodule is used to acquire a preset quadrilateral passenger flow statistics area;

[0184] The passenger flow statistics submodule is used to perform passenger flow statistics based on the relationship between each trajectory and the quadrilateral passenger flow statistics area.

[0185] In a specific implementation of the present application, the passenger flow statistics submodule includes:

[0186] A store passing determination unit, used for determining that the target corresponding to the trajectory is a store passing when the trajectory passes through any two store passing lines in the quadrilateral passenger flow statistical area;

[0187] A judgment unit, configured to judge whether the target corresponding to the trajectory is exiting the store or passing through the store according to the trajectory when the trajectory is generated from the quadrilateral passenger flow statistical area and passes through any store passing line in the quadrilateral passenger flow statistical area;

[0188] A store entry determination unit is used to determine that the target corresponding to the trajectory is entering the store when the trajectory passes through the store entry and exit line in the quadrilateral passenger flow statistical area and the movement direction corresponding to the trajectory is from the inside of the quadrilateral passenger flow statistical area to the outside of the quadrilateral passenger flow statistical area;

[0189] The store exit determination unit is used to determine that the target corresponding to the trajectory is to exit the store when the trajectory passes through the store entry and exit line in the quadrilateral customer flow statistical area and the movement direction corresponding to the trajectory is from the outside of the quadrilateral customer flow statistical area to the inside of the quadrilateral customer flow statistical area.

[0190] Corresponding to the above method embodiment, see Figure 5 , Figure 5 A schematic diagram of a passenger flow counting device provided in this application, which may include:

[0191] A memory 332, for storing computer programs;

[0192] The processor 322 is used to implement the steps of the passenger flow counting method of the above method embodiment when executing the computer program.

[0193] For details, please refer to Figure 6 , Figure 6 A specific structural diagram of a passenger flow counting device provided in this embodiment, which may have relatively large differences due to different configurations or performances, may include a processor (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the passenger flow counting device 301.

[0194] The passenger flow counting device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0195] The steps in the passenger flow counting method described above can be implemented by the structure of the passenger flow counting device.

[0196] Corresponding to the above method embodiment, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0197] The target detection model is used to detect the current frame image to obtain the detection frames corresponding to the targets contained in the current frame image; a preset number of historical image frames corresponding to the current frame image are obtained; the tracking frames corresponding to the detection frames are determined from the historical image frames through feature matching and detection frame distance matching; the trajectories corresponding to the targets are updated according to the detection frames and the tracking frames; and passenger flow statistics are performed according to the trajectories.

[0198] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0199] For an introduction to the computer-readable storage medium provided in this application, please refer to the above method embodiment, and this application will not go into details here.

[0200] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the devices, equipment and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0201] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the technical solution and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A passenger flow counting method, characterized in that: include: Detecting the current frame image using the target detection model to obtain detection frames corresponding to the targets contained in the current frame image; Acquire a preset number of historical image frames corresponding to the current frame image; Determine the tracking frames corresponding to the respective detection frames from each historical image frame through feature matching and detection frame distance matching; Update the trajectories corresponding to each target according to each detection frame and each tracking frame; Passenger flow statistics are performed based on each trajectory.

2. The passenger flow counting method according to claim 1, characterized in that: Use the target detection model to detect the current frame image, including: The head and shoulder detection model is used to detect the current frame image.

3. The passenger flow counting method according to claim 1, characterized in that: After obtaining the detection frames corresponding to the respective targets contained in the current frame image, and before determining the tracking frames corresponding to the respective detection frames from the historical image frames by feature matching and detection frame distance matching, the method further includes: Get the aspect ratio of each detection frame; Determine the detection frame whose aspect ratio is greater than a preset value as an invalid detection frame; Accordingly, the tracking frames corresponding to the respective detection frames are determined from the historical image frames through feature matching and detection frame distance matching, including: The tracking frames corresponding to the detection frames except the invalid detection frames are determined from each historical image frame through feature matching and detection frame distance matching.

4. The passenger flow counting method according to claim 1, characterized in that: The tracking frames corresponding to each detection frame are determined from each historical image frame through feature matching and detection frame distance matching, including: Match the historical detection frame sets corresponding to each detection frame from each historical image frame through the feature extraction model; For each detection frame, calculate the cosine distance between the detection frame and each historical detection frame in the corresponding historical detection frame set; Determine the minimum cosine distance corresponding to each detection frame; Determine the historical detection frame corresponding to the minimum cosine distance less than or equal to the preset distance value as the tracking frame of the corresponding detection frame; For a detection frame whose minimum cosine distance is greater than the preset distance value, the Hungarian algorithm is used to match the detection frame with each historical detection frame in the corresponding historical detection frame set, and the successfully matched historical detection frame is determined as the tracking frame of the corresponding detection frame; For the detection frame that fails to be matched using the Hungarian algorithm, calculate the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set; The tracking frame corresponding to the detection frame is determined according to each Euclidean distance.

5. The passenger flow counting method according to claim 4, characterized in that: Calculating the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set includes: Acquire a nearest neighbor historical detection frame in a nearest neighbor image frame of the current frame image from a set of historical detection frames corresponding to the detection frame; When it is determined that the length change and / or width change of the detection frame compared to the nearest neighbor historical detection frame exceeds a preset change threshold, filtering out the detection frame; For the detection frames remaining after filtering, determining the track disappearance time according to the detection frames and the corresponding nearest neighbor historical detection frames; Acquire the moving speed of the detection frame, and calculate the product of the moving speed and the track disappearance time to obtain the predicted distance; Obtaining the actual distance between the detection frame and the corresponding nearest neighbor historical detection frame; When it is determined that the actual distance is less than the predicted distance, determining that the target corresponding to the detection frame leaves; When it is determined that the actual distance is greater than or equal to the predicted distance, the Euclidean distance between the detection frame and each historical detection frame in the corresponding historical detection frame set is calculated.

6. The passenger flow counting method according to claim 1, characterized in that: Passenger flow statistics are performed based on each track, including: Get the preset quadrilateral passenger flow statistics area; Passenger flow statistics are performed according to the relationship between each trajectory and the quadrilateral passenger flow statistics area.

7. The passenger flow counting method according to claim 6, characterized in that: Passenger flow statistics are performed according to the relationship between each trajectory and the quadrilateral passenger flow statistics area, including: When the trajectory passes through any two store-passing lines in the quadrilateral passenger flow statistical area, determining that the target corresponding to the trajectory is a store-passing line; When the trajectory is generated from the quadrilateral passenger flow statistical area and passes through any store-passing line in the quadrilateral passenger flow statistical area, judging whether the target corresponding to the trajectory is leaving the store or passing through the store according to the trajectory; When the trajectory passes through the store entry and exit line in the quadrilateral customer flow statistical area, and the movement direction corresponding to the trajectory is from the inside of the quadrilateral customer flow statistical area to the outside of the quadrilateral customer flow statistical area, it is determined that the target corresponding to the trajectory is entering the store; When the trajectory passes through the store entry and exit line in the quadrilateral customer flow statistical area, and the movement direction corresponding to the trajectory is from the outside of the quadrilateral customer flow statistical area to the inside of the quadrilateral customer flow statistical area, it is determined that the target corresponding to the trajectory is exiting the store.

8. A passenger flow counting device, characterized in that: include: A detection frame acquisition module is used to detect the current frame image using the target detection model to obtain detection frames corresponding to the targets contained in the current frame image; A historical image frame acquisition module, used to acquire a preset number of historical image frames corresponding to the current frame image; A tracking frame determination module is used to determine the tracking frames corresponding to each detection frame from each historical image frame through feature matching and detection frame distance matching; A trajectory update module is used to update the trajectory corresponding to each target according to each detection frame and each tracking frame; The passenger flow statistics module is used to perform passenger flow statistics based on each trajectory.

9. A passenger flow counting device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the passenger flow counting method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the passenger flow counting method according to any one of claims 1 to 7 are implemented.

Images