Method and device for detecting human flow

Abstract

The application provides a kind of people flow detection method and equipment, including obtaining the real-time video data of people flow detection area;Using trained detection model, real-time video data is processed, and the pedestrian detection frame of current frame in real-time video data is obtained;Pedestrian detection frame data of current frame is input into target tracker, and the pedestrian prediction frame of future frame is obtained, and pedestrian detection frame and pedestrian prediction frame are matched, and each pedestrian detection frame is assigned identification number according to matching result;According to the detection line of people flow detection direction, the people flow statistical result is obtained according to the identification number of pedestrian detection frame passing through detection line.Through the above setting, high-precision, high real-time people flow detection can be realized with lower power consumption.

Description

Technical Field

[0001] This application relates to the field of image recognition, and more particularly to a method and device for detecting pedestrian flow. Background Technology

[0002] With societal development and a growing population, the demand for real-time pedestrian flow detection systems is increasing in sectors such as commerce, traffic control, and public safety. Analyzing pedestrian flow data can not only protect people's lives but also support business marketing and other activities.

[0003] Currently, commonly used methods for pedestrian flow detection and statistics mainly include: physical contact statistics, infrared statistics, and visual statistics. Physical contact statistics rely on devices such as turnstiles and pressure-sensitive pedals, and are mostly used in narrow entrances and exits where pedestrians pass sequentially. Although the detection accuracy is high, the passage speed is slow and the application scenarios are limited. Infrared statistics have the advantages of low equipment installation requirements and relatively convenient use, but their detection error is relatively large, and the detection results are easily affected by external factors such as heat sources, making them difficult to apply to open areas. Visual statistics rely on surveillance cameras, and have the advantages of convenient installation, inexpensive equipment, and applicability to large open areas. However, factors such as pedestrians occluding each other, other objects obscuring pedestrians, and background changes reduce the accuracy of pedestrian flow detection. Summary of the Invention

[0004] This application provides a method and device for detecting pedestrian flow, which aims to deploy an improved pedestrian flow detection model on a low-power edge computing platform to achieve high-precision and high-real-time pedestrian flow detection.

[0005] One embodiment of this application provides a method for detecting pedestrian flow, the method being deployed on an FPGA edge computing platform, the method comprising:

[0006] Acquire real-time video data of the pedestrian flow detection area;

[0007] The trained detection model is used to process real-time video data to obtain pedestrian detection boxes in the current frame of the real-time video data.

[0008] Input the pedestrian detection box data of the current frame into the target tracker to obtain the pedestrian prediction box of the future frame, match the pedestrian detection box and the pedestrian prediction box, and assign an identifier to each pedestrian detection box according to the matching result;

[0009] Detection lines are drawn according to the direction of pedestrian flow detection, and pedestrian flow statistics are obtained based on the identification numbers of pedestrian detection boxes that pass through the detection lines.

[0010] In one embodiment, a trained detection model is used to process real-time video data to obtain pedestrian detection boxes for the current frame in the real-time video data, specifically including:

[0011] The feature extraction network trained in the detection model is used to process the current frame data in real-time video data. The activation function in the detection model is determined based on the computing resources of the FPGA edge computing platform. The feature extraction network of the detection model also includes a channel attention module, which is used to analyze the importance of each channel to the prediction results of the detection model.

[0012] The feature fusion network trained in the detection model is used to process the results of the feature extraction network; the detection head structure trained in the detection model is used to process the results of the feature fusion network to obtain pedestrian detection boxes; duplicate detection box results are filtered out by non-maximum suppression before the pedestrian detection boxes of the current frame are output.

[0013] In one embodiment, a pedestrian detection dataset is constructed based on the mutual occlusion of pedestrians, specifically including:

[0014] Select the first pedestrian detection dataset, and add some data from the second pedestrian detection dataset to the first pedestrian detection dataset to construct the third pedestrian detection dataset;

[0015] Transform the bounding box positions in the label file of the third pedestrian detection dataset, crop the bounding box positions, and construct the pedestrian detection dataset;

[0016] The detection model was trained using a pedestrian detection dataset.

[0017] In one embodiment, the detection model is trained using a pedestrian detection dataset, specifically including:

[0018] The pedestrian detection dataset is processed using data augmentation techniques to obtain the augmented pedestrian detection dataset.

[0019] The detection model was trained using the pedestrian detection dataset after data augmentation.

[0020] In one embodiment, before processing real-time video data using a detection model to obtain the detection box of a pedestrian in the current frame of the real-time video data, the method further includes:

[0021] The detection model is compressed.

[0022] When deploying the detection model on an FPGA platform, the network structure of the detection model is designed for parallel and pipelined design oriented towards the output channel.

[0023] In one embodiment, the pedestrian detection bounding box data of the current frame is input into the target tracker to obtain the pedestrian prediction bounding box of the future frame. The pedestrian detection bounding box and the pedestrian prediction bounding box are then matched, and an identifier is assigned to each pedestrian detection bounding box according to the matching result. Specifically, this includes:

[0024] The pedestrian detection bounding box data of the current frame is input into the target tracker, and the pedestrian prediction bounding box of the next frame is obtained by the Kalman filter in the target tracker.

[0025] The cross-union ratio (CUN ratio) is calculated between the pedestrian detection bounding box of the current frame and the pedestrian prediction bounding box obtained by predicting the pedestrian detection bounding box of the previous frame through the Kalman filter. A cost matrix is ​​constructed based on the CUN ratio to map the pedestrians detected in the two frames.

[0026] The cost matrix is ​​used as input to the Hungarian algorithm in the target tracker to obtain linear matching results. When the pedestrian detection box of the current frame matches the pedestrian prediction box obtained by predicting the pedestrian detection box of the previous frame through the Kalman filter, the Kalman filter is updated and an identifier is assigned to each pedestrian detection box.

[0027] In one embodiment, a detection line is drawn according to the direction of pedestrian flow detection, and the pedestrian flow statistics are obtained based on the identification number of the pedestrian detection box passing through the detection line, specifically including:

[0028] The direction of pedestrian flow detection is bidirectional. Two non-coincident straight lines are drawn perpendicular to the direction of pedestrian flow detection. These non-coincident straight lines are the detection lines.

[0029] Based on the order in which pedestrian detection box markers pass through two non-coincident straight lines, the bidirectional pedestrian flow in the real-time video is calculated.

[0030] Another embodiment of this application provides an electronic device, including: a memory for storing a program; and a processor for executing the program stored in the memory. When the program is executed, the processor is used to execute the crowd flow detection method provided in the above embodiment.

[0031] Another embodiment of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the crowd flow detection method involved in the above embodiments.

[0032] Another embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the pedestrian flow detection method provided in the above embodiments.

[0033] This application provides a pedestrian flow detection method and device. It utilizes the lightweight YOLOX-tiny model and improves it to form the YOLOX-tiny-F pedestrian detection model. Based on the characteristics of FPGA hardware, YOLOX-tiny-F selects an easily computed activation function. Considering the application scenario of long-distance overhead camera shooting and the detection target being only the combination of pedestrian heads and shoulders, two prediction heads are reduced, and the complexity of the feature fusion network is decreased, thereby reducing the number of model parameters and computational load, and improving the model's detection speed. Addressing the issue of densely clustered pedestrians in some areas during pedestrian flow detection, a spatial attention mechanism is introduced into the prediction head. By analyzing the degree of pedestrian clustering in the image, different areas with varying pedestrian density are detected separately, enhancing the model's detection accuracy and reducing the false negative rate. The YOLOX-tiny-F detection model, together with a target tracker, forms a pedestrian flow detection model. When deployed on an FPGA, the detection model undergoes fixed-point quantitative compression, and the network structure is designed with parallel and pipelined approaches for the output channel, further improving the real-time performance of the pedestrian flow detection model. After the surveillance camera acquires real-time video data, it is input into the pedestrian flow detection model to detect pedestrians in the real-time video and assign identification numbers to the pedestrian detection boxes. Detection lines are drawn according to the direction of pedestrian flow detection, and pedestrian flow statistics are obtained based on the identification numbers of the pedestrian detection boxes that pass through the detection lines. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] Figure 1 A flowchart illustrating a pedestrian flow detection method provided in an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of the YOLOX-tiny-F network model structure provided in an embodiment of this application;

[0037] Figure 3 This is a flowchart illustrating the training of a detection model for a pedestrian flow detection method provided in an embodiment of this application.

[0038] Figure 4 This is a flowchart illustrating the process of assigning identifiers to pedestrian detection frames in a pedestrian flow detection method provided in one embodiment of this application;

[0039] Figure 5 This is a schematic diagram illustrating the method for detecting pedestrian flow by defining detection lines and counting pedestrian flow in an embodiment of this application.

[0040] Figure 6 This is a schematic diagram of the structure of a people flow detection device provided in an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0042] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0044] In open commercial districts, popular tourist attractions, and other areas with high pedestrian traffic, the importance of real-time pedestrian flow detection systems is becoming increasingly apparent in ensuring people's safety and experience. Compared to physical contact statistics and infrared statistics, visual statistics, relying on surveillance cameras, are better suited for large, open scenes.

[0045] When pedestrian traffic is high, factors such as pedestrians occluding each other, other objects occluding pedestrians, and background changes can affect the accuracy of visual statistical methods for detecting pedestrian traffic. Furthermore, visual statistical methods for detecting pedestrian traffic require real-time analysis of image data captured by cameras, necessitating appropriate hardware to handle computationally intensive tasks.

[0046] To address the aforementioned technical problems, this application provides a pedestrian flow detection method and device, aiming to deploy an improved pedestrian flow detection model on a low-power edge computing platform to achieve high-precision and high-real-time pedestrian flow detection. The technical concept of this application is as follows: A Field Programmable Gate Array (FPGA) with both data parallelism and pipeline parallelism is selected as the edge computing platform; the YOLO (You Only Look Once) object detection framework based on deep learning is used as the object detector; the YOLO object detection network structure is modified to suit the characteristics of FPGA hardware; the object detector and object tracker are combined to form a pedestrian flow detection model; the pedestrian flow detection model is deployed on an FPGA chip; and the network structure is designed with parallel and pipelined features oriented towards the output channel; a camera inputs real-time video data into the pedestrian flow detection model to obtain the identifier of each pedestrian detection box; and based on the entry and exit of pedestrian detection box identifiers at the boundaries of the pedestrian flow detection area, pedestrian flow statistics are obtained.

[0047] like Figure 1 As shown, one embodiment of this application provides a method for detecting pedestrian flow. This method is deployed on an FPGA edge computing platform and specifically includes:

[0048] S101. Obtain real-time video data of the pedestrian flow detection area.

[0049] In this step, real-time video data is read from surveillance cameras deployed in the pedestrian flow detection area and then input into the detection model.

[0050] S102. Use the trained detection model to process the real-time video data and obtain the pedestrian detection box of the current frame in the real-time video data.

[0051] The detection model can identify pedestrians in the current frame and their positions in the image. By using rectangular boxes as detection boxes to mark pedestrians and their positions, the detection boxes for pedestrians in the current frame of real-time video data can be obtained.

[0052] S103. Input the pedestrian detection box data of the current frame into the target tracker to obtain the pedestrian prediction box of the future frame, and match the pedestrian detection box and the pedestrian prediction box. Assign an identifier to each pedestrian detection box according to the matching result.

[0053] The current frame may contain multiple pedestrians. Therefore, the pedestrian detection bounding boxes of the current frame are input into the target tracker to obtain multiple pedestrian prediction bounding boxes for the next frame. The pedestrian prediction bounding boxes for the next frame determine the pedestrian's trajectory and predict the pedestrian's position in the next frame. Only when the pedestrian detection bounding box and the corresponding pedestrian prediction bounding box match can it be confirmed that the pedestrian in different frames is the same target. At this point, an identifier is assigned to each pedestrian detection bounding box based on the matching result.

[0054] S104. Delineate the detection line according to the direction of pedestrian flow detection, and obtain the pedestrian flow statistics result based on the identification number of the pedestrian detection box passing through the detection line.

[0055] The detection line delineates the pedestrian flow statistics area. Each pedestrian detection box has a unique identifier. When a pedestrian detection box passes the detection line, the pedestrian's movement direction can be recorded. Based on the identifier of the pedestrian detection box that passes the detection line, the pedestrian flow within the detection area can be counted.

[0056] In the above technical solution, the surveillance camera acquires real-time video data within the pedestrian flow detection area, identifies pedestrians and their positions in the current frame image through a trained detection model, and the target tracker obtains the predicted bounding boxes of pedestrians in future frames based on the data of the pedestrian detection boxes. When the pedestrian detection box and the corresponding predicted pedestrian box match, the target tracker tracks the same target in multiple frames, assigns an identifier to each pedestrian detection box, and obtains the pedestrian flow statistics based on the order and number of the pedestrian detection box identifiers that pass through the detection line.

[0057] One embodiment of this application provides a pedestrian flow detection method, which uses a trained detection model to process real-time video data to obtain pedestrian detection boxes in the current frame of the real-time video data, specifically including:

[0058] The feature extraction network trained in the detection model is used to process the current frame data in real-time video data. The activation function in the detection model is determined based on the computing resources of the FPGA edge computing platform. The feature extraction network of the detection model also includes a channel attention module, which is used to analyze the importance of each channel to the prediction results of the detection model.

[0059] Among them, the lightweight YOLOX-tiny is improved to form YOLOX-tiny-F. YOLOX-tiny-F serves as a pedestrian detection model. As a deep learning-based object detection algorithm, the YOLO object detection model boasts extremely high detection accuracy. YOLOX is the latest version of the YOLO object detection model, and the lightweight YOLOX-tiny is more suitable for deployment on FPGA platforms. Given the limited computing resources on FPGAs, the activation function in the YOLOX-tiny-F network structure needs to be selected appropriately based on the platform's characteristics. Introducing channel attention into the feature extraction network of YOLOX-tiny-F and analyzing the importance of each channel to the prediction results of the detection model can yield a set of channel weights, thereby improving the model's generalization performance and accuracy.

[0060] The feature fusion network trained in the detection model is used to process the results of the feature extraction network; the detection head structure trained in the detection model is used to process the results of the feature fusion network to obtain pedestrian detection boxes; duplicate detection box results are filtered out by non-maximum suppression before the pedestrian detection boxes of the current frame are output.

[0061] Feature fusion networks can reuse and optimize the features extracted by feature extraction networks, further enhancing the feature extraction capabilities of the detection model. The trained detection head structure in the detection model processes the results of the feature fusion network to obtain pedestrian detection boxes. When pedestrians occupy a large proportion of the image, a single pedestrian may be identified by multiple detection boxes. Therefore, non-maximum suppression is needed to filter out duplicate detection boxes before the pedestrian detection box for the current frame can be output.

[0062] For example: Figure 2 As shown, the YOLOX-tiny-F network structure consists of three parts: a feature extraction network, a feature fusion network, and a detection head. The backbone feature extraction network of the YOLOX-tiny-F object detection model includes a Focus module, a CBL module, a CSP module, and a channel attention module.

[0063] The Focus module can split a single input feature map into four feature maps at equal intervals, reducing the feature map size and increasing the number of channels. The CBL module consists of a 2D convolutional layer, a BN layer, and a Leaky ReLU activation function. The kernel sizes in the 2D convolutional layer are set to 3×3 and 1×1. Through the model control state machine, the 3×3 convolutional module deployed on the FPGA can be utilized more effectively, improving the computation speed of the 1×1 convolution. Using Leaky ReLU as the activation function reduces the resource consumption on the FPGA compared to SiLU and sigmoid activation functions. The output of each CSP module is connected to the channel attention module, which inputs feature map information of different sizes. The channel attention module analyzes the importance of each channel to the YOLOX-tiny-F prediction result and obtains the weights of each channel. The attention mechanism can improve the generalization performance and accuracy of the detection model.

[0064] The feature fusion network adopts a structure that combines FPN and PAN. The FPN layer conveys strong semantic features from top to bottom, while the PAN tower conveys localization features from bottom to top, thus fully integrating feature map information of different sizes.

[0065] The YOLOX-tiny-F detection head includes a classification prediction structure, a detection box coordinate prediction structure, and a detection box background judgment structure. Since this application only needs to detect pedestrians and the number of target categories is relatively small, the classification prediction structure can be directly incorporated into the detection box background judgment structure. Optimizing the structure of the detection head can not only reduce the computational cost of the detection model but also reduce the convergence difficulty during model training. The detection head processes the results of the feature fusion network to obtain pedestrian detection boxes. A single pedestrian may be identified by multiple small detection boxes. Non-maximum suppression (NMS) based on intersection-over-union (IOU) is used to filter out duplicate detection boxes, obtaining the detection box with the largest IOU value in the local region. This detection box is the pedestrian detection box for the current frame.

[0066] In the above technical solution, the lightweight YOLOX-tiny model is improved to construct the YOLOX-tiny-F detection model. The easily computed Leaky ReLU function is selected as the activation function in the YOLOX-tiny-F structure. A channel attention module is introduced into the backbone feature extraction network to improve the generalization performance and accuracy of the detection model. The detection head structure is optimized according to the characteristic of single detection object. By inputting the current frame image data into the trained YOLOX-tiny-F model, the detection box of the pedestrian in the current frame can be obtained.

[0067] like Figure 3 As shown, one embodiment of this application provides a pedestrian flow detection method. Based on the situation of pedestrians occluding each other, a pedestrian detection dataset is constructed, and the detection model is trained using the pedestrian detection dataset. Specifically, the method includes the following steps:

[0068] S201. Select the first pedestrian detection dataset, add some data from the second pedestrian detection dataset to the first pedestrian detection dataset, and construct the third pedestrian detection dataset.

[0069] For example, MOT17Det is chosen as the first pedestrian detection dataset, and MOT20Det as the second. Adding a sparser portion of the MOT20Det dataset to MOT17Det results in the third pedestrian detection dataset. In the third pedestrian detection dataset, the training set contains 6673 images with 426,500 labeled targets, averaging 63.91 targets per training image. The test set contains 743 images with 48,032 labeled targets, averaging 64.65 targets per test image.

[0070] S202. Convert the bounding box positions in the label file of the third pedestrian detection dataset, crop the bounding box positions, and construct the pedestrian detection dataset.

[0071] In this step, to reduce the risk of pedestrians occluding each other when there are many people, the bounding boxes in the dataset label file can be transformed. The original target being tested is cropped from the full body of the pedestrian to a combination of the pedestrian's head and shoulders, thus constructing a pedestrian detection dataset.

[0072] S203. Use data augmentation techniques to process the pedestrian detection dataset to obtain the augmented pedestrian detection dataset.

[0073] In this step, data augmentation techniques can increase the amount of data in the training set of the pedestrian detection dataset, making the training set as diverse as possible and enabling the trained detection model to have stronger generalization ability.

[0074] S204. Train the detection model using the pedestrian detection dataset after data augmentation.

[0075] For example, in an Ubuntu 16.04 system environment, using PyTorch 1.8 as the deep learning framework, CUDA 10.2 as the GPU computing platform, OpenCV 4.5 as the computer vision processing library, cuDNN as the deep neural network acceleration library, and NumPy 1.16 as the numerical computation extension library, the YOLOX-tiny-F model was trained for 200 rounds using an augmented pedestrian detection dataset. Based on the idea of ​​transfer learning, the model structure was frozen for the first 50 rounds to stabilize model parameters, avoid overfitting of feature weights, accelerate model training, and facilitate model convergence. After each round of training, only the model weight files with high prediction accuracy were saved. The first 50 rounds mainly trained the channel attention module and the detection head module, while in the last 150 rounds, the network was gradually unfrozen and trained from back to front, ultimately completing the training of the YOLOX-tiny-F detection model.

[0076] In the above technical solution, the pedestrian detection dataset is improved based on the situation of pedestrians occluding each other, and the detection model is trained using the improved dataset according to the idea of ​​transfer learning to obtain a high-precision detection model.

[0077] One embodiment of this application provides a pedestrian flow detection method. Before processing real-time video data using a detection model to obtain the detection box of pedestrians in the current frame of the real-time video data, the method further includes:

[0078] The detection model is compressed.

[0079] The size of the detection model directly impacts its accuracy. Deeper models generally offer superior performance, but also place higher demands on data storage and computational resources. Deploying the YOLOX-tiny-F detection model on an FPGA places even greater demands on computational resources, necessitating compression optimization. Weight quantization allows for more efficient application of the detection model on the FPGA with minimal impact on the model's performance; therefore, 8-bit fixed-point quantization can be applied to the network weights in YOLOX-tiny-F.

[0080] When deploying the detection model on an FPGA platform, the network structure of the detection model is designed for parallel and pipelined design oriented towards the output channel.

[0081] For example, a 3×3 feature map data sliding window module is designed to convert a single-line image data stream into three lines of image data conforming to the feature map structure. A single clock cycle completes 9×n multiplication calculations, where n is the output parallelism. By using a model-controlled state machine and a 3×3 convolution module deployed on an FPGA, parallel acceleration of 1×1 convolutions is achieved. During the cyclic operations of the convolutional layer, pipeline optimization strategies can be employed in the underlying parallel computing units to perform the next task in advance, even before the current task is completed.

[0082] In the above technical solution, the YOLOX-tiny-F detection model is compressed by fixed-point quantization so that the detection model can be better applied on FPGA. At the same time, based on the characteristics of FPGA's data parallelism and pipeline parallelism, the network structure of the detection model is designed with output channel-oriented parallel and pipeline design to improve the real-time performance of the detection model.

[0083] like Figure 4 As shown, one embodiment of this application provides a pedestrian flow detection method, which inputs the pedestrian detection box data of the current frame into a target tracker to obtain the pedestrian prediction box of the future frame, matches the pedestrian detection box and the pedestrian prediction box, and assigns an identifier to each pedestrian detection box according to the matching result. The specific method includes the following steps:

[0084] S301. Input the pedestrian detection box data of the current frame into the target tracker, and use the Kalman filter in the target tracker to predict the pedestrian prediction box of the next frame.

[0085] The target tracker employs the Simple Online and Realtime Tracking (SORT) algorithm, which consists of a Kalman filter and a Hungarian algorithm. The Kalman filter can predict the target's trajectory and correct the prediction using tracking results with high confidence. Therefore, after inputting the pedestrian detection box data of the current frame into the SORT target tracker, the pedestrian prediction box of the next frame can be obtained through the Kalman filter.

[0086] S302. Calculate the intersection-union ratio (IUU) of the pedestrian detection box in the current frame and the pedestrian prediction box obtained by predicting the pedestrian detection box in the previous frame through the Kalman filter, and construct the cost matrix based on the IUU.

[0087] Perform IOU matching between the pedestrian detection bounding box in the current frame and the pedestrian prediction bounding box in the previous frame, and calculate the cost matrix based on the IOU matching result.

[0088] S303. The cost matrix is ​​used as the input of the Hungarian algorithm in the target tracker to obtain the linear matching result. When the pedestrian detection box of the current frame matches the pedestrian prediction box obtained by predicting the pedestrian detection box of the previous frame through the Kalman filter, the Kalman filter is updated and an identifier is assigned to each pedestrian detection box.

[0089] In this step, the linear matching result can reflect the pairing of the pedestrian detection box in the current frame with the pedestrian prediction box in the previous frame. When the match is successful, it indicates that the tracking of the previous frame and the current frame is successful. The state space equation in the Kalman filter is updated, and an identifier number is assigned to each corresponding pedestrian detection box in the current frame.

[0090] In the above technical solution, the SORT target tracker acquires pedestrian detection box data, the Kalman filter acquires the corresponding pedestrian prediction box based on the data, and the Hungarian algorithm assigns multiple pedestrian detection boxes and pedestrian prediction boxes, so that the pedestrian prediction box finds the pedestrian detection box with the highest matching degree, thereby achieving the tracking effect. Each tracked pedestrian detection box is assigned an identifier number.

[0091] One embodiment of this application provides a method for detecting pedestrian flow, which involves defining detection lines based on the direction of pedestrian flow detection, and obtaining pedestrian flow statistics based on the identification numbers of pedestrian detection frames passing through the detection lines. Specifically, the method includes:

[0092] The pedestrian flow detection direction is bidirectional. Two non-coincident straight lines are drawn perpendicular to the pedestrian flow detection direction, and these non-coincident straight lines are the detection lines.

[0093] In the pedestrian flow detection area, the flow is divided into two detection directions according to the main direction of pedestrian movement, and detection lines perpendicular to the detection directions are drawn to form the pedestrian flow statistics area.

[0094] Based on the order in which pedestrian detection box markers pass through two non-coincident straight lines, the bidirectional pedestrian flow in the real-time video is calculated.

[0095] The order in which pedestrian detection box markers pass through two detection lines determines the direction of pedestrian movement, and the pedestrian flow is accumulated in that direction. Based on the pedestrian detection box markers passing through the pedestrian flow statistics area, the bidirectional pedestrian flow in the real-time video can be counted.

[0096] For example: Figure 5 As shown in the figure, pedestrians move in the upper right and lower left directions. Two detection lines are drawn perpendicular to these two directions. The real-time bidirectional pedestrian flow statistics are obtained according to the order in which the pedestrian detection box identifiers pass through these two detection lines.

[0097] In the above scheme, two detection lines are drawn in the pedestrian flow detection area according to the direction of pedestrian flow detection. The pedestrian flow statistics can be obtained by counting the number of pedestrian detection box signs passing through these two detection lines.

[0098] like Figure 6 As shown, another embodiment of this application provides a pedestrian flow detection device 100, comprising:

[0099] The acquisition module 101 is used to acquire real-time video data of the pedestrian flow detection area;

[0100] The detection module 102 is used to process real-time video data using a trained detection model to obtain pedestrian detection boxes in the current frame of the real-time video data.

[0101] The tracking module 103 is used to process the pedestrian detection box data of the current frame, obtain the pedestrian prediction box of the future frame, match the pedestrian detection box and the pedestrian prediction box, and assign an identifier to each pedestrian detection box according to the matching result.

[0102] The statistics module 104 is used to delineate detection lines according to the direction of pedestrian flow detection and obtain pedestrian flow statistics results based on the identification number of the pedestrian detection box that passes through the detection line.

[0103] In one embodiment, the detection module 102 is specifically used for:

[0104] The current frame data in real-time video data is processed using a feature extraction network trained in the detection model.

[0105] The results of the feature extraction network are processed using the feature fusion network trained in the detection model;

[0106] The pedestrian detection boxes are obtained by processing the results of the feature fusion network using the pre-trained detection head structure in the detection model. Duplicate detection boxes are filtered out by non-maximum suppression before the pedestrian detection boxes of the current frame are output.

[0107] In one embodiment, the tracking module 103 is specifically used for:

[0108] The pedestrian detection bounding box data of the current frame is input into the target tracker, and the pedestrian prediction bounding box of the next frame is obtained by the Kalman filter in the target tracker.

[0109] The intersection-union ratio (IUU) of the pedestrian detection bounding box in the current frame and the pedestrian prediction bounding box obtained by predicting the pedestrian detection bounding box in the previous frame through the Kalman filter is calculated, and the cost matrix is ​​constructed based on the IUU.

[0110] The cost matrix is ​​used as input to the Hungarian algorithm in the target tracker to obtain linear matching results. When the pedestrian detection box of the current frame matches the pedestrian prediction box obtained by predicting the pedestrian detection box of the previous frame through the Kalman filter, the Kalman filter is updated and an identifier is assigned to each pedestrian detection box.

[0111] In one embodiment, the statistics module 104 is specifically used for:

[0112] Two non-coincident straight lines are drawn perpendicular to the direction of pedestrian flow detection; these non-coincident straight lines are the detection lines.

[0113] Based on the order in which pedestrian detection box markers pass through two non-coincident straight lines, the bidirectional pedestrian flow in the real-time video is calculated.

[0114] like Figure 7 As shown, one embodiment of this application provides an electronic device 200, which includes a memory 201 and a processor 202.

[0115] The memory 201 is used to store computer instructions that can be executed by the processor;

[0116] The processor 202 implements the various steps of the method in the above embodiments when executing computer instructions. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0117] Optionally, the memory 201 can be either independent or integrated with the processor 202. When the memory 201 is configured independently, the electronic device also includes a bus for connecting the memory 201 and the processor 202.

[0118] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the methods described above.

[0119] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the various steps in the methods described above.

[0120] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0121] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for detecting pedestrian flow, characterized in that, The method is deployed on an FPGA edge computing platform, and the method includes: Acquire real-time video data of the pedestrian flow detection area; The real-time video data is processed using a trained detection model to obtain pedestrian detection boxes in the current frame of the real-time video data. The trained detection model is compressed using a fixed-point quantization method. The trained detection model includes a detection box background judgment structure and a classification prediction structure incorporated into the detection box background judgment structure. The pedestrian detection box data of the current frame is input into the target tracker to obtain the pedestrian prediction box of the future frame. The target tracker adopts a simple online real-time tracking algorithm, including the Hungarian algorithm. The cost matrix input to the Hungarian algorithm is calculated from the result of IOU matching. The pedestrian detection box and the pedestrian prediction box are matched, and an identifier is assigned to each pedestrian detection box according to the matching result. The detection line is drawn according to the direction of pedestrian flow detection, and the pedestrian flow statistics are obtained according to the identification number of the pedestrian detection box passing through the detection line. The process includes: the direction of pedestrian flow detection is bidirectional, and two non-overlapping straight lines are drawn in a direction perpendicular to the direction of pedestrian flow detection. The non-overlapping straight lines are the detection lines. The bidirectional pedestrian flow in the real-time video is counted according to the order in which the pedestrian detection box identification number passes through the two non-overlapping straight lines.

2. The method according to claim 1, characterized in that, The real-time video data is processed using a trained detection model to obtain pedestrian detection boxes for the current frame in the real-time video data, specifically including: The current frame data in the real-time video data is processed using the feature extraction network trained in the detection model; wherein, the activation function in the detection model is determined based on the computing resources of the FPGA edge computing platform, and the feature extraction network of the detection model also includes a channel attention module, which is used to analyze the importance of each channel to the prediction result of the detection model; The feature extraction network's results are processed using the feature fusion network trained in the detection model; the pedestrian detection box is obtained by processing the feature fusion network's results using the detection head structure trained in the detection model; duplicate detection box results are filtered out by non-maximum suppression, and then the pedestrian detection box of the current frame is output.

3. The method according to claim 1, characterized in that, Based on the mutual occlusion of pedestrians, a pedestrian detection dataset is constructed, specifically including: Select the first pedestrian detection dataset, and add some data from the second pedestrian detection dataset to the first pedestrian detection dataset to construct the third pedestrian detection dataset; The bounding box positions in the label file of the third pedestrian detection dataset are transformed, and the bounding box positions are cropped to construct the pedestrian detection dataset; The detection model is trained using the pedestrian detection dataset.

4. The method according to claim 3, characterized in that, Training the detection model using the pedestrian detection dataset specifically includes: The pedestrian detection dataset is processed using data augmentation techniques to obtain a data-augmented pedestrian detection dataset. The detection model is trained using the pedestrian detection dataset after data augmentation.

5. The method according to claim 1, characterized in that, Before processing the real-time video data using a detection model to obtain the detection bounding box for pedestrians in the current frame of the real-time video data, the method further includes: The detection model is compressed. When the detection model is deployed on an FPGA platform, the network structure of the detection model is designed for parallel and pipelined design oriented towards the output channel.

6. The method according to claim 1, characterized in that, The pedestrian detection bounding box data of the current frame is input into the target tracker to obtain the pedestrian prediction bounding box of the future frame. The pedestrian detection bounding box and the pedestrian prediction bounding box are matched, and an identifier is assigned to each pedestrian detection bounding box according to the matching result. Specifically, this includes: The pedestrian detection bounding box data of the current frame is input into the target tracker, and the pedestrian prediction bounding box of the next frame is obtained by the Kalman filter in the target tracker. The cross-union ratio (CUN ratio) is calculated between the pedestrian detection box in the current frame and the pedestrian prediction box obtained by predicting the pedestrian detection box in the previous frame through the Kalman filter. A cost matrix is ​​constructed based on the CUN ratio to map the pedestrians detected in the two frames. The cost matrix is ​​used as input to the Hungarian algorithm in the target tracker to obtain a linear matching result. When the pedestrian detection box of the current frame matches the pedestrian prediction box obtained by predicting the pedestrian detection box of the previous frame through the Kalman filter, the Kalman filter is updated and an identifier is assigned to each pedestrian detection box.

7. An electronic device, characterized in that, include: Memory, used to store programs; A processor is configured to execute the program stored in the memory, wherein when the program is executed, the processor is configured to perform the pedestrian flow detection method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the pedestrian flow detection method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images