A people flow counting method and system based on TOF technology

Abstract

The application relates to a people flow counting method and system based on a TOF technology, which comprises the following steps: S1, shooting a TOF video; S2, acquiring position information of all pedestrians and interference objects in each frame and filtering out misrecognized parts in the position information of the pedestrians; S3, inputting the position information of the pedestrians into a dynamic cache area with a predetermined length frame by frame and calculating and acquiring intermediate track paths of all pedestrian targets in a current frame; S4, updating the current frame in real time, updating the intermediate track paths of all pedestrian targets in the current frame; S5, when a certain pedestrian target disappears in the picture, filtering the intermediate track paths to obtain a final track path of the pedestrian target, determining a people flow path and a people flow direction of the pedestrian target and then comprehensively counting the people flow path and the people flow. The application can effectively reduce the influence of non-pedestrian interference objects on people flow counting results in various scenes and improve the accuracy of people flow counting results in different scenes.

Description

Technical Field

[0001] This invention belongs to the field of people flow statistics technology, and in particular relates to a people flow statistics method and system based on TOF technology. Background Technology

[0002] In numerous scenarios such as subways, buses, and shopping malls, real-time pedestrian flow statistics can provide crucial information for pedestrian management, resource management, and management decisions. With the development of communication technology, various pedestrian counting technologies based on video images have emerged. However, most existing pedestrian counting technologies are based on surveillance videos captured by ordinary cameras, and their statistical results are easily affected by scene factors such as lighting and visibility, lacking accuracy.

[0003] TOF technology is a distance sensing technology based on active light sources. The sensor emits modulated near-infrared light, which is reflected when it encounters an object. The sensor calculates the distance to the photographed object by calculating the time difference or phase difference between the emission and reflection of the light, thus generating depth information. Combined with traditional camera shooting, the three-dimensional outline of the object can be presented as a topographic map with different colors representing different distances.

[0004] Using Time-of-Flight (TOF) cameras to capture video data, combined with object detection algorithms, can avoid the influence of scene factors such as lighting and visibility, thus improving the accuracy of pedestrian flow statistics. Regarding TOF camera-based pedestrian flow statistics methods, some existing technologies exist, but most only detect pedestrians in the video frame, without considering interference from non-pedestrian background objects such as wheelchairs, bicycles, and backpacks. Therefore, the accuracy of the pedestrian flow statistics is still affected by the video shooting scene, and the statistical error is larger in scenes with many interfering objects. Summary of the Invention

[0005] The purpose of this invention is to solve one of the above-mentioned technical problems and to provide a method and system for counting people based on Time-of-Flight (TOF) technology.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for counting people based on Time-of-Flight (TOF) technology includes the following steps: S1: Use a TOF camera to capture TOF video in real time; S2: Detect pedestrian targets and non-pedestrian interference targets in the TOF video frame by frame, output all pedestrian position information and interference position information in each frame, and filter out the misidentified interference position information in the pedestrian position information; S3: Frame by frame, the extracted pedestrian position information after filtering out the location information of interference objects is input into a dynamic buffer of a predetermined length. After the dynamic buffer is filled, the intermediate trajectory path of all pedestrian targets in the current frame is obtained based on the pedestrian position information in the current frame and the pedestrian position information in each frame of the dynamic buffer. The current frame is the last frame stored in the dynamic buffer. S4: Update the current frame in the dynamic buffer in real time. Each time the current frame is updated, the intermediate trajectory path of all pedestrian targets in the current frame is also updated and calculated. S5: When a pedestrian disappears from the TOF video frame, multiple intermediate trajectory paths of the pedestrian are filtered to obtain the final trajectory path of the pedestrian. Based on the final trajectory path, the pedestrian flow path and direction are determined. The real-time pedestrian flow path statistics are updated based on the pedestrian flow path, and the real-time pedestrian flow statistics are updated based on the pedestrian flow direction.

[0007] In some embodiments of the present invention, the method for filtering out misidentified interference location information in pedestrian location information includes the following steps: Frame by frame, the interaction test is performed on all pedestrian location information and all interference object location information in each frame, and the interaction amount between each pedestrian target and each interference object target is calculated. When the interaction between a pedestrian and an interfering object reaches a predetermined interaction threshold, the location information of that pedestrian is filtered out.

[0008] In some embodiments of the present invention, the method for calculating the interaction amount between each pedestrian target and each interfering target includes the following steps: Iterate through the pedestrian and distracting targets in a single frame of the image; Calculate the distance between the center point of the detection box of each pedestrian target and the center point of the detection box of each interfering target, and calculate the intersection-union ratio (IU) between the detection boxes of each pedestrian target and each interfering target. Based on the distance and IU, calculate the interaction amount between each pedestrian target and each interfering target.

[0009] In some embodiments of the present invention, step S1 further includes the following steps: Based on the distance between the TOF camera and the object being photographed, multiple distance segments and a corresponding rendering coefficient for each distance segment are set. The image of each distance segment is rendered using its corresponding rendering coefficient to obtain a depth image. All images in the TOF video were converted into depth images for video object detection.

[0010] In some embodiments of the present invention, the method for obtaining the intermediate trajectory paths of all pedestrian targets in the current frame includes: A multi-level matching method based on IOU matching and cascading matching is used to perform multi-level position matching on pedestrian position information in the dynamic buffer, and output the intermediate trajectory path of all pedestrian targets in the current frame.

[0011] In some embodiments of the present invention, the method for performing multi-level position matching includes the following steps: Starting from the current frame, perform Hungarian matching on all pedestrian position information in each frame and all pedestrian position information in the previous frame, and continue matching forward until all image frames in the dynamic buffer have been matched. Then, output the intermediate trajectory path of all pedestrian targets in the current frame.

[0012] In some embodiments of the present invention, the method for determining the pedestrian flow direction based on the final trajectory path includes: The position information of the pedestrian target in each frame of the image is used as the trajectory points that make up the final trajectory path of the pedestrian target. Linear fitting is performed on all trajectory points on the final trajectory path. Based on the linearly fitted path, the positions where the pedestrian target appears and disappears are determined, and then the direction of pedestrian flow is determined.

[0013] In some embodiments of the present invention, the following steps are also included: Determine the scene category captured by the TOF camera; correct the real-time pedestrian flow statistics based on the scene category captured by the TOF camera.

[0014] Some embodiments of the present invention further provide a people flow statistics system based on Time-of-Flight (TOF) technology, including: TOF camera: Used to capture TOF video data; Target detection module: Used to perform video target detection frame by frame on the TOF video and output the location information of the detected targets; the detected targets include pedestrian targets and interference targets; The timing matching module is used to input pedestrian location information into the dynamic buffer frame by frame, update the current frame in the dynamic buffer in real time, perform multi-level position matching on the pedestrian location information in the dynamic buffer each time the current frame is updated, and output the intermediate trajectory path of all pedestrian targets in the current frame. Trajectory filtering module: When a pedestrian disappears from the TOF video frame, it filters multiple intermediate trajectory paths of the pedestrian and outputs the final trajectory path of the pedestrian. The comprehensive statistics module is used to determine the pedestrian flow path and direction based on the final trajectory path output by the trajectory filtering module, and to update the real-time pedestrian flow path statistics based on the pedestrian flow path and the real-time pedestrian flow statistics based on the pedestrian flow direction.

[0015] In some embodiments of the present invention, the target detection module includes a pedestrian detection unit, an interference detection unit, and an interaction testing unit; The pedestrian detection unit is used to detect pedestrian targets in the TOF video frame by frame and output the location information of all pedestrians in each frame; The interference detection unit is used to detect interference targets in the TOF video frame by frame and output the location information of all interference targets in each frame; The interaction test unit is used to perform interaction tests on pedestrian location information and interference object location information in each frame of image and calculate the interaction amount. When the interaction amount between a pedestrian target and an interference object target reaches a predetermined threshold, the location information of the pedestrian target is filtered out.

[0016] The beneficial effects of this invention are as follows: 1. This invention uses a TOF camera to capture TOF video data, performs target detection on the TOF video data, and then counts pedestrian paths and pedestrian flow. Compared with the traditional method of counting pedestrian flow based on surveillance video captured by ordinary cameras, it can avoid the influence of scene factors such as lighting and visibility on the statistical results, making the statistical method provided by this invention adaptable to more scenarios and improving the accuracy of pedestrian path and pedestrian flow statistics in various scenarios. 2. This invention can detect non-pedestrian interference in TOF video images. By conducting interactive tests between pedestrian targets and non-pedestrian targets, it removes the location information of misidentified interference in pedestrian location information, improves the accuracy of pedestrian intermediate trajectory route and final trajectory route calculation results, and further improves the accuracy of pedestrian flow path and pedestrian flow statistics results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a people flow statistics method based on Time-of-Flight (TOF) technology. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0020] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0021] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0022] The technical solution of the present invention will be described in detail below with reference to specific embodiments and accompanying drawings.

[0023] As attached Figure 1 As shown, in an illustrative embodiment of a pedestrian flow counting method based on TOF technology according to the present invention, the pedestrian flow counting method includes the following steps.

[0024] S1: Use a TOF camera to capture TOF video in real time.

[0025] In some embodiments of the present invention, step S1 further includes the following steps.

[0026] Based on the mounting height of the TOF camera, that is, based on the distance between the TOF camera and the subject, multiple distance segments are set. For example, three distance segments are set: 0-10 cm, 10 cm-40 cm, and 40 cm-120 cm. A corresponding rendering coefficient is set for each distance segment. The image of each distance segment is rendered using its corresponding rendering coefficient to obtain a depth image with different enhancement effects and clarity for each distance segment.

[0027] After the TOF camera captures TOF video in real time, all images in the TOF video are converted into depth images for subsequent video target detection.

[0028] S2: Using a deep learning-based detection model or an edge-feature-based detection algorithm, detect pedestrian targets and non-pedestrian interference targets such as wheelchairs, bicycles, and backpacks frame by frame in the TOF video converted into a depth image. Output the location information of all pedestrians and all interference targets in each frame. When the overlap between the positions of pedestrians and interference targets in the TOF video is high, the detection model or algorithm may misidentify interference targets as pedestrian targets and output the location information of interference targets as pedestrian location information. Therefore, before performing pedestrian flow statistics based on the pedestrian location information of each frame, it is necessary to correct the detection results of each frame of the TOF video and filter out the misidentified interference location information in the pedestrian location information of each frame to improve the accuracy of the pedestrian flow statistics results.

[0029] In some embodiments of the present invention, the method for filtering out misidentified interference location information in pedestrian location information includes the following steps.

[0030] The interaction test is performed frame by frame on all pedestrian and interference location information. The interaction test uses IoU distance and Euclidean distance for joint judgment and calculates the interaction amount between each pedestrian target and each interference target.

[0031] When the interaction between a pedestrian and an interfering object reaches a predetermined interaction threshold, the location information of that pedestrian is filtered out.

[0032] In some embodiments of the present invention, the method for calculating the interaction amount between each pedestrian target and each interfering target includes the following steps.

[0033] Iterate through pedestrian targets and distracting targets in a frame of an image.

[0034] Calculate the distance between the center point of the detection box for each pedestrian target and the center point of the detection box for each interfering target, and calculate the intersection-union ratio (IUGR) between the detection boxes for each pedestrian target and each interfering target. The interaction amount between each pedestrian target and each interfering target is then calculated according to their respective predetermined weights.

[0035] S3: Input the pedestrian position information after filtering out the location information of interference objects into a dynamic buffer of a predetermined length frame by frame. When the dynamic buffer is filled, perform multi-target tracking on all pedestrian targets in the current frame. Based on the pedestrian position information in the current frame and the pedestrian position information in each frame of the dynamic buffer, match and calculate to output the intermediate trajectory path of all pedestrian targets in the current frame.

[0036] The current frame is the last frame stored in the dynamic buffer, meaning it's the most recently added frame. Once the dynamic buffer is full, the current frame is continuously updated with the input of new video frames. The purpose of setting up the dynamic buffer is to store pedestrian position information for a period of time preceding the current frame.

[0037] In some embodiments of the present invention, the method for obtaining the intermediate trajectory paths of all pedestrian targets in the current frame includes the following steps.

[0038] A multi-level matching method based on IOU matching and cascading matching is used to perform multi-level position matching on pedestrian position information in the dynamic buffer to obtain the intermediate trajectory path of all pedestrian targets in the current frame.

[0039] In some embodiments of the present invention, the method for performing multi-level position matching includes the following steps.

[0040] Starting from the current frame, Hungarian matching is performed on all pedestrian position information in each frame and then sequentially with all pedestrian position information in the previous frame. This matching continues forward until all image frames in the dynamic buffer have been matched. In other words, multi-level position matching starts from the current frame and traces back to the earliest image frame stored in the dynamic buffer. After matching is complete, the intermediate trajectory paths of all pedestrian targets in the current frame are calculated. Since the dynamic buffer only stores pedestrian position information from a period preceding the current frame, the calculated intermediate trajectory paths represent the movement trajectories of pedestrian targets during that period.

[0041] S4: Based on real-time captured TOF video, the current frame in the dynamic buffer is updated in real time. Each time the current frame is updated, the intermediate trajectory paths of all pedestrian targets in the current frame are also updated and calculated. It should be noted that for the same pedestrian target, the intermediate trajectory path of the pedestrian target is calculated every time the current frame is updated. Therefore, multiple intermediate trajectory paths of the pedestrian target will be generated during the continuous updating of the current frame.

[0042] S5: When a pedestrian disappears from the TOF video frame, a Kalman filter algorithm is used to filter multiple intermediate trajectory paths of the pedestrian to obtain the final trajectory path. Based on the final trajectory path, the pedestrian flow path and direction are determined. The pedestrian flow path is a dynamic path obtained based on the final trajectory path. The pedestrian flow paths and directions of each pedestrian in the TOF video are comprehensively statistically analyzed. Real-time pedestrian flow path statistics are updated based on the pedestrian flow paths, and entry and exit pedestrian flow statistics are performed based on the pedestrian flow directions, with the real-time pedestrian flow statistics also updated.

[0043] In some embodiments of the present invention, the method for determining the pedestrian flow direction based on the final trajectory path includes the following steps.

[0044] The position information of the pedestrian target in each frame of the image is used as the trajectory points that make up the final trajectory path of the pedestrian target. Linear fitting is performed on all trajectory points on the final trajectory path. Based on the linearly fitted path, the positions where the pedestrian target appears and disappears are determined, and then the direction of pedestrian flow is determined.

[0045] In some embodiments of the present invention, the following steps are also included.

[0046] The system corrects for biases in the statistically obtained pedestrian traffic data based on big data. First, a large amount of Time-of-Flight (TOF) video data from different shooting scenarios is acquired. Pedestrian traffic in the acquired TOF videos is then statistically analyzed both manually and using algorithms (steps S1-S5). The average error of the pedestrian traffic statistics for each scenario is calculated based on the results of both manual and algorithmic analysis. Scenario categories can be defined based on pedestrian traffic, such as sparse traffic, normal traffic, and crowded traffic.

[0047] Before updating the real-time pedestrian path statistics and real-time pedestrian flow statistics in step S5, the scene category captured by the TOF camera is first determined; the real-time pedestrian flow statistics are corrected based on the current scene category and the calculated average error.

[0048] Some embodiments of the present invention further provide a people flow statistics system based on TOF technology, including a TOF camera, a target detection module, a time-series matching module, a trajectory filtering module, and a comprehensive statistics module.

[0049] The TOF camera is used to capture TOF video data.

[0050] The target detection module performs video target detection frame by frame on the TOF video and outputs the location information of the detected targets. The detected targets include pedestrian targets and interfering targets.

[0051] The timing matching module is used to input pedestrian location information into the dynamic buffer frame by frame. Each time the current frame is updated, multi-level position matching is performed on the pedestrian location information in the dynamic buffer, and the intermediate trajectory path of all pedestrian targets in the current frame is output.

[0052] The trajectory filtering module is used to filter multiple intermediate trajectory paths of a pedestrian when the pedestrian disappears from the TOF video frame, and output the final trajectory path of the pedestrian.

[0053] The comprehensive statistics module is used to determine the pedestrian's path and direction based on the final trajectory path output by the trajectory filtering module, and to update the real-time pedestrian path statistics based on the pedestrian path and the real-time pedestrian flow statistics based on the pedestrian direction.

[0054] In some embodiments of the present invention, the target detection module includes a pedestrian detection unit, an interference detection unit, and an interaction testing unit.

[0055] The pedestrian detection unit is used to detect pedestrian targets in the TOF video frame by frame and output the position information of all pedestrian targets in each frame.

[0056] The interference detection unit is used to detect interference targets in the TOF video frame by frame and output the position information of all interference targets in each frame.

[0057] The interaction test unit is used to perform interaction tests on pedestrian location information and interference object location information in each frame of image and calculate the interaction amount. When the interaction amount between a pedestrian target and an interference object target reaches a predetermined threshold, the location information of the pedestrian target is filtered out.

[0058] Finally, it should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0059] The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A method for counting people based on Time-of-Flight (TOF) technology, characterized in that, Includes the following steps: S1: Use a TOF camera to capture TOF video in real time; S2: Use a deep learning-based detection model or an edge feature-based detection algorithm to detect pedestrian targets and non-pedestrian interference targets in the TOF video frame by frame, output all pedestrian position information and interference position information in each frame, and filter out the misidentified interference position information in the pedestrian position information. The method for filtering out misidentified interfering object location information from pedestrian location information includes the following steps: Frame by frame, the interaction test is performed on all pedestrian location information and all interference object location information in each frame, and the interaction amount between each pedestrian target and each interference object target is calculated. When the interaction between a pedestrian and an interfering object reaches a predetermined interaction threshold, the location information of that pedestrian is filtered out. The method for calculating the interaction amount between each pedestrian target and each interfering target includes the following steps: Iterate through the pedestrian and distracting targets in a single frame of the image; Calculate the distance between the center point of the detection box of each pedestrian target and the center point of the detection box of each interfering target, and calculate the intersection-union ratio of the detection boxes of each pedestrian target and each interfering target. Based on the distance and the intersection-union ratio, calculate the interaction amount between each pedestrian target and each interfering target. S3: Input the pedestrian position information after filtering out the location information of interference objects into a dynamic buffer of a predetermined length frame by frame. After the dynamic buffer is filled, calculate the intermediate trajectory path of all pedestrian targets in the current frame based on the pedestrian position information in the current frame and the pedestrian position information in each frame of the dynamic buffer. The current frame is the last frame stored in the dynamic buffer. S4: Update the current frame in the dynamic buffer in real time, and update the intermediate trajectory path of all pedestrian targets in the current frame each time the current frame is updated; S5: When a pedestrian disappears from the TOF video frame, multiple intermediate trajectory paths of the pedestrian are filtered to obtain the final trajectory path of the pedestrian. Based on the final trajectory path, the pedestrian flow path and direction are determined. The real-time pedestrian flow path statistics are updated based on the pedestrian flow path, and the real-time pedestrian flow statistics are updated based on the pedestrian flow direction.

2. The pedestrian flow counting method based on TOF technology according to claim 1, characterized in that, Step S1 also includes the following steps: Based on the distance between the TOF camera and the object being photographed, multiple distance segments and a corresponding rendering coefficient for each distance segment are set. The image of each distance segment is rendered using its corresponding rendering coefficient to obtain a depth image. The images in the TOF video are all converted into depth images for video object detection.

3. The pedestrian flow counting method based on TOF technology according to claim 1, characterized in that, Methods for obtaining the intermediate trajectory paths of all pedestrian targets in the current frame include: The pedestrian location information in the dynamic buffer is matched in multiple levels using a multi-level matching method based on IOU matching and cascading matching, and the intermediate trajectory path of all pedestrian targets in the current frame is output.

4. The pedestrian flow counting method based on TOF technology according to claim 3, characterized in that, The method for performing multi-level location matching includes the following steps: Starting from the current frame, Hungarian matching is performed on all pedestrian position information in each frame and all pedestrian position information in the previous frame, and so on, until all image frames in the dynamic buffer are matched. Then, the intermediate trajectory path of all pedestrian targets in the current frame is output.

5. The pedestrian flow counting method based on TOF technology according to claim 1, characterized in that, Methods for determining the pedestrian flow direction based on the final trajectory path include: The position information of the pedestrian target in each frame of the image is used as the trajectory points that make up the final trajectory path of the pedestrian target. Linear fitting is performed on all trajectory points on the final trajectory path. Based on the linearly fitted path, the positions where the pedestrian target appears and disappears are determined, and then the direction of pedestrian flow is determined.

6. The pedestrian flow counting method based on TOF technology according to claim 1, characterized in that, It also includes the following steps: Determine the scene category captured by the TOF camera; correct the real-time pedestrian flow statistics based on the scene category captured by the TOF camera.

7. A people counting system based on Time-of-Flight (TOF) technology, used to implement the people counting method based on TOF technology as described in any one of claims 1-6, comprising: TOF camera: Used to capture TOF video data; Target detection module: Used to perform video target detection frame by frame on the TOF video and output the location information of the detected targets; The timing matching module is used to input pedestrian location information into the dynamic buffer frame by frame, update the current frame in the dynamic buffer in real time, and update the intermediate trajectory path of all pedestrian targets in the current frame each time the current frame is updated. Trajectory filtering module: When a pedestrian disappears from the TOF video frame, it filters multiple intermediate trajectory paths of the pedestrian and outputs the final trajectory path of the pedestrian. The comprehensive statistics module is used to determine the pedestrian flow path and direction based on the final trajectory path output by the trajectory filtering module, and to update the real-time pedestrian flow path statistics results based on the pedestrian flow path and the real-time pedestrian flow statistics results based on the pedestrian flow direction.

Images