A method, device, electronic device and storage medium for counting the entry and exit of personnel

Abstract

This application discloses a method, device, electronic device, and storage medium for counting people entering and exiting the passage. The method includes: generating a 4D super-resolution dense point cloud in the current frame using a millimeter-wave radar sensor; wherein the millimeter-wave radar sensor is positioned directly above the passageway and facing vertically downward; performing multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud; performing multi-extended target tracking based on the multi-scale clustering result to obtain a multi-extended target tracking result of the multi-scale clustering result; and counting people passing through the passageway based on the multi-extended target tracking result. Embodiments of the present application can accurately count people entering and exiting the passageway while also reducing power consumption and saving costs.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of radar technology, and in particular to a method, device, electronic device, and storage medium for counting people entering and exiting. Background Art

[0002] People entry and exit detection and counting technology can monitor the entry and exit of people at places like doorways and hallways in real time, detecting and counting people entering and leaving. It can be widely used in smart buildings, security management, business analytics, and other fields. For example, detecting the number of people entering and exiting a doorway or passageway can be used for business data analysis, allowing businesses to rationally allocate staff and formulate business strategies. The number of people entering and exiting important places such as subway entrances, stations, and airports is crucial for controlling traffic flow and public safety. Furthermore, the number of people entering and exiting can be used to indirectly infer the number of people in a certain area (such as a room), allowing fire departments to quickly locate people in the event of an emergency such as a fire.

[0003] At present, the main methods for detecting and counting people entering and exiting are: (1) Methods based on passive infrared sensors (PIR). PIR sensors detect people by sensing infrared rays emitted by the human body. The advantages are low power consumption and low price; however, they can only detect the presence of people at close range and cannot distinguish between multiple people or detect the direction of people entering and exiting. (2) Methods based on red, green, and blue (RGB) cameras. This method can provide rich information in images and videos, but if intelligent perception tasks such as detecting and counting people are to be performed, a powerful processor is required to complete it, which is costly, and the performance drops sharply in low light and darkness. It also brings the problem of infringing personal privacy, which greatly hinders its application and deployment. (3) Methods based on time of flight (TOF) cameras. TOF cameras generate images with depth information by sending laser pulses. This method does not infringe personal privacy, but TOF cameras are very expensive and also require high-performance processors for complex processing of high-resolution image information, and have high power consumption. (4) Methods based on millimeter wave radar. However, the existing millimeter wave radar-based methods have three problems that affect the accuracy of people counting. The first issue is that the sideways mounting of the radar causes strong multipath interference. The second issue is that the clusters formed by point cloud clustering are simply used as the location and trajectory of the person target in that frame of data. However, the complexity of people's movements significantly affects the radar point cloud. If the point clouds of multiple people are clustered together during movement, or if the point cloud of a single person cannot be clustered, the counting accuracy will be greatly reduced. The third issue is that the internal characteristics of the three-dimensional point cloud clusters are not utilized to further analyze the clusters to improve detection accuracy. Summary of the Invention

[0004] The present application provides a method, device, electronic device and storage medium for counting people entering and leaving, which can accurately count people entering and leaving, and can also reduce power consumption and save costs.

[0005] In a first aspect, an embodiment of the present application provides a method for counting people entering and exiting, the method comprising:

[0006] Generate a 4D super-resolution dense point cloud in the current frame using a millimeter-wave radar sensor; wherein the millimeter-wave radar sensor is positioned directly above the entrance and exit channel and is placed vertically downward;

[0007] Performing multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud;

[0008] Performing multi-extended target tracking based on the multi-scale clustering result to obtain a multi-extended target tracking result of the multi-scale clustering result;

[0009] The people passing through the entry and exit passage are counted based on the multi-extended target tracking results.

[0010] In a second aspect, an embodiment of the present application further provides a people entry and exit counting device, the device comprising: a point cloud generation module, a multi-scale clustering module, a multi-extended target tracking module and a people counting module; wherein,

[0011] The point cloud generation module is configured to generate a 4D super-resolution dense point cloud in the current frame using a millimeter-wave radar sensor; wherein the millimeter-wave radar sensor is disposed directly above the entrance and exit passage and is placed vertically downward;

[0012] The multi-scale clustering module is used to perform multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud;

[0013] The multi-extended target tracking module is used to perform multi-extended target tracking based on the multi-scale clustering result to obtain a multi-extended target tracking result of the multi-scale clustering result;

[0014] The people counting module is used to count people passing through the entry and exit channels based on the multi-extended target tracking results.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0016] one or more processors;

[0017] a memory for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for counting people entering and exiting as described in any embodiment of the present application.

[0019] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for counting people entering and exiting as described in any embodiment of the present application is implemented.

[0020] The embodiments of the present application propose a method, device, electronic device and storage medium for counting people entering and exiting. First, a millimeter-wave radar sensor is used to generate a 4D super-resolution dense point cloud in the current frame. Then, the 4D super-resolution dense point cloud is clustered at multiple scales. Then, multi-extended target tracking is performed based on the multi-scale clustering results. Finally, the people passing through the entry and exit channels are counted based on the multi-extended target tracking results. That is, in the technical solution of the present application, the entry and exit of people are counted by installing the millimeter-wave radar sensor at the door or the top of the channel and pointing it vertically downward, while identifying the entry and exit direction of the people. Even if the distance between people is close, the number of people entering and exiting can be accurately detected, and even when many people pass by densely, the number can be accurately counted. Moreover, it is not affected by ambient light, can work all day and all weather, has low power consumption, is small in size, and does not infringe on personal privacy. In existing millimeter-wave radar-based methods, the sideways installation of the radar will cause strong multipath interference, and the movement of people will also reduce the accuracy of the counting. Therefore, compared with the prior art, the method, device, electronic device and storage medium for counting people entering and leaving the embodiment of the present application can accurately count people entering and leaving, and can also reduce power consumption and save costs; moreover, the technical solution of the embodiment of the present application is simple and convenient to implement, easy to popularize, and has a wider range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the first flow chart of the method for counting people entering and exiting provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of an installation scenario for a millimeter-wave radar sensor according to an embodiment of the present application;

[0023] Figure 3 A second flow chart of the method for counting people entering and exiting provided in an embodiment of the present application;

[0024] FIG4( a ) is a schematic diagram of the structure of point cloud data generated when a person enters or exits according to an embodiment of the present application;

[0025] FIG4( b ) is a schematic diagram of the structure of point cloud data generated when two adjacent persons enter or exit according to an embodiment of the present application;

[0026] Figure 5A schematic diagram of the structure of the personnel entry and exit area provided in the embodiment of the present application;

[0027] Figure 6 A third flow chart of the method for counting people entering and exiting provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of a personnel entry and exit counting device provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present application and are not intended to limit the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the present application, not all of the structures.

[0031] Example 1

[0032] Figure 1 This is a first flow chart of the method for counting people in and out provided in an embodiment of the present application. This method can be executed by a people in and out counting device or electronic device. The device or electronic device can be implemented in software and / or hardware. The device or electronic device can be integrated into any smart device with network communication function. Figure 1 As shown, the method for counting people entering and exiting may include the following steps:

[0033] S101. Generate a 4D super-resolution dense point cloud in the current frame using a millimeter-wave radar sensor; wherein the millimeter-wave radar sensor is arranged directly above the entrance and exit channel and placed vertically downward.

[0034] In this step, the electronic device can generate a 4D super-resolution dense point cloud in the current frame through the millimeter-wave radar sensor; wherein the millimeter-wave radar sensor is arranged directly above the entrance and exit channel and placed vertically downward. The embodiment of the present application can use a radar chip or other chips. Specifically, the electronic device can first perform fast Fourier transform processing on the signal received by the millimeter-wave radar sensor in the fast time dimension; then use a super-resolution multiple signal classification algorithm to extract multiple angle information from the fast Fourier transform result; then detect each angle information to obtain a set of point cloud data in a spherical coordinate system; and obtain the speed estimate value of each point cloud data in the set of point cloud data; based on the spherical coordinates and speed estimate values of each point cloud data, a 4D super-resolution dense point cloud is obtained.

[0035] Figure 2This is a schematic diagram of the installation scenario of the millimeter wave radar sensor provided in the embodiment of this application. Figure 2 As shown, the millimeter-wave radar sensor can include a radar module that can be installed directly above a doorway or passageway, facing vertically downward. People passing through the passageway can move from area A to area B, or from area B to area A.

[0036] S102 , performing multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud.

[0037] In this step, the electronic device may perform multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud. Specifically, the electronic device may first perform global-scale clustering on the 4D super-resolution dense point cloud to obtain a global-scale clustering result of the 4D super-resolution dense point cloud; and then perform local-scale clustering on the 4D super-resolution dense point cloud based on the global-scale clustering result to obtain a local-scale clustering result of the 4D super-resolution dense point cloud.

[0038] S103 , performing multi-extended target tracking based on the multi-scale clustering result to obtain a multi-extended target tracking result of the multi-scale clustering result.

[0039] In this step, the electronic device can perform multi-extended target tracking based on the multi-scale clustering results, obtaining a multi-extended target tracking result of the multi-scale clustering results. Specifically, the electronic device can first calculate the distance between all existing trajectories and each point cloud data in the set of point cloud data; and based on the distance between all existing trajectories and each point cloud data, associate all existing trajectories with the global clusters in the multi-scale clustering results; and then, based on the association results of all existing trajectories with the global clusters, associate all existing trajectories with each point cloud data in the set of point cloud data.

[0040] S104: Counting people passing through the entry and exit channels based on the multi-extended target tracking results.

[0041] The method for counting people entering and exiting proposed in the embodiment of the present application first generates a 4D super-resolution dense point cloud in the current frame through a millimeter-wave radar sensor, then performs multi-scale clustering on the 4D super-resolution dense point cloud, and then performs multi-extended target tracking based on the multi-scale clustering results. Finally, the number of people passing through the entry and exit channels is counted based on the multi-extended target tracking results. That is, in the technical solution of the present application, the entry and exit of people is counted by installing the millimeter-wave radar sensor at the door or the top of the channel and pointing it vertically downward, while identifying the entry and exit direction of the people. Even if the distance between people is close, the number of people entering and exiting can be accurately detected, and even when many people pass by densely, the number can be accurately counted; and it is not affected by ambient light, can work all day and all weather, has low power consumption, is small in size, and does not infringe on personal privacy. In the existing method based on millimeter-wave radar, the sideways installation of the radar will cause strong multi-path interference, and the movement of people will also reduce the accuracy of the counting. Therefore, compared with the existing technology, the personnel entry and exit counting method proposed in the embodiment of the present application can accurately count the number of people entering and exiting, and can also reduce power consumption and save costs; moreover, the technical solution of the embodiment of the present application is simple and convenient to implement, easy to popularize, and has a wider range of applications.

[0042] Example 2

[0043] Figure 3 This is a second flow chart of the method for counting people entering and exiting provided in the embodiment of the present application. It is further optimized and expanded based on the above technical solution and can be combined with the above optional implementation methods. Figure 3 As shown, the method for counting people entering and exiting may include the following steps:

[0044] S301 , performing fast Fourier transform processing on the signal received by the millimeter-wave radar sensor in the fast time dimension.

[0045] In this step, the electronic device can perform fast Fourier transform processing on the signal received by the millimeter-wave radar sensor in the fast time dimension. Specifically, the electronic device can install the millimeter-wave radar sensor vertically downward directly above the doorway or aisle. The millimeter-wave radar sensor radiates electromagnetic wave signals in the form of frequency-modulated continuous waves through the antenna. After being scattered by doors, walls, the ground, and possible human targets, the millimeter-wave radar sensor receives the echo signal, mixes it, and samples it through an analog-to-digital converter (ADC). The resulting signal can be expressed as y(m, n, k), which is a three-dimensional data cube signal; where m represents the fast time dimension signal sampling, n represents the slow time dimension signal sampling, and k represents the antenna dimension signal sampling.

[0046] Next, perform FFT processing on the fast time dimension m of the signal y(m,n,k) and extract the distance information according to the following formula: x′[r,n,k]=FFT{y[m,n,k]}; where x′[r,n,k] is the result of performing FFT processing on the fast time dimension m to extract the distance information.

[0047] Then, static clutter is removed for each range unit by subtracting the mean of the corresponding slow time dimension from each range unit: Among them, x′[r,n,k] is the result of FFT extraction of distance information in the fast time dimension m, x[r,n,k] is the result of static clutter removal, and N is the slow time dimension.

[0048] S302: Use a super-resolution multiple signal classification algorithm to extract multiple angle information from the fast Fourier transform result.

[0049] In this step, the electronic device can use a super-resolution multiple signal classification algorithm to extract multiple angle information from the fast Fourier transform result. Specifically, the covariance matrix of the echo signal is constructed according to the following formula: The covariance matrix is decomposed into eigenvalues according to the following formula: The spectrum value of this angle is determined according to the following formula: Among them, R kk,r is the constructed signal covariance matrix, N is the number of chirp waveforms in a frame of data, U s is the signal subspace eigenvector matrix, Σ s is the signal subspace eigenvalue matrix, U n is the noise subspace eigenvector matrix, Σ n is the noise subspace eigenvalue matrix, is the steering vector of the array element, θ and They are the azimuth and elevation angles being searched respectively. The super-resolution multiple signal classification algorithm in the embodiment of the present application needs to determine the number of signal sources. The present application can set the number of signal sources to 2. Although the number of signal sources is set to 2, the detection strategy of the present application is to detect multiple points from the angle spectrum, not just 2 peak points, so that more points can be obtained and a denser point cloud can be generated, which is beneficial for subsequent personnel target detection. The detector in the embodiment of the present application can use a threshold detector or a constant false alarm detector to obtain a set of point cloud data in a spherical coordinate system.

[0050] S303. Detect each angle information to obtain a set of point cloud data in a spherical coordinate system; obtain a velocity estimate value for each point cloud data in the set of point cloud data; and obtain a 4D super-resolution dense point cloud based on the spherical coordinates and velocity estimates of each point cloud data.

[0051] In this step, the electronic device can detect each angle information to obtain a set of point cloud data in the spherical coordinate system; and obtain the velocity estimation value of each point cloud data in the set of point cloud data; based on the spherical coordinates and velocity estimation value of each point cloud data, a 4D super-resolution dense point cloud is obtained. Specifically, the electronic device can use a unit average constant false alarm rate (CA-CFAR) detector to detect the extracted angle information and obtain a set of point cloud data in the spherical coordinate system. Then, beamforming and FFT are used to obtain the velocity estimate v of each point. At this time, each point is four-dimensional, that is, in Represents the spatial coordinates in the spherical coordinate system and can be converted into coordinates (x, y, z) in the Cartesian coordinate system, where (x, y) represents the horizontal coordinates and z represents the vertical height.

[0052] Figure 4(a) is a schematic diagram of the structure of the point cloud data generated when one person enters or exits according to an embodiment of the present application. Figure 4(b) is a schematic diagram of the structure of the point cloud data generated when two adjacent people enter or exit according to an embodiment of the present application. As shown in Figures 4(a) and 4(b), the point clouds generated according to the above steps may be generated by doorways, walls, and the ground. These are interference point clouds and can be removed according to the actual scene. For example, if the doorway is 2 meters wide and the millimeter-wave radar sensor is installed in the middle, only the point cloud between -1 meter and 1 meter on the y-axis (assuming the y-axis direction is the door width direction) is retained. For example, assuming the door is 2.2 meters high, only the point cloud within 2.2 meters on the z-axis is retained.

[0053] S304 , performing multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud.

[0054] In this step, the electronic device can perform multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud. Specifically, the electronic device can first perform global-scale clustering on the 4D super-resolution dense point cloud to obtain a global-scale clustering result of the 4D super-resolution dense point cloud; then perform local-scale clustering on the 4D super-resolution dense point cloud based on the global-scale clustering result to obtain a local-scale clustering result of the 4D super-resolution dense point cloud. The purpose of multi-scale clustering in the embodiment of the present application is to better detect personnel targets and perform data association, and provide a basis for multi-extended target tracking. First, global-scale clustering is performed to detect the global information of the point cloud, and then local-scale clustering is performed to detect the local information of the point cloud.

[0055] Global-scale clustering: Global-scale clustering considers the (x, y) coordinates of the entire point cloud for each frame of data. This involves projecting all detected points onto a horizontal plane and clustering them using the density-based clustering algorithm (DBSCAN). Successfully clustered clusters are called global clusters. The characteristics of global clustering are: ① Points generated by a person target will appear in at most one global cluster (possibly because the number of points is too small to be clustered); ② Points generated by multiple people may appear in the same global cluster, typically due to the proximity of people and the limited resolution of the sensor.

[0056] Local scale clustering: Local scale clustering acts on a global cluster, but at the same time takes into account the three-dimensional coordinates (x, y, z) of the detection points within the global cluster. The purpose of local scale clustering is to perform further target detection on the global cluster to detect whether a global cluster contains two human targets. The specific approach is to set one (or more) heights for interception projection, only considering points above the set height, and then use the density clustering algorithm DBSCAN for clustering again. The cluster that is successfully clustered is called a local cluster. If there is one local cluster, it is one target. If there are two local clusters, then this global cluster actually contains two human targets. For example, assuming the height parameter is set to 1.5m, only the point cloud 1.5m above the ground is retained, and then DBSCAN clustering is performed on it. If the number of local clusters is 1, then there is one human target. If the number of local clusters is 2, then the global cluster actually contains two human targets. For another example, assuming that multiple height parameters are set, such as 1.7m, 1.6m, and 1.5m, and the above operation is performed multiple times, if the number of clusters is 2 once, there are 2 personnel targets.

[0057] S305 , performing multi-extended target tracking based on the multi-scale clustering result to obtain a multi-extended target tracking result of the multi-scale clustering result.

[0058] In this step, the electronic device can perform multi-extended target tracking based on the multi-scale clustering results, obtaining a multi-extended target tracking result of the multi-scale clustering results. Specifically, the electronic device can first calculate the distance between all existing trajectories and each point cloud data in the set of point cloud data; and based on the distance between all existing trajectories and each point cloud data, associate all existing trajectories with the global clusters in the multi-scale clustering results; and then, based on the association results of all existing trajectories with the global clusters, associate all existing trajectories with each point cloud data in the set of point cloud data.

[0059] Due to the high resolution of millimeter-wave radar, dozens or even hundreds of detection points can be obtained for human targets. Tracking such targets differs significantly from traditional point target tracking and is known as extended target tracking. Based on the results of multi-scale clustering, this application uses multi-extended target tracking technology to detect and track each person passing through a doorway or passageway, maximizing the likelihood of each person's movement trajectory.

[0060] In one embodiment, the criteria for trajectory initiation are: ① If a point in a global cluster is not associated with an existing trajectory, then the global cluster is the starting position of a new trajectory; ② If a point in a global cluster is associated with only one existing trajectory, then the global cluster is clustered at a local scale. If two local clusters are clustered, the local cluster closest to the existing associated trajectory is the latest position of the trajectory, and the other local cluster is the starting position of the new trajectory.

[0061] In one embodiment, data association based on multi-scale clustering refers to associating existing trajectories with detection points in the current frame. The purpose is to continuously maintain and continue the trajectory. The average position of all detection points associated with the trajectory is the position of the trajectory in the current frame. Assuming that there are several trajectories and the current frame generates a set of point clouds, a distance parameter r (for example, r = 0.4m) is set as the maximum association distance. Data association is carried out in two stages. The first stage is to associate the trajectory with at most one global cluster. The second stage is to complete the final association between the trajectory and the detection points.

[0062] In the first phase, the distances between all trajectories and all detection points are calculated, and detection points are assigned to the closest trajectory, provided that the distance between the detection point and the trajectory is less than the parameter r. There are three possible scenarios: ① If the detection points associated with the trajectory are not in any global cluster, these detection points are finally associated; ② If the detection points associated with the trajectory are in the same global cluster, the trajectory is associated with that global cluster. ③ If the detection points associated with the trajectory are in multiple global clusters, the closest point to the trajectory is found among these detection points, and the global cluster to which this point belongs is associated with the trajectory.

[0063] In the second stage, the point cloud associated with a trajectory is narrowed down to a global cluster. The distances between all trajectories and the detection points in the global cluster associated with them in the first stage are calculated. The detection points are then assigned to the closest trajectory, with the distance between the detection points and the trajectory being less than the parameter r. This completes the association of existing trajectories with detection points in the current frame. The average position of all detection points associated with a trajectory is the position of the trajectory in the current frame. If a trajectory is not associated with any detection points across multiple frames, the trajectory is terminated and a decision is made as to whether to proceed with people counting.

[0064] S306: Count the people passing through the entry and exit channels based on the multi-extended target tracking results.

[0065] In a specific embodiment of the present application, when the trajectory ends, whether to count people is determined for the trajectory based on the preset entry and exit areas and the starting and ending positions of the trajectory. Figure 5 This is a schematic diagram of the structure of the personnel entry and exit area provided in the embodiment of this application. Figure 5 As shown, area A and area B are defined as the entry or exit areas, respectively. When a trajectory ends in area A or area B, the distance difference between the initial position and the end position of the trajectory along the entry and exit direction (the y-axis in the figure) is calculated. When the distance exceeds the threshold, the target is considered to have entered or left. After the counting is completed, the trajectory is deleted. For example, in Example 1, when the target is entering, the trajectory ends in area A. The initial position is subtracted from the current position. When the difference is greater than the threshold (such as 1m), the target is considered to have entered. For another example, when the target is leaving, the trajectory ends in area B. The initial position is subtracted from the current position. When the difference is less than the negative threshold, the target is considered to have left.

[0066] The method for counting people entering and exiting proposed in the embodiment of the present application first generates a 4D super-resolution dense point cloud in the current frame through a millimeter-wave radar sensor, then performs multi-scale clustering on the 4D super-resolution dense point cloud, and then performs multi-extended target tracking based on the multi-scale clustering results. Finally, the number of people passing through the entry and exit channels is counted based on the multi-extended target tracking results. That is, in the technical solution of the present application, the entry and exit of people is counted by installing the millimeter-wave radar sensor at the door or the top of the channel and pointing it vertically downward, while identifying the entry and exit direction of the people. Even if the distance between people is close, the number of people entering and exiting can be accurately detected, and even when many people pass by densely, the number can be accurately counted; and it is not affected by ambient light, can work all day and all weather, has low power consumption, is small in size, and does not infringe on personal privacy. In the existing method based on millimeter-wave radar, the sideways installation of the radar will cause strong multi-path interference, and the movement of people will also reduce the accuracy of the counting. Therefore, compared with the existing technology, the personnel entry and exit counting method proposed in the embodiment of the present application can accurately count the number of people entering and exiting, and can also reduce power consumption and save costs; moreover, the technical solution of the embodiment of the present application is simple and convenient to implement, easy to popularize, and has a wider range of applications.

[0067] Example 3

[0068] Figure 6 This is a third flow chart of the method for counting people entering and exiting provided in the embodiment of the present application. It is further optimized and expanded based on the above technical solution and can be combined with the above optional implementation methods. Figure 6 As shown, the method for counting people entering and exiting may include the following steps:

[0069] S601: Perform fast Fourier transform processing on the signal received by the millimeter-wave radar sensor in the fast time dimension.

[0070] S602: Use a super-resolution multiple signal classification algorithm to extract multiple angle information from the fast Fourier transform result.

[0071] S603. Detect each angle information to obtain a set of point cloud data in a spherical coordinate system; obtain a velocity estimate value for each point cloud data in the set of point cloud data; and obtain a 4D super-resolution dense point cloud based on the spherical coordinates and velocity estimates of each point cloud data.

[0072] S604 , performing global scale clustering on the 4D super-resolution dense point cloud to obtain a global scale clustering result of the 4D super-resolution dense point cloud.

[0073] In this step, global-scale clustering considers the (x, y) coordinates of all point clouds in each frame. This involves projecting all detection points onto a horizontal plane and clustering them using the density-based clustering algorithm (DBSCAN). Successfully clustered clusters are called global clusters. The characteristics of global clusters are: ① Points generated by a person target will appear in at most one global cluster (possibly because the number of points is too small to be clustered); ② Points generated by multiple people may appear in the same global cluster, typically due to the proximity of people and the limited resolution of the sensor.

[0074] S605 , performing local scale clustering on the 4D super-resolution dense point cloud based on the global scale clustering result to obtain a local scale clustering result of the 4D super-resolution dense point cloud.

[0075] The local scale clustering in the embodiment of the present application acts on a global cluster, but at the same time takes into account the three-dimensional coordinates (x, y, z) of the detection points in the global cluster. The purpose of local scale clustering is to perform further target detection on the global cluster to detect whether a global cluster contains two personnel targets. The specific method is to set one (or more) heights for interception projection, only consider points above the set height, and use the density clustering algorithm DBSCAN for clustering again. The cluster that successfully clusters is called a local cluster. If there is one local cluster, it is one target. If there are two local clusters, then this global cluster actually contains two personnel targets.

[0076] S606 , performing multi-extended target tracking based on the multi-scale clustering result to obtain a multi-extended target tracking result of the multi-scale clustering result.

[0077] S607: Count the people passing through the entry and exit channels based on the multi-extended target tracking results.

[0078] In an embodiment of the present application, a millimeter-wave radar sensor is installed directly above a doorway or passageway and placed vertically downward, so that the electromagnetic waves can reach the human target as directly as possible, reducing multipath effect interference; a two-dimensional array radar is used to generate a dense super-resolution 4D point cloud through a super-resolution multiple signal classification algorithm, while removing noise in non-interested areas; an embodiment of the present application proposes a multi-scale clustering method to process the point cloud, which can extract both global and local information of the point cloud, thereby improving the accuracy of human target detection; a multi-extended target tracking method based on multi-scale clustering is proposed, which can form multiple effective trajectories even in a dense crowd of people; the distance difference between the starting position and the ending position of the trajectory is calculated to determine whether a person is entering or exiting and the direction of entry and exit.

[0079] Compared with existing technologies, the advantages of this application are: (1) The millimeter wave radar sensor device is installed just above the door or passage and placed vertically downward, so that the electromagnetic wave reaches the human target as directly as possible, reducing the interference of multipath effects; (2) It is not affected by ambient light and can work all day and all weather without infringing on personal privacy; (3) The computational complexity of this application is low and it can be run on a general DSP processor, which can achieve small size, low cost and low power consumption; (4) The multi-scale clustering method is used to process the point cloud, which can extract both local information and overall information of the point cloud, thereby improving the accuracy of human target detection; (5) A multi-scale clustering-based multi-extended target tracking method is proposed, which can form effective multiple trajectories even in the case of dense crowds; (6) The accuracy is high and it can detect and count people entering and leaving at the same time and determine the direction of entry and exit of people. Even two people entering and leaving very close to each other can be accurately distinguished, and the counting is also relatively accurate when multiple people enter and leave.

[0080] In the embodiment of the present application, radar equipment is deployed for a long time at a certain entrance and exit to evaluate the accuracy of counting people in and out. The door is 2.3 meters high and 1.3 meters wide. The experiment lasts from 11 a.m. to 6 p.m. one day, for a total of 7 hours. The real data of people entering and leaving is obtained by manual counting, which is used as a comparison. By counting the accuracy rate at one-hour time intervals, it can be found that the system counts accurately in most cases, with only occasional sporadic errors. In the 7-hour experiment, the actual manual counting results were 618 people entering / exiting, of which only 9 errors occurred, and the final accuracy rate of counting people in and out was 98.5%. The experimental results show that this work can run at the entrance and exit for a long time to detect and count the number of people entering and leaving, and has a very high accuracy rate.

[0081] The method for counting people entering and exiting proposed in the embodiment of the present application first generates a 4D super-resolution dense point cloud in the current frame through a millimeter-wave radar sensor, then performs multi-scale clustering on the 4D super-resolution dense point cloud, and then performs multi-extended target tracking based on the multi-scale clustering results. Finally, the number of people passing through the entry and exit channels is counted based on the multi-extended target tracking results. That is, in the technical solution of the present application, the entry and exit of people is counted by installing the millimeter-wave radar sensor at the door or the top of the channel and pointing it vertically downward, while identifying the entry and exit direction of the people. Even if the distance between people is close, the number of people entering and exiting can be accurately detected, and even when many people pass by densely, the number can be accurately counted; and it is not affected by ambient light, can work all day and all weather, has low power consumption, is small in size, and does not infringe on personal privacy. In the existing method based on millimeter-wave radar, the sideways installation of the radar will cause strong multi-path interference, and the movement of people will also reduce the accuracy of the counting. Therefore, compared with the existing technology, the personnel entry and exit counting method proposed in the embodiment of the present application can accurately count the number of people entering and exiting, and can also reduce power consumption and save costs; moreover, the technical solution of the embodiment of the present application is simple and convenient to implement, easy to popularize, and has a wider range of applications.

[0082] Example 4

[0083] Figure 7 This is a schematic diagram of the structure of the personnel entry and exit counting device provided in the embodiment of the present application. Figure 7 As shown, the personnel entry and exit counting device includes: a point cloud generation module 701, a multi-scale clustering module 702, a multi-extended target tracking module 703 and a personnel counting module 704; wherein,

[0084] The point cloud generation module 701 is configured to generate a 4D super-resolution dense point cloud in the current frame using a millimeter-wave radar sensor; wherein the millimeter-wave radar sensor is disposed directly above the entrance and exit passage and is placed vertically downward;

[0085] The multi-scale clustering module 702 is used to perform multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud;

[0086] The multi-extended target tracking module 703 is configured to perform multi-extended target tracking based on the multi-scale clustering result to obtain a multi-extended target tracking result of the multi-scale clustering result;

[0087] The people counting module 704 is configured to count people passing through the entry and exit passage based on the multi-extended target tracking result.

[0088] Furthermore, the point cloud generation module 701 is specifically used to perform fast Fourier transform processing on the signal received by the millimeter-wave radar sensor in the fast time dimension; use a super-resolution multiple signal classification algorithm to extract multiple angle information from the fast Fourier transform result; detect each angle information to obtain a set of point cloud data in a spherical coordinate system; and obtain a speed estimation value for each point cloud data in the set of point cloud data; and obtain the 4D super-resolution dense point cloud based on the spherical coordinates and speed estimation values of each point cloud data.

[0089] Furthermore, the multi-scale clustering module 702 is specifically used to perform global scale clustering on the 4D super-resolution dense point cloud to obtain a global scale clustering result of the 4D super-resolution dense point cloud; and perform local scale clustering on the 4D super-resolution dense point cloud based on the global scale clustering result to obtain a local scale clustering result of the 4D super-resolution dense point cloud.

[0090] Furthermore, the multi-extended target tracking module 703 is specifically used to calculate the distances between all existing trajectories and each point cloud data in the set of point cloud data; and according to the distances between all existing trajectories and each point cloud data, associate all existing trajectories with the global clusters in the multi-scale clustering results; and according to the association results of all existing trajectories with the global clusters, associate all existing trajectories with each point cloud data in the set of point cloud data.

[0091] The above-mentioned personnel entry and exit counting device can execute the method provided by any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in this embodiment, please refer to the personnel entry and exit counting method provided by any embodiment of the present application.

[0092] Example 5

[0093] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 A block diagram of an exemplary electronic device suitable for implementing the embodiments of the present application is shown. Figure 8 The electronic device 12 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0094] like Figure 8 As shown, electronic device 12 is implemented as a general-purpose computing device. Components of electronic device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0095] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0096] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0097] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 8 Not shown, often called a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present application.

[0098] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0099] The electronic device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the electronic device 12, and / or any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the electronic device 12 via the bus 18. It should be understood that although Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0100] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the personnel entry and exit counting method provided in the embodiment of the present application.

[0101] Example 6

[0102] An embodiment of the present application provides a computer storage medium.

[0103] The computer-readable storage medium of the embodiment of the present application can adopt any combination of one or more computer-readable media.Computer-readable media can be computer-readable signal media or computer-readable storage media.Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof.More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof.In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0104] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0105] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0106] The computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0107] Note that the above are only preferred embodiments of the present application and the technical principles employed. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present application. The scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for counting people entering and leaving, characterized in that: The method comprises: The signal received by the millimeter-wave radar sensor is subjected to a fast Fourier transform in the fast time dimension; multiple angle information is extracted from the fast Fourier transform result using a super-resolution multiple signal classification algorithm; each angle information is detected to obtain a set of point cloud data in a spherical coordinate system; and a velocity estimate is obtained for each point cloud data set; a 4D super-resolution dense point cloud is obtained based on the spherical coordinates and velocity estimates of each point cloud data set; wherein the millimeter-wave radar sensor is positioned directly above the entrance and exit channel and is placed vertically downward; Performing multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud; Calculating the distances between all existing trajectories and each point cloud data in the set of point cloud data; and associating all existing trajectories with global clusters in the multi-scale clustering result based on the distances between all existing trajectories and each point cloud data; narrowing the point cloud range associated with the trajectory to the global cluster based on the association results of all existing trajectories with the global cluster, calculating the distances between all trajectories and the point cloud data on the associated global cluster, and associating all existing trajectories with the point cloud data on the associated global cluster to obtain a multi-extended target tracking result of the multi-scale clustering result; The people passing through the entry and exit passage are counted based on the multi-extended target tracking results.

2. The method according to claim 1, characterized in that The 4D super-resolution dense point cloud is subjected to multi-scale clustering, including: Performing global scale clustering on the 4D super-resolution dense point cloud to obtain a global scale clustering result of the 4D super-resolution dense point cloud; The 4D super-resolution dense point cloud is subjected to local-scale clustering based on the global-scale clustering result to obtain a local-scale clustering result of the 4D super-resolution dense point cloud.

3. A personnel entry and exit counting device, characterized in that: The device includes: a point cloud generation module, a multi-scale clustering module, a multi-extended target tracking module and a people counting module; wherein, The point cloud generation module is configured to perform a fast Fourier transform (FFT) on the signal received by the millimeter-wave radar sensor in the fast time dimension; extract multiple angle information from the FFT result using a super-resolution multiple signal classification algorithm; detect each angle information to obtain a set of point cloud data in a spherical coordinate system; and obtain a velocity estimate for each point cloud data in the set of point cloud data; and obtain a 4D super-resolution dense point cloud based on the spherical coordinates and velocity estimates of each point cloud data. The millimeter-wave radar sensor is positioned directly above the entrance and exit channel and is placed vertically downward. The multi-scale clustering module is used to perform multi-scale clustering on the 4D super-resolution dense point cloud to obtain a multi-scale clustering result of the 4D super-resolution dense point cloud; The multi-extended target tracking module is configured to calculate the distances between all existing trajectories and each point cloud data in the set of point cloud data; and, based on the distances between all existing trajectories and each point cloud data, associate all existing trajectories with global clusters in the multi-scale clustering result; based on the association results between all existing trajectories and the global clusters, narrow the point cloud range associated with the trajectories to the global clusters, calculate the distances between all trajectories and the point cloud data on the associated global clusters, and associate all existing trajectories with the point cloud data on the associated global clusters to obtain a multi-extended target tracking result of the multi-scale clustering result; The people counting module is used to count people passing through the entry and exit channels based on the multi-extended target tracking results.

4. The device according to claim 3, characterized in that The multi-scale clustering module is specifically used to perform global scale clustering on the 4D super-resolution dense point cloud to obtain a global scale clustering result of the 4D super-resolution dense point cloud; based on the global scale clustering result, the 4D super-resolution dense point cloud is clustered at a local scale to obtain a local scale clustering result of the 4D super-resolution dense point cloud.

5. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for counting people entering and exiting according to any one of claims 1 to 2.

6. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for counting people entering and exiting according to any one of claims 1 to 2 is implemented.

Images