Personnel detector for detecting when person passes through doorway

Abstract

A person detector (1) for detecting when a person passes through a doorway is provided. The person detector (1) comprises: a first image source (11); a second image source (12); a processor (60); and a memory (64) storing instructions (67) that, when executed by the processor, cause the person detector (1) to: receive a first image stream from a first image source (11); determining a confidence index of the single person passing through the doorway based on the first image stream; when the confidence indicator indicates confidence, determining that the individual person has passed the doorway; and when the confidence indicator indicates non-confidence, receiving a second image stream from a second image source (12) and determining, based on the second image stream, how many persons have passed the doorway.

Description

Technical Field

[0001] The present disclosure relates to the field of people detectors, and more particularly to a people detector for detecting when a person passes through a doorway of a door beside which the people detector is mounted. Background Art

[0002] People detectors can be used in many different environments. For example, people detectors can be used to detect the flow of people in, for example, a theme park or a shopping mall. People detectors can also be used to count the number of people in a specific space, such as a commercial or residential property, and this count can then be used to control heating, ventilation, and air conditioning (HVAC) or to track the number of people in a specific space, for example, for fire safety purposes or office utilization metrics.

[0003] There are people detectors that provide good accuracy, such as those based on traditional digital imaging devices, but these people detectors are quite expensive and require a large amount of power.

[0004] Power consumption is very important. If a people detector that is both energy-efficient and accurate can be provided, this makes installation easier because such a people detector can be battery-powered, thus reducing or eliminating the need for cumbersome and expensive wiring. Summary of the Invention

[0005] An object of the present invention is to provide an energy-efficient and accurate people counter.

[0006] According to a first aspect, there is provided a people detector for detecting when a person passes through a doorway of a door beside which the people detector is mounted. The people detector includes: a first image source; a second image source; a processor; and a memory storing instructions that, when executed by the processor, cause the people detector to perform the following operations: receive a first image stream from the first image source; determine a confidence metric for a single person passing through the doorway based on the first image stream; when the confidence metric indicates confidence, determine that a single person has passed through the doorway; and when the confidence metric indicates non-confidence, receive a second image stream from the second image source, the second image stream at least partially overlapping in time with the first image stream, and determine how many people have passed through the doorway based on the second image stream, wherein, compared to people detection based on the first image stream, people detection based on the second image stream consumes more energy, and the second image source 12 has a higher resolution than the first image source 11.

[0007] Each image in the first image stream may include depth data, in which case the instructions for determining the confidence metric include instructions that, when executed by a processor, cause a person detector to perform the following operations: for each image in the first image stream, fit a two-dimensional distribution function and determine the confidence metric based on the degree of similarity between the two-dimensional distribution function and the image.

[0008] The instructions for determining the confidence metric may include instructions that, when executed by a processor, cause a person detector to perform the following operations: for each image in the first image stream, remove pixels depicting a door before fitting the two-dimensional distribution function.

[0009] The first image source may be a time-of-flight camera device, in which case each image in the first image stream includes a pixel matrix, where each pixel includes a depth value.

[0010] The instructions for determining the confidence metric may include instructions that, when executed by a processor, cause a person detector to perform the following operations: for each image in the first image stream, determine the centroid, compare the movement of the centroid relative to a previous image in the first image stream, and determine the confidence metric based on the movement.

[0011] The instructions for determining the confidence metric may include instructions that, when executed by a processor, cause a person detector to determine that the confidence metric indicates non-confidence when the centroid changes direction.

[0012] The instructions for determining the confidence metric may include instructions that, when executed by a processor, cause a person detector to determine that the confidence metric indicates non-confidence when there is more than one depth minimum in the first image stream.

[0013] The first image source may include images from two sensors, in which case the first image stream is based on two sub-streams from the two sensors respectively.

[0014] The person detector may be configured to transition from a dormant state to an active state based on receiving a signal indicating the approach of a person.

[0015] According to a second aspect, a method for detecting when a person passes through a doorway of a door with a person detector mounted beside it is provided. The method is performed by the person detector. The method includes: receiving a first image stream from a first image source; determining a confidence metric for a single person passing through the doorway based on the first image stream; when the confidence metric indicates confidence, determining that the single person has passed through the doorway; and when the confidence metric indicates non-confidence, receiving a second image stream from a second image source, the second image stream at least partially overlapping the first image stream in time, and determining how many people have passed through the doorway based on the second image stream, wherein the person detection based on the second image stream consumes more energy than the person detection based on the first image stream, and the second image source 12 has a higher resolution than the first image source 11.

[0016] Each image in the first image stream may include depth data, in which case determining the confidence metric includes: for each image in the first image stream, fitting a two-dimensional distribution function and determining the confidence metric based on the degree of similarity between the two-dimensional distribution function and the image.

[0017] Determining the confidence metric may include: for each image in the first image stream, removing the pixels depicting the door before fitting the two-dimensional distribution function.

[0018] The first image source may be a time-of-flight camera device, in which case each image in the first image stream includes a pixel matrix, where each pixel includes a depth value.

[0019] Determining the confidence metric may include: for each image in the first image stream, determining the centroid, comparing the movement of the centroid relative to a previous image in the first image stream, and determining the confidence metric based on the movement.

[0020] Determining the confidence metric may include: determining that the confidence metric indicates non-confidence when the centroid changes direction.

[0021] Each image in the first image stream may include depth data, in which case determining the confidence metric includes: determining that the confidence metric indicates non-confidence when there are more than one depth minimum in the first image stream.

[0022] The first image source may include images from two sensors, in which case the first image stream is based on two sub-streams respectively from the two sensors.

[0023] The method may further include: receiving a proximity signal indicating the approach of a person; and transitioning from a dormant state to an active state based on the proximity signal.

[0024] According to a third aspect, there is provided a computer program for detecting when a person passes through a doorway of a door with a person detector installed beside it. The computer program includes computer program code which, when executed on the person detector, causes the person detector to perform the following operations: receive a first image stream from a first image source; determine a confidence metric for a single person passing through the doorway based on the first image stream; when the confidence metric indicates confidence, greater than a threshold, determine that a single person has passed through the doorway; and when the confidence metric indicates non-confidence, less than the threshold, receive a second image stream from a second image source, the second image stream at least partially overlapping in time with the first image stream, and determine how many people have passed through the doorway based on the second image stream, wherein the person detection based on the second image stream consumes more energy compared to the person detection based on the first image stream by a second image source having a higher resolution than the first image source 11.

[0025] According to a fourth aspect, there is provided a computer program product which includes the computer program according to the third aspect and a computer-readable device including a non-transitory memory storing the computer program.

[0026] Generally, unless otherwise clearly defined herein, all terms used in the claims should be interpreted according to their ordinary meaning in the technical field. Unless otherwise clearly stated, all references to "an / the element, device, component, apparatus, step, etc." should be publicly interpreted as referring to at least one instance of the element, device, component, apparatus, step, etc. Unless clearly stated, the steps of any method disclosed herein need not be performed in the exact order disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Aspects and embodiments will now be described by way of example with reference to the accompanying drawings, in which:

[0028] Figure 1 is a schematic diagram showing an environment in which the embodiments proposed herein can be applied;

[0029] Figures 2A to 2B illustrates a schematic image of a situation when a single person passes through Figure 1 the doorway;

[0030] Figures 3A to 3B illustrates a schematic image of a situation when two people pass through Figure 1 the doorway;

[0031] Figure 4A , Figure 4B , Figures 4C to 4D illustrates schematic diagrams of scenarios where the movement of people passing through the doorway is consistent and inconsistent;

[0032] Figure 5 is a flowchart showing a method for detecting when a person passes through a doorway where a person detector is mounted beside the doorway;

[0033] Figure 6 is showing Figure 1 a schematic diagram of the components of the person detector; and

[0034] Figure 7 shows an example of a computer program product including a computer-readable device. DETAILED DESCRIPTION

[0035] Aspects of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown. However, these aspects may be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of all aspects of the invention to those skilled in the art. Throughout the specification, the same reference numerals refer to the same elements.

[0036] The embodiments presented herein provide a method for detecting a person passing through a doorway based on (at least) two image sources. The first image source is a low-power image source with relatively low accuracy. The second image source is a high-power image source with relatively high accuracy. The first image source is used for primary detection, and if it is possible to determine with sufficient confidence that a single person (or no person) has passed based on the first image source, no further person detection is required for that event. On the other hand, if the confidence is low, the second image source is used in secondary detection to detect the number of people passing through the doorway. In this way, the low-power preliminary detection is used when it is accurate enough, while the high-power secondary detection is only used when needed. Compared with the prior art, this provides accurate person detection with less power consumption.

[0037] Figure 1It is a schematic diagram showing an environment in which the embodiments proposed herein can be applied. Access between the first physical space 14 and the second physical space 16 is restricted by a door 15 that can be closed or opened. The door 15 can be any type of door (e.g., revolving door, sliding door, rolling shutter door) and is provided in the doorway 5 between the first physical space 14 and the second physical space 16. In order to control one-way or two-way access between the physical spaces 14 and 16, a locking device can be provided to selectively unlock and lock the door 15. The open state of the door 15, such as open or closed, can be detected by an optional door sensor 18. The door sensor 18 can be provided in the door frame (as shown), in the door 15, using parts in both the door frame and the door, in the hinge, in a door closer (not shown), or in a door opener. The door sensor 18 can detect the open state of the door 15 in any suitable manner, such as using a magnetometer and a magnet, a magnetic rotary position sensor, using resistance, using impedance, using visual sensing, or using an accelerometer and / or a gyroscope.

[0038] A personnel detector 1 is installed beside the door 15, in this case inside the doorway 5. The personnel detector 1 can be a separate device, or the personnel detector can be installed in a door closer, a door opener, in an electronic lock, in a credential reader, or an EAC (electronic access control) button for unlocking or opening the door. The personnel detector 1 includes a first image source 11 and a second image source 12, both of which are used to detect any person passing through the doorway 5. In this way, the personnel detector can detect when a person enters the first physical space 14 from the second physical space 16, and when a person enters the second physical space 16 from the first physical space 14. Optionally, a proximity sensor 14 is provided, which is inside or outside the personnel sensor 1. The proximity sensor can be, for example, an ultrasonic sensor, a passive infrared sensor, or a radar.

[0039] The image sources 11 and 12 can, for example, capture images of the space beside the doorway in a two-dimensional (2D) or three-dimensional (3D) manner, such as based on time-of-flight (ToF) detection, visual imaging (i.e., a camera device), infrared detection, thermal detection, lidar, radar, etc. It should be noted that the term "image" will be interpreted broadly as any captured representation of the local physical environment. Thus, an image can be represented as a 2D array of pixels, a 2D array of pixels with depth data added for each pixel, a point cloud (in 3D space), a set of polygons in 3D space, etc.

[0040] The number of captured images may be relatively small, but still works well for the purpose of person detection. For example, when the capture rate of the first image source is in the range of 15 Hz to 30 Hz, 4 to 5 images from the imaging device may be sufficient.

[0041] One difference between the first image source 11 and the second image source 12 is that the second image source 12 consumes more energy per unit time when in use than the first image source 11. In other words, the first image source 11 is more energy-efficient than the second image source 12. Additionally, compared to the first image source 11, the second image source 12 enables more accurate detection of one or more persons passing through the doorway 5. This improved detection accuracy of the second image source 12 (at the cost of more energy usage) can be achieved, for example, by having a greater resolution than the first image source 11. The increased energy usage of the second image source may be due to the increased need to process more pixels at the higher resolution in the second image source 12. In one embodiment, the first image source 11 and the second image source 12 are based on the same hardware, which provides an image of lower resolution as the first image source 11 and an image of higher resolution as the second image source 12. In one embodiment, the first image source 11 and the second image source 12 are based on the same image capture hardware, but the second image source 12 is enabled based on the lighting source, resulting in the second image source 12 consuming more energy as a result.

[0042] In one embodiment, the first image source 11 provides images of very low resolution, such as 40×30, 8×8, or 4×4 pixels, and the second image source 12 provides images of slightly higher resolution, such as 128×128 or 256×256 pixels. Even though the first image source 11 has such a low resolution, this is sufficient to detect the presence of a person in many cases, as described in more detail below. Additionally, the fewer pixels to be processed, the less power is required for processing to detect a person. When needed, the (relatively) higher resolution second image source 12 is used, consuming more power while enabling more accurate detection of one or more persons passing through the doorway.

[0043] The slightly higher resolution second image source 12 is only employed when the confidence metric indicates a low confidence that the first image source 11 has captured a single person walking through the doorway 5. In this way, when the person counter 1 is relatively certain (e.g., compared to a threshold) based on the first image source 11 that a single person has walked through the doorway 5, this is recorded without having to use the more accurate but more power-consuming second image source 12.

[0044] Using multiple image streams from a first image source 11 and, if necessary, from a second image source 12, the person detector 1 is thus able to detect when a person 7 moves from a first physical space 14 to a second physical space 16 (and to detect when a person 7 moves from the second physical space 16 to the first physical space 14). It is also possible to detect the number of persons passing through. This can be used when an intruder tries to enter a restricted physical space by following a person with legitimate access (a process also known as piggybacking or tailgating).

[0045] The person detector 1 records the detection of the person 7 and the optional direction of movement, and the person detector 1 can transmit this information to an external device 8. The external device 8 can be an alarm system, a building automation system or any other suitable device. The external device 8 can be located at the same location as the person detector 1, or the external device 8 can be located remotely.

[0046] The external device 8 can use the person detection information to determine, for example, how many persons are in the first physical space 14 and / or the second physical space 16. The external device 8 can use this information to control, for example, ventilation, heating, cooling and / or lighting. In addition, this information can be used for office utilization metrics, for example, when utilization is low, enabling employees to plan when to go to the office (instead of working from home). Alternatively or additionally, if an emergency occurs, the person detection information can be used to find out where the persons are located and how many persons need to be evacuated. Alternatively or additionally, this information can be used for people counting, for example, for theme parks, public transport stations and shops.

[0047] When there is only one doorway in a physical space (e.g., a room, an office or a residence), the person detector can thus be used to count how many persons are in the physical space at any point in time. When there are multiple doorways in a physical space, each doorway can be provided with a person detector. This enables the external device to track the number of persons in the physical space by detecting each time a person enters or leaves through any of the doorways.

[0048] The physical space (for which the number of persons is tracked) can be any type of space in which persons are present. Thus, the physical space can be a room, a group of rooms (e.g., an office), a residence, a shop, an arena, a theme park, a part of an external space, or any one or more of any other commercial space, public space or living space.

[0049] Optionally, the physical spaces are hierarchically arranged such that a number of physical spaces are jointly represented by an access area, such as an access area in an office environment. In this way, the external device 8 can track the number of people in each physical space (e.g., a room) and also track the number of people in a combined access area (e.g., an office building). The number of people in each access area (based on people counting) can be compared with the number of people who have entered using the access control system, and thus can be used to detect the occurrence of piggybacking.

[0050] Figures 2A to 2B Schematic images 20, 21 depicting the situation when a single person passes through the doorway 5 are shown. Figure 2A The image 20 in [reference] is captured using the first image source 11, which is an 8×8 ToF sensor, i.e., 64 pixels. For each pixel, the shading indicates the distance from the sensor to the nearest object in that pixel. Darker shading indicates a closer distance compared to lighter shading. In this example, in the image, the doorway is vertically centered and the opening is horizontal.

[0051] First refer to Figure 2A , the darker shading in the middle - left 30 of the image 20 indicates that the object is closer to the sensor. The image 20 is captured when a person walks through the doorway.

[0052] Now refer to Figure 2B , the image 21 is the result of the person detector 1 generating the image 21 by fitting a two - dimensional distribution function 31 to the Figure 2A image in [reference]. It has been found that a two - dimensional distribution function that works well for this purpose is the two - dimensional distribution function. However, any suitable distribution function can be used. Comparing the Figure 2B generated image 21 in [reference] with the Figure 2A captured image 20 in [reference], it can be seen that the images 20, 21 are not the same but are very similar.

[0053] Figures 3A to 3B Schematic images 22, 23 depicting the situation when two people pass through the doorway are shown. Similar to Figure 2A , Figure 3A the image in [reference] is captured using the first image source 11, which is an 8×8 ToF sensor, i.e., 64 pixels (as Figure 2A ). Here too, for each pixel, the shading indicates the distance from the sensor to the nearest object in that pixel, and darker shading indicates a closer distance compared to lighter shading.

[0054] First refer to Figure 3A, the darker shadows 33a in the upper right and 33b in the lower right of the image 22 indicate that the object is closer to the sensor. In this example, the captured image 22 was captured when two people walked through the doorway, corresponding to the two sets of darker shadows 33a, 33b. Thus, the two people are shown as two minima (in terms of distance from the sensor) in the image. These two minima may be two people, and thus this is a clear indication of disbelief in a single person passing through the doorway.

[0055] Now refer to Figure 3B , similar to the image 21 of Figure 2B , the image 23 is the result of the person detector 1 generating the image 23 by fitting a two-dimensional distribution (a Gaussian distribution in this example) function 34 to the Figure 3A image in Figure 3B . Comparing the generated image 23 in Figure 2A with the captured image 22 in

[0056] Figures 2A to 2B and Figures 3A to 3B , it can be seen that the images 22, 23 are completely different here. This difference is due to the fact that the situation of two people does not fit well with the circular two-dimensional Gaussian function (in the plane shown in the image 23). Figures 2A to 2B scene of Figures 3A to 3B , more details are needed to determine whether one or more people have passed through the doorway.

[0057] Figures 4A to 4D depicts a schematic diagram 25 showing scenes where the movement of people passing through the doorway is consistent and inconsistent. Figure 25 is based on the same type of pictures mentioned above with reference to Figures 2A to 2B and Figures 3A to 3B . However, here, each captured image has been evaluated to calculate the centroid 35 as the weighted average position in the x-y dimension based on the proximity to the sensor. In this example, in the image, the doorway is vertically in the middle and the opening is horizontal.

[0058] In Figure 4A , the centroid 35 is located at the position in the lower part of Figure 25. When the centroid 35 is first detected, the fact that the centroid 35 is at the bottommost part of Figure 25 is already an indication of upward movement.

[0059] Following in time afterFigure 4A After Figure 4B in, the centroid 35 is slightly higher in Figure 25, which confirms the hypothesis that the person is walking upward through the doorway.

[0060] For each captured image containing an object, the centroid is calculated, and the object velocity can be approximated by dividing the difference in the centroid of the object between two frames (see Figure 4A and Figure 4B the distance between the centroids 35 therein) by the time difference (i.e., Figure 4A the image of Figure 4B and the time difference between the images of

[0061] In Figure 4B after, there are two scenarios.

[0062] In the first scenario, which is followed in time by Figure 4B after Figure 4C as shown, the centroid 35 is located in the upper part of Figure 25, indicating that the person continues to walk upward through the doorway. This scenario is a continuous movement scenario.

[0063] In the second scenario (an alternative to the first scenario), which is followed in time by Figure 4B after Figure 4D as shown, the centroid 35 is located in the lower part of Figure 25, indicating inconsistent movement of the person. This could be the case where the person returns towards the bottom of Figure 25. However, this inconsistent movement could also be due to a second person entering the field of view of the sensor, which causes the centroid to be pulled downward due to the presence of this second person.

[0064] It can be seen that when inconsistent movement occurs based on the first image source 11, this may be due to multiple people passing through the doorway, and it is not possible to confidently determine the passage of a single person through the doorway based solely on the first image source 11. Therefore, when inconsistent movement is determined, the second image source 12 is employed to more accurately detect people, although at the cost of more energy usage.

[0065] Figure 5It is a flowchart showing when a person passes through the doorway 5 of the door 15 with the person detector 1 installed beside. This method is executed by the person detector 1.

[0066] In the optional step 38 of receiving a proximity signal, the person detector receives a proximity signal indicating the approach of a person, such as receiving a proximity signal from the proximity sensor mentioned above.

[0067] In the optional step 39 of transitioning to an active state, the person detector transitions from a sleep state to an active state based on the proximity signal.

[0068] When steps 38 and 39 are executed, the person detector 1 is usually in a sleep (low power) mode and is thus awakened by a signal from the proximity sensor that a person has been detected approaching within a threshold distance. The proximity sensor can be, for example, an ultrasonic sensor, a passive infrared sensor, or a radar. The proximity sensor can be external to the proximity sensor 1 or form part of the proximity sensor.

[0069] In the step 40 of receiving a first stream, the person detector 1 receives a first image stream from the first image source 11.

[0070] The first image source 11 can be, for example, a time-of-flight camera device sensor. In this case, each image in the first image stream includes a pixel matrix, where each pixel includes a depth value, as described above with reference to Figure 2A and Figure 3A described.

[0071] In one embodiment, the first image source includes images from two sensors. In this case, the first image stream is based on two sub-streams from the two sensors respectively. The two sensors can be used to provide a wider coverage for a wider doorway. In this embodiment, the overlapping field of view between the two sensors can be calculated. Objects detected in the pixels in the overlapping area of the two sensors can be cross-checked, where specific confidence thresholds for the two sensors must be met to classify the passing direction. Or, more complexly, two sensor confidence functions can be deployed to arrive at a combined confidence or non-confidence decision. Then, objects in the non-overlapping fields of view are evaluated independently, thus expanding the spatial coverage of the algorithm.

[0072] In one embodiment, one sensor is angled slightly away from the doorway to avoid two head objects merging when two people walk closely behind each other. Such an embodiment takes advantage of the fact that if one sensor is angled slightly, the likelihood that the frame will be non-conclusive in both sensors simultaneously is smaller. Therefore, the determination can be based on the two streams to achieve higher confidence.

[0073] The two sub-streams can be combined in the time domain during post-processing, for example, using any suitable interpolation process for the combination. This interpolation ensures that the data of the two sub-streams are at the same moment.

[0074] In the confidence metric determination step 42, the person detector 1 determines a confidence metric for a single (i.e., one and only one) person passing through the doorway based on the first image stream. In one embodiment, determining the confidence metric includes: for each image in the first image stream in which an object is detected, fitting a two-dimensional distribution function and determining the confidence metric based on the degree of similarity between the two-dimensional distribution function and the image. Optionally, for each image in the first image stream, the person detector removes the pixels depicting the door 15 before fitting the two-dimensional distribution function. The two-dimensional distribution function can be any convex two-dimensional distribution function, such as a two-dimensional Gaussian function, a von Mises distribution, a gamma distribution, etc.

[0075] For each image composed of a pixel matrix, the fitting of the two-dimensional distribution function can be performed according to the following, where each pixel includes depth data, for example, as Figure 2A and 3A depicted and described above.

[0076] 1: Check if there are a sufficient (greater than a threshold) number of pixels that capture the object, which is indicated by a depth less than a threshold depth. If there are a sufficient number of pixels that capture the object, the process continues. Otherwise, the process ends.

[0077] 2: Remove the pixels that capture the door. The determination of the position of the door in the image can be based on image analysis or on receiving a door status signal indicating the open state of the door. Image analysis can be based on analytical removal of the door according to the configured installation position, or on a machine learning algorithm trained on various door positions with marked expected results.

[0078] When using a door status signal, the signal can be one that can indicate fully open, fully closed, and multiple different opening degrees between fully open and fully closed. For example, the door status signal can be a state in an enumeration of potential states (e.g., an enumeration of opening degrees such as {0, 15, 30, 45, 60, 75, 90}), where the closest match is selected. Alternatively, the door status signal can be a numerical value (an integer or a floating-point number) indicating the opening degree, which is, for example, in the range of [0, 1], where 0 indicates closed and 1 indicates fully open, or in the range of [0, 180], where 0 indicates closed and 180 indicates an opening of 180 degrees. It should be noted that the examples of fully open doors at 90 degrees and 180 degrees are just two examples, and the embodiments proposed herein can be applicable to any degree value of full opening. Alternatively, the door status signal is an analog electrical signal. When the door is a revolving door, the measurement of the degree is mainly applicable. For sliding doors or rolling shutters, other types of indicators are more applicable.

[0079] 3: In the case where x and y are coordinates in an image, x 0 , y 0 is the center position of the Gaussian function (or other distribution function), σ is the standard deviation, and A is the amplitude. x 0 , y 0 , σ, and A are optimized such that the two-dimensional Gaussian function f(x, y) is as close as possible to the image data, measured by the mean squared error. f(x, y) is expressed according to the following formula:

[0080]

[0081] An example of the optimization algorithm is scipy.curve_fit from the Scipy community, and an example of the nonlinear least squares solver is trf (trust region reflective solver). However, most known optimization algorithms will be applicable to this task.

[0082] x 0 , y 0 is restricted between [0, N], where N is the image resolution in the dimension under discussion (8 in the example described herein). The amplitude A should be restricted between [ε, L + ε], where L is the minimum depth recorded in the frame, and ε is the interval length, which should be relatively small. σ can also be restricted around a relatively small interval, for example, corresponding to the variation in the shape of a person's head.

[0083] 4: Calculate the normalized cross-correlation between the 2D Gaussian and the image, and this normalized cross-correlation can be used as a confidence metric for a single person passing through the doorway.

[0084] In one embodiment, for each image in the first image stream, the person detector 1 determines the centroid, compares the movement of the centroid relative to the previous image in the first image stream, and determines a confidence metric for a person passing through the doorway based on the movement. More specifically, when the centroid changes direction, the confidence metric can be determined by indicating non-confidence. The change in direction may need to be strong enough to be considered inconsistent.

[0085] The process will now be described in more detail.

[0086] 1: Save each image with a relevant set of objects (pixels with low depth and close to each other) to memory.

[0087] 2: Calculate the intersection over union value between the new image and the previous image. If the value is higher than a specific threshold, link the images to each other and save them as a tracked object.

[0088] 3: Calculate the centroid of each image (weighted average position in the x-y dimension). If the centroid of the first image of the tracked object is in the topmost row of the image matrix, assume a downward passing direction, and if the centroid is in the bottom row, assume an upward passing direction.

[0089] 4: Calculate the object movement by comparing the centroid of the latest image with the centroid of the previous image. If the movement is inconsistent with the assumed passing direction, the movement is considered inconsistent.

[0090] 5: For the movement detected in the image stream to be classified as inconsistent, the condition is that the movement of N or more images is inconsistent with the previous image. In this embodiment, the confidence metric for a person passing through the doorway can be the number of images with inconsistent movement.

[0091] In one embodiment, when each image in the first image stream includes depth data, when there are more than one depth minimum in the first image stream, the confidence metric is determined to indicate non-confidence. Then, each depth minimum is considered a person. Since two people are detected next, the confidence metric that only one person passes through the doorway is set to non-confidence.

[0092] In one embodiment, using the detection with the first image stream is used to determine when two people may be traveling in opposite directions and meet at the doorway, in which case the confidence metric is determined to indicate non-confidence.

[0093] In the conditional confidence step 44, the person detector 1 evaluates the confidence metric determined in step 42 according to a threshold. For example, when two persons are very close to passing (normal passing or passing while carried), if a person is carrying a large backpack or a rolled-up bag, or if a person is holding an umbrella (a process usually used to deceive overhead detectors based on camera devices), the confidence may be low. Therefore, the reasons for the reduced confidence may be legitimate or illegitimate.

[0094] When a two-dimensional Gaussian function is fitted and the normalized cross-correlation between the two-dimensional Gaussian function and the image is used as the confidence metric, an example of a suitable threshold is 0.6. Therefore, when the cross-correlation is greater than the threshold 0.6, the confidence metric indicates that it is confident that a single person passes through the doorway. Otherwise, it is not confident. Other values are also possible and can be adjusted based on real-life performance.

[0095] When inconsistent movement is used to determine the confidence of a single person passing through the doorway, the number of images with inconsistent movement can be used to determine the confidence. For example, when the number of images with inconsistent movement is greater than the threshold, it is not confident that a single person walks through the doorway. Otherwise, it is confident.

[0096] If the confidence metric indicates confidence that a single person passes through the doorway (e.g., based on the thresholds mentioned above), the method proceeds to the determine single person step 46. Otherwise, the method proceeds to the evaluate second stream step 48. Confidence is interpreted here as relative (not absolute) confidence, i.e., confidence is based on the confidence metric being on one side of a predetermined threshold.

[0097] In the determine single person step 46, the person detector 1 determines that a single person has passed through the doorway.

[0098] In the evaluate second stream step 48, the person detector 1 receives a second image stream from the second image source 12 11. The second image stream overlaps at least partially in time with the first image stream. In addition, the person detector 1 determines how many persons have passed through the doorway in a secondary detection based on the second image stream. Since the second image source 12 has a higher resolution than the first image source 11, the person detection based on the second image stream consumes more energy than the person detection based on the first image stream. The person detection of the first image stream includes the determine confidence metric step 42. The person detection of the second image stream includes the evaluate second stream step 48.

[0099] The secondary detection can be located locally at the person detector or use a remote server. The secondary detection can be based on machine learning and / or analytical rules.

[0100] It should be noted that the reception of the second stream can end before the first stream. In addition, the second stream can have a lower sampling rate than the first stream.

[0101] When boarding is determined or suspected, it can be reported to an external device.

[0102] This method can be repeated any number of times to track the number of people passing through the doorway over time.

[0103] Optionally, after a period of inactivity, e.g., defined as no people being detected near the people sensor 1 within a threshold amount of time, the people sensor 1 enters a sleep state to conserve power.

[0104] Using the embodiments proposed herein, a very energy-efficient people detector is provided. This enables the people detector to be battery-powered, which significantly simplifies installation. In addition, the embodiments proposed herein can be implemented using a simple, relatively low-resolution image sensor, thereby saving cost and processing power.

[0105] More specifically, since the second image stream is received only when the confidence metric indicates non-confidence and the second image source has a higher resolution than the first image source, both accuracy and energy efficiency are achieved simultaneously. The low-power preliminary detection is used when it is accurate enough, while the high-power secondary detection is used only when needed. Thus, accurate and energy-efficient people detection is provided.

[0106] Figure 6 is a schematic diagram of the components of the people detector 1 shown Figure 1 using any combination of one or more of a suitable central processing unit (CPU), graphics processing unit (GPU), neural processing unit (NPU), multiprocessor, microcontroller, digital signal processor (DSP), etc. capable of executing the software instructions 67 stored in the memory 64, so the memory 64 can be a computer program product. Alternatively, the processor 60 can be implemented using an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The processor 60 can be configured to execute the method described above with reference to Figure 5 described.

[0107] The memory 64 can be any combination of random access memory (RAM) and / or read only memory (ROM). The memory 64 also includes a permanent storage device, which can be, for example, any single memory or combination of a magnetic memory, an optical memory, a solid state memory, or even a remotely installed memory.

[0108] A data memory 66 is also provided for reading and / or storing data during the execution of the software instructions in the processor 60. The data memory 66 can be any combination of RAM and / or ROM.

[0109] The personnel detector 1 further includes an I / O interface 62 for communicating with external and / or internal entities. The I / O interface 62 includes connections to a first image source 11 and a second image source 12. The first image source 11 and the second image source 12 may be external to the personnel detector 1 or may form part of the personnel detector 1, as Figure 1 shown.

[0110] Other components of the personnel detector 1 are omitted so as not to obscure the concepts presented herein.

[0111] Figure 7 An example of a computer program product 90 including a computer-readable device is shown. On this computer-readable device, a computer program 91 can be stored, which can cause a processor to execute the method according to the embodiments described herein. In this example, the computer program product is in the form of a removable solid-state memory such as a Universal Serial Bus (USB) drive. As described above, the computer program product can also be embodied in the memory of a device such as Figure 6 the computer program product 64. Although the computer program 91 is schematically shown here as part of a removable solid-state memory, the computer program can be stored in any manner suitable for the computer program product, which is another type of removable solid-state memory or an optical disc such as a CD (Compact Disc), DVD (Digital Versatile Disc), or Blu-ray Disc.

[0112] A list of embodiments enumerated in Roman numerals is now presented here from another perspective.

[0113] i. A personnel detector for detecting when a person passes through the doorway of a door beside which the personnel detector is installed, the personnel detector comprising:

[0114] A first image source;

[0115] A second image source;

[0116] A processor; and

[0117] A memory storing instructions which, when executed by the processor, cause the personnel detector to perform the following operations:

[0118] Receive a first image stream from the first image source;

[0119] Determine a confidence metric for a single person passing through the doorway based on the first image stream;

[0120] When the confidence metric indicates confidence, determine that a single person has passed through the doorway; and

[0121] When the confidence metric indicates non-confidence, receive a second image stream from the second image source, the second image stream overlapping the first image stream at least partially in time, and determine how many people have passed through the doorway based on the second image stream, wherein people detection based on the second image stream consumes more energy than people detection based on the first image stream.

[0122] ii. The people detector according to embodiment i, wherein each image in the first image stream includes depth data, and wherein the instructions for determining the confidence metric include instructions that, when executed by the processor, cause the people detector to perform the following operations: for each image in the first image stream, fit a two-dimensional distribution function and determine the confidence metric based on the degree of similarity between the two-dimensional distribution function and the image.

[0123] iii. The people detector according to embodiment ii, wherein the instructions for determining the confidence metric include instructions that, when executed by the processor, cause the people detector to perform the following operations: for each image in the first image stream, remove the pixels depicting the door before fitting the two-dimensional distribution function.

[0124] iv. The people detector according to embodiment ii or iii, wherein the first image source is a time-of-flight imaging device, and wherein each image in the first image stream includes a pixel matrix, wherein each pixel includes a depth value.

[0125] v. The people detector according to any of the preceding embodiments, wherein the instructions for determining the confidence metric include instructions that, when executed by the processor, cause the people detector to perform the following operations: for each image in the first image stream, determine the centroid, compare the movement of the centroid relative to a previous image in the first image stream, and determine the confidence metric based on the movement.

[0126] vi. The people detector according to embodiment v, wherein the instructions for determining the confidence metric include instructions that, when executed by the processor, cause the people detector to perform the following operations: determine that the confidence metric indicates non-confidence when the centroid changes direction.

[0127] vii. The people detector according to any of the preceding embodiments, wherein the instructions for determining the confidence metric include instructions that, when executed by the processor, cause the people detector to perform the following operations: determine that the confidence metric indicates non-confidence when there are more than one depth minimum in the first image stream.

[0128] viii. The personnel detector according to any one of the foregoing embodiments, wherein the first image source includes images from two sensors, and wherein the first image stream is based on two sub-streams respectively from the two sensors.

[0129] ix. The personnel detector according to any one of the foregoing embodiments, wherein the personnel detector is configured to transition from a sleep state to an active state based on a received signal indicating the approach of a person.

[0130] x. A method for detecting when a person passes through a doorway of a door with a personnel detector mounted beside it, the method being performed by the personnel detector, the method comprising:

[0131] Receiving a first image stream from a first image source;

[0132] Determining a confidence metric for a single person passing through the doorway based on the first image stream;

[0133] When the confidence metric indicates confidence, determining that a single person has passed through the doorway; and

[0134] When the confidence metric indicates non-confidence, receiving a second image stream from the second image source, the second image stream at least partially overlapping in time with the first image stream, and determining how many people have passed through the doorway based on the second image stream, wherein, compared with the personnel detection based on the first image stream, the personnel detection based on the second image stream consumes more energy.

[0135] xi. The method according to embodiment x, wherein each image in the first image stream includes depth data, and

[0136] wherein determining the confidence metric includes: for each image in the first image stream, fitting a two-dimensional distribution function, and determining the confidence metric based on the degree of similarity between the two-dimensional distribution function and the image.

[0137] xii. The method according to embodiment xi, wherein determining the confidence metric includes: for each image in the first image stream, removing the pixels depicting the door before fitting the two-dimensional distribution function.

[0138] xiii. The method according to embodiment xi or xii, wherein the first image source is a time-of-flight imaging device, and wherein each image in the first image stream includes a pixel matrix, wherein each pixel includes a depth value.

[0139] xiv. The method according to any one of embodiments x to xiii, wherein determining the confidence metric includes: for each image in the first image stream, determining a centroid, comparing the movement of the centroid relative to a previous image in the first image stream, and determining the confidence metric based on the movement.

[0140] xv. The method according to embodiment xiv, wherein determining the confidence metric includes: determining that the confidence metric indicates non-confidence when the centroid changes direction.

[0141] xvi. The method according to any one of embodiments x to xv, wherein each image in the first image stream includes depth data, and wherein determining the confidence metric includes: determining that the confidence metric indicates non-confidence when there are more than one depth minimums in the first image stream.

[0142] xvii. The method according to any one of embodiments x to xvi, wherein the first image source includes images from two sensors, and wherein the first image stream is based on two sub-streams respectively from the two sensors.

[0143] xvii. The method according to any one of embodiments x to xvii, further comprising:

[0144] receiving a proximity signal indicating the proximity of a person; and

[0145] transitioning from a sleep state to an active state based on the proximity signal.

[0146] xix. A computer program for detecting when a person passes through a doorway of a door with a person detector mounted beside it, the computer program including computer program code which, when executed on the person detector, causes the person detector to perform the following operations:

[0147] receiving a first image stream from a first image source;

[0148] determining a confidence metric for a single person passing through the doorway based on the first image stream;

[0149] determining that a single person has passed through the doorway when the confidence metric indicates confidence and is greater than a threshold; and

[0150] when the confidence metric indicates non-confidence and is less than the threshold, receiving a second image stream from a second image source, the second image stream at least partially overlapping in time with the first image stream, and determining how many people have passed through the doorway based on the second image stream, wherein the person detection based on the second image stream consumes more energy than the person detection based on the first image stream.

[0151] xx. A computer program product, the computer program product comprising a computer program according to Embodiment XIX and a computer-readable device, the computer-readable device comprising a non-transitory memory in which the computer program is stored.

[0152] Aspects of the present disclosure have been described above mainly with reference to several embodiments. However, as will be readily understood by those skilled in the art, other embodiments are equally possible within the scope of the invention as defined by the appended patent claims, in addition to the embodiments disclosed above. Therefore, while aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The aspects and embodiments disclosed herein are for illustrative purposes and are not intended to be limiting, and the true scope and spirit are indicated by the appended claims.

Claims

1. A personnel detector (1) for detecting when a person passes through the doorway (5) of a door (15) beside which the personnel detector (1) is installed, the personnel detector (1) comprises: a first image source (11); a second image source (12); a processor (60); and a memory (64) storing instructions (67) which, when executed by the processor, cause the personnel detector (1) to perform the following operations: receive a first image stream from the first image source (11); determine a confidence metric for a single person passing through the doorway based on the first image stream; when the confidence metric indicates confidence, determine that a single person has passed through the doorway; and when the confidence metric indicates non - confidence, receive a second image stream from the second image source (12), the second image stream at least partially overlapping the first image stream in time, and determine how many people have passed through the doorway based on the second image stream, wherein, compared to the personnel detection based on the first image stream, the personnel detection based on the second image stream consumes more energy, and the second image source (12) has a higher resolution than the first image source (11).

2. The personnel detector (1) according to claim 1, wherein, each image in the first image stream includes depth data, and wherein the instructions for determining the confidence metric include instructions (67) which, when executed by the processor, cause the personnel detector (1) to perform the following operations: for each image in the first image stream, fit a two - dimensional distribution function and determine the confidence metric based on the degree of similarity between the two - dimensional distribution function and the image.

3. The personnel detector (1) according to claim 2, wherein, the instructions for determining the confidence metric include instructions (67) which, when executed by the processor, cause the personnel detector (1) to perform the following operations: for each image in the first image stream, remove the pixels depicting the door (15) before fitting the two - dimensional distribution function.

4. The personnel detector (1) according to claim 2 or 3, wherein, the first image source (11) is a time - of - flight imaging device, and each image in the first image stream includes a pixel matrix, wherein each pixel includes a depth value.

5. The personnel detector (1) according to any one of the preceding claims, wherein, the instructions for determining the confidence metric include instructions (67) which, when executed by the processor, cause the personnel detector (1) to perform the following operations: for each image in the first image stream, determine the centroid, compare the movement of the centroid relative to the previous image in the first image stream, and determine the confidence metric based on the movement.

6. The personnel detector (1) according to claim 5, wherein, the instructions for determining the confidence metric include instructions (67) which, when executed by the processor, cause the personnel detector (1) to perform the following operations: determine that the confidence metric indicates non - confidence when the centroid changes direction.

7. The person detector (1) according to any one of the preceding claims, wherein, the instructions for determining the confidence metric include instructions (67) that, when executed by the processor, cause the person detector (1) to perform the following operations: determining that the confidence metric indicates non-confidence when there are more than one depth minimum in the first image stream.

8. The person detector (1) according to any one of the preceding claims, wherein, the first image source includes images from two sensors, and wherein the first image stream is based on two sub-streams respectively from the two sensors.

9. The person detector (1) according to any one of the preceding claims, wherein, the person detector (1) is configured to transition from a sleep state to an active state based on receiving a signal indicating the approach of a person.

10. A method for detecting when a person passes through a doorway (5) of a door (15) beside which a person detector (1) is installed, the method being performed by the person detector (1), the method comprising: receiving (40) a first image stream from a first image source (11); determining (42) a confidence metric for a single person passing through the doorway based on the first image stream; when the confidence metric indicates confidence, determining (46) that a single person has passed through the doorway; and when the confidence metric indicates non-confidence, receiving (48) a second image stream from the second image source (12), the second image stream at least partially overlapping in time with the first image stream, and determining based on the second image stream how many people have passed through the doorway, wherein, compared with the person detection based on the first image stream, the person detection based on the second image stream consumes more energy, and the second image source (12) has a higher resolution than the first image source (11).

11. The method according to claim 10, wherein, each image in the first image stream includes depth data, and wherein determining (42) the confidence metric includes: for each image in the first image stream, fitting a two-dimensional distribution function and determining the confidence metric based on the degree of similarity between the two-dimensional distribution function and the image.

12. The method according to claim 11, wherein, determining (42) the confidence metric includes: for each image in the first image stream, removing the pixels depicting the door (15) before fitting the two-dimensional distribution function.

13. The method according to claim 11 or 12, wherein, the first image source (11) is a time-of-flight imaging device, and wherein each image in the first image stream includes a pixel matrix, and each pixel includes a depth value.

14. The method according to any one of claims 10 to 13, wherein, determining (42) the confidence metric includes: for each image in the first image stream, determining the centroid, comparing the movement of the centroid relative to the previous image in the first image stream, and determining the confidence metric based on the movement.

15. The method according to claim 14, wherein, Determining (42) the confidence metric includes: determining that the confidence metric indicates non-confidence when the centroid changes direction.

16. The method according to any one of claims 10 to 15, wherein, each image in the first image stream includes depth data, and wherein determining (42) the confidence metric includes: determining that the confidence metric indicates non-confidence when there are more than one depth minimum in the first image stream.

17. The method according to any one of claims 10 to 16, wherein, the first image source includes images from two sensors, and wherein the first image stream is based on two sub-streams respectively from the two sensors.

18. The method according to any one of claims 10 to 17, further comprises: receiving (38) a proximity signal indicating that a person is approaching; and transitioning (39) from a dormant state to an active state based on the proximity signal.

19. A computer program (67, 91) for detecting when a person passes through a doorway (5) of a door (15) beside which a person detector (1) is installed, the computer program comprising computer program code which, when executed on the person detector (1), causes the person detector (1) to perform the following operations: receiving a first image stream from a first image source (11); determining a confidence metric for a single person passing through the doorway based on the first image stream; determining that a single person has passed through the doorway when the confidence metric indicates confidence and is greater than a threshold; and when the confidence metric indicates non-confidence and is less than the threshold, receiving a second image stream from a second image source (12), the second image stream at least partially overlapping in time with the first image stream, and determining based on the second image stream how many people have passed through the doorway, wherein, compared with the person detection based on the first image stream, the person detection based on the second image stream consumes more energy, and the second image source (12) has a higher resolution than the first image source (11).

20. A computer program product (64, 90), the computer program product (64, 90) comprising the computer program according to claim 19 and a computer-readable device, the computer-readable device comprising a non-transitory memory in which the computer program is stored.

Images