Systems and methods for detecting, tracking, and counting human subjects of interest using improved height calculations

Abstract

The present invention discloses a technique for evaluating height data. Height values ​​of a tracked object are obtained within a specified time period. A cluster including at least one of the height values ​​is generated. A cluster having a maximum number of height values ​​is identified. A height of the tracked object is estimated based on the identified cluster.

Description

Background of the Invention

[0001] Field of the Invention

[0002] The present invention relates generally to the field of object detection, tracking and counting. Specifically, the present invention is a computer-implemented detection and tracking system and process for detecting and tracking human objects of interest appearing in camera images taken, for example, at one or more entrances to a facility and counting the number of human objects of interest that enter or leave the facility within a given time period. Background Art

[0004] Traditionally, various methods for detecting and counting the passage of objects have been proposed. U.S. Patent No. 7,161,482 describes an integrated electronic article surveillance (EAS) and people counting system. The EAS component establishes an interrogation zone by an antenna positioned adjacent to the interrogation zone at the exit point of the protected area. The people counting component includes: a person detection device, the person detection device is used to detect the passage of a person through an associated channel and provide a person detection signal; and another person detection device, the other person detection device is placed at a predefined distance from the first device and is configured to detect another person detection signal. Then, in response to these signals, the two signals are processed into an output representing the direction of travel.

[0005] Basically, there are two types of systems that use video images to locate and track human objects of interest. One type uses a monocular video stream or image sequence to extract, identify, and track the object of interest. The other type uses two or more video sensors to derive a range map or height map from multiple intensity images and uses the range map or height map as the primary data source.

[0006] In monocular systems, objects of interest are detected and tracked by applying background subtraction or by adaptive template matching or by contour tracking. The main problem with methods using background subtraction is the presence of background clutter, which adversely affects the robustness and reliability of system performance. Another problem is that the background update rate is difficult to adjust in practical applications. The problems with methods using adaptive template matching are:

[0007] 1) Object detection tends to drift away from the true location of the object, or fixate on strong features in the background; and

[0008] 2) Detection is susceptible to occlusion. Methods using contour tracking have difficulty in overcoming the degradation caused by intensity gradients in the background near the object's outline. In addition, all of the previously mentioned methods are susceptible to changes in lighting conditions, shadows, and sunlight.

[0009] In stereo or multi-sensor systems, the intensity images captured by the sensors are converted into distance maps or height maps, and the conversion is not affected by adverse factors such as changing lighting conditions, strong shadows or sunlight.

[0010] Therefore, the performance of stereo systems remains very robust and reliable in the presence of adverse factors such as unfavorable lighting conditions. In addition, it is easier to segment, detect and track objects using distance or height information than using intensity information.

[0011] Most state-of-the-art stereo systems use range background subtraction to detect objects of interest. Range background subtraction suffers from the same problems as monocular background subtraction methods, such as background clutter and difficulty distinguishing between multiple closely located objects.

[0012] U.S. Patent No. 6,771,818 describes a system and process for identifying and locating persons or objects of interest by selectively clustering blobs to generate "candidate blob clusters" within a scene and comparing the blob clusters to a model representing persons and objects of interest in the scene. Comparison of the candidate blob clusters to the model identifies the blob clusters as the closest match or matches to the model. Continuous real-time depth images may be captured and analyzed in real time to provide continuous identification and location of persons or objects as a function of time.

[0013] U.S. Patent Nos. 6,952,496 and 7,092,566 relate to a system and process for best identifying each of a set of persons and objects in an image of a scene using a color image, a color histogram, a technique for compensating for variations, and a match quality summation method. The image is segmented to extract regions that may correspond to persons and objects of interest and a histogram is calculated for each extracted region. The histogram is compared to a pre-computed model histogram, and if the similarity exceeds a specified threshold, the histogram is designated as corresponding to a person or object. The designated histogram may also be stored as an additional model histogram.

[0014] U.S. Patent No. 7,176,441 describes a counting system for counting the number of people passing through a monitoring line set in the width direction of a path. A laser is installed to illuminate the monitoring line with a slit ray and an image capture device is deployed to photograph an area including the monitoring line. When a person passes through the monitoring line, the slit ray is interrupted on the monitoring line, and the number of people passing through is counted based on one-dimensional data generated from an image obtained by photographing.

[0015] Despite all the prior art in this field, no invention has yet developed a technology that enables unobtrusive detection and tracking of moving human subjects, requires low budget and maintenance while providing accurate traffic counting results and having the ability to differentiate between incoming and outgoing traffic, moving and stationary objects, and between objects at different heights. Therefore, the main object of the present invention is to provide an unobtrusive traffic detection, tracking and counting system that involves low cost, easy and low maintenance, high speed processing, and is able to provide time stamped results that can be further analyzed.

[0016] Additionally, people counting systems typically create anonymous traffic counts. However, in retail traffic monitoring, this may not be sufficient. For example, some situations may require store employees to accompany customers through access points monitored by object tracking and counting systems, such as fitting rooms. In these cases, existing systems are unable to track and count employees and customers separately. The present invention will address this shortcoming. Summary of the invention

[0017] According to one aspect of the present invention, a method for counting and tracking defined objects includes the steps of receiving subset data with a data capture device, wherein the subset data is associated with defined objects and includes a unique identifier, entry time, exit time, and location data for each defined object. The method may further include the steps of receiving the subset data at a counting system, counting the defined objects, tracking the defined objects, associating the locations of the defined objects with predefined areas, and / or generating path data by plotting the X and Y coordinates of the defined objects within the predefined area over consecutive time periods.

[0018] In some embodiments, the method may further include the step of receiving location data at the data capture device, wherein the location data is received from tracking technology that detects cellular signals emitted from one or more mobile phones or signals emitted from a membership card, employee badge, train ticket or airline ticket, rental car key, hotel key, store-sponsored credit or debit card, or loyalty rewards card with an RFID chip.

[0019] In some embodiments, the cellular signal comprises a T-IMSI signal, a CDMA signal, or a Wi-Fi signal.

[0020] In some embodiments, the method may further include the step of receiving data regarding physical characteristics of the object from another independent system.

[0021] In some embodiments, the another independent system may be selected from the group consisting of a point of sale system, a loyalty rewards system, a point of sale trigger message, and a Mechanical Turk.

[0022] In some embodiments, the method may further include the steps of converting the subset data into sequence records and creating a sequence array of all of the sequence records, wherein the sequence record may include: (a) a unique ID, which is an unsigned integer associated with a mobile phone, a telephone number associated with the mobile phone, or any other unique number, character, or combination thereof associated with an object, (b) a start time, which may include information indicating the time when the object was first detected within the coverage area, (c) an end time, which may include information indicating the time when the object was last detected within the coverage area of ​​the data capture device, and / or (d) an array of references to all tracks that overlap with a particular sequence record.

[0023] In some embodiments, the method may further include the step of determining a stay time of the object within the predefined area by subtracting an end time of the predefined area from a start time of the predefined area.

[0024] In some embodiments, the method may further include the step of using the Z coordinate to further track objects within the predefined area defined by the plurality of floors.

[0025] In some embodiments, the method may further include the step of generating a report using the subset data, the report displaying at least one of the number of objects within a predefined area, the number of objects within the predefined area during a specific time period, the number of predefined areas visited by objects, or the dwell time in one or more predefined areas.

[0026] In some embodiments, the method may further include the steps of aggregating the most common paths taken by objects using at least the path data and the subset data and correlating the path data information with dwell times.

[0027] In some embodiments, the method may further include the step of generating a report showing at least one of: (a) the most common paths taken by objects within a store, including corresponding dwell times, (b) changes in shopping patterns by time period or season, and / or (c) traffic patterns used by store security or HVAC systems as resources are increased or decreased at specific times.

[0028] In some embodiments, the method may further include the step of generating a conversion rate by: (a) loading transaction data associated with the transactions conducted, (b) loading traffic data including traffic counts and categorized by time periods, and / or (c) dividing the transactions by the traffic counts for the time period.

[0029] In some embodiments, the transaction data may include sales volume, quantity of items purchased, specific items purchased, date and time of the transaction, the registration form used for the transaction, and the sales partner that completed the transaction.

[0030] In some embodiments, the method may further include the step of generating a report showing a comparison between purchasers and non-purchasers based on at least one of dwell time, predefined area, or time period.

[0031] In accordance with these and other objects which will become apparent hereinafter, the invention will be described with particular reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic perspective view of a facility in which the system of the present invention is installed;

[0033] Figure 2 is a diagram showing an image capture device connected to an exemplary counting system of the present invention;

[0034] Figure 3 is a diagram illustrating a sequence of converting one or more stereo image pairs captured by the system of the present invention into a height map that is analyzed to track and count human subjects;

[0035] Figure 4 is a flow chart describing the processing flow of performing human object detection, tracking and counting according to the system of the present invention;

[0036] Figure 5 is a flow chart describing the processing flow of object tracking;

[0037] FIG. 6A to FIG. 6B is a flow chart describing the processing flow of the tracking analysis;

[0038] FIG. 7A to FIG. 7B is the first part of a flowchart describing the processing flow of suboptimal positioning of unpaired trajectories;

[0039] FIG. 8A to FIG. 8B is the second part of the flowchart of FIG7 , describing the processing flow of suboptimal positioning of unpaired trajectories;

[0040] FIG. 9A to FIG. 9B is a flow chart describing the processing flow of the second pass matching of trajectories to object detection;

[0041] Fig.10 is a flow chart describing the process flow of trajectory update or creation;

[0042] Fig.11 is a flow chart describing the processing flow of trajectory merging;

[0043] Fig.12 is a flow chart describing the processing flow of trajectory updating;

[0044] Fig.13 is a conceptual diagram showing an example scenario in which the counting system could be deployed;

[0045] Fig.14 is relative to Fig.13 A simplified diagram of an example embodiment of a stereoscopic optical sensor unit is described;

[0046] Fig.15 is relative to Fig.13 A simple block diagram of an example embodiment of an arrangement of components of a stereoscopic optical sensor unit is described;

[0047] Fig.16 is relative to Fig.13 A simple block diagram of the arrangement of components of the counting system described;

[0048] Fig.17 is an example conceptual diagram of an example embodiment of a counting zone defined for a counting system;

[0049] Fig.18 This is an example concept diagram of a grid view of 8 connected spots;

[0050] Fig.19 is an example conceptual diagram of a top view of spots indicating individuals; and

[0051] Fig. 20 and Fig.21 is used to determine relative to Fig.13 and Fig.14 A flow chart describing a method of height calculation in a stereoscopic optical counting system. DETAILED DESCRIPTION

[0052] The present specific embodiments are presented in terms of programs, data structures or processes executed on a computer or computer network. The software programs implemented by the system can be written in languages ​​such as JAVA, C, C++, C#, assembly language, Python, PHP or HTML. However, those skilled in the art will appreciate that other languages ​​can be used in place of the aforementioned languages ​​or in combination with the aforementioned languages.

[0053] 1. System Components

[0054] refer to Figure 1 , Figure 2 and Figure 3, the present invention is a system 10 that includes at least one image capture device 20 electronically or wirelessly connected to a counting system 30. In the illustrated embodiment, the at least one image capture device 20 is mounted above one or more entrances 21 of a facility 23 for capturing images from the one or more entrances 21. Facilities such as malls or stores with wide entrances typically require more than one image capture device to fully cover the entrance. The area captured by the image capture device 20 is a field of view 44. Each image, along with the time at which the image was captured, is a frame 48 ( Figure 3 ).

[0055] Typically, the image capture device includes a video sensor 46 having two or more video sensors ( Figure 2 ), which allows the camera to simulate human binocular vision. A pair of stereo images includes frames 48 taken by each video sensor 46 of the camera. A height map 56 is then constructed from the pair of stereo images by calculations involving finding corresponding pixels in the corrected frames 52, 53 of the stereo image pair.

[0056] Gate zone 84 is the area of ​​height map 56 that marks the start of the incoming trajectory and the end of the outgoing trajectory. Interior zone 86 is the area that marks the end of the incoming trajectory and the start of the outgoing trajectory. Dead zone 90 is the area of ​​field of view 44 that is not processed by counting system 30.

[0057] Video sensor 46( Figure 2 ) receives photons through a lens, and the photons react to electrons in the image capture device 20 and form a light image. The image capture device 20 then converts the light image into a digital signal, through which the device 20 obtains a digital raw frame 48 ( Figure 3 ). A pixel is a single point in the raw frame 48. The raw frame 48 typically includes hundreds of thousands or millions of pixels arranged in rows and columns.

[0058] Examples of video sensors 46 for use in the present invention include CMOS (Complementary Metal Oxide Semiconductor) sensors and / or CCD (Charge Coupled Device) sensors. However, the type of video sensor 46 should not be considered limiting, and any video sensor 46 compatible with the present system may be employed.

[0059] The counting system 30 includes three main components: (I) a boot loader 32; (2) a system management and communication component 34; and (3) a counting component 36.

[0060] The boot loader 32 executes at system power-up and loads the main application into memory 38 for execution.

[0061] The system management and communication component 34 includes a task scheduler, a database interface, a recording function, and a TCP / IP or PPP communication protocol. The database interface includes a module for pushing and storing data generated by the counting component 36 to a database at a remote site. The recording function provides operations such as writing user-defined events to a database, sending emails, and video recording.

[0062] The counting assembly 36 is a key component of the system 10 and is described in further detail below.

[0063] 2. Counting components.

[0064] In the illustrated embodiment of the present invention, at least one image capture device 20 and a counting system 30 are integrated into a single image capture and processing device. The single image capture and processing device can be installed anywhere above one or more entrances of a facility 23. Data output from the single image capture and processing device can be transmitted to a database through a system management and communication component 34 for storage and further analysis.

[0065] Figure 4 is a diagram showing the process flow of the counting component 36. The process is: (1) obtain raw frame (box 100); (2) rectification (box 102); (3) disparity map generation (box 104); (4) height map generation (box 106); (5) object detection (box 108); and (6) object tracking (box 110).

[0066] refer to Figures 1 to 4 In block 100, the image capture device 20 obtains raw image frames 48 (eg, one frame per second) of the field of view 44 from the video sensor 46. Figure 3 ). Each pixel in raw frame 48 records the color and light intensity of a location in field of view 44. When image capture device 20 takes a snapshot, each video sensor 46 of device 20 simultaneously produces a different raw frame 48. One or more pairs of raw frames 48 taken simultaneously are then used to generate a height map 56 of field of view 44, as will be described.

[0067] When multiple image capture devices 20 are used, the trajectories 88 generated by each image capture device 20 are merged before proceeding to block 102 .

[0068] Block 102 uses calibration data of a stereo camera (not shown) stored in the image capture device 20 to correct the raw stereo frame 48. The correction operation corrects the effects of lens distortion on the raw frame 48. The calibration data includes the optical center of each sensor, lens distortion information, focal length, and the relative pose of one sensor with respect to another sensor. After correction, straight lines in the real world that have been distorted into curves in the raw stereo frame 48 are corrected and restored to straight lines. The frames resulting from the correction are referred to as corrected frames 52, 53 ( Figure 3 ).

[0069] Block 104 creates a disparity map 50 from each pair of corrected frames 52, 53 ( Figure 3 ). The disparity map 50 is an image map where each pixel includes a disparity value. The term disparity was originally used to describe the 2-dimensional vector between the positions of corresponding features seen by the left and right eyes. The corrected frames 52, 53 in the pair are compared to each other to match features. The disparity is calculated as the difference between the positions of the same feature in frame 52 and frame 53.

[0070] Block 106 converts disparity map 50 into height map 56. Each pixel of height map 56 includes a height value and an xy coordinate, where the height value is represented by the maximum ground height of all points at the same location in field of view 44. In the remainder of the description, height map 56 is sometimes referred to as a frame.

[0071] 2.1 Object Detection

[0072] Object detection (block 108) is the process of locating candidate objects 58 in the height map 56. One objective of the present invention is to detect human objects standing or walking in relatively flat areas. Because the human objects of interest are much higher than the ground, local maxima in the height map 56 typically represent the head of a human object or an occasional raised hand or other object carried on the shoulders of a human object walking in the counting area 84, 86 ( Figure 1 ). Thus, local maxima of height map 56 are identified as locations of potential human object 58 detections. Each potential human object 58 detection is represented in height map 56 by a local maximum having a height greater than a predefined threshold and all distances to other local maxima above a predefined range.

[0073] Sometimes, some human objects of interest do not appear as local maxima, such as because the height map 56 is affected by false detection due to the snow blindness effect during the generation of the disparity map 50 or the human objects of interest stand close to taller objects (such as walls or doors). To overcome this problem, the present invention searches for suboptimal positions near the latest local maximum as candidate positions of the human objects of interest, which will be described later.

[0074] A run is a group of consecutive pixels with the same non-zero height value on the same row of the height map 56. Each run is represented by a four-tuple (row, start column, end column, height). In practice, the height map 56 is usually represented by a group of runs in order to improve processing performance and also perform object detection on runs rather than on pixels.

[0075] Object detection consists of four stages: 1) background reconstruction; 2) first-pass component detection; 3) second-pass object detection; and 4) merging closely located detections.

[0076] 2.1.1 Component Definition and Properties

[0077] If pixel q and pixel p share an edge or vertex in height map 56, and both p and q have non-zero height values, then q is an eight-neighbor pixel of p. A pixel can have up to eight neighbor pixels.

[0078] If for each pair of pixels Pi and Pi in a set of pixels E, there exists a sequence of pixels Pi'...,Pi such that all pixels in the sequence belong to the set E and each pair of two adjacent pixels are eight-neighborhood pixels of each other, then E is an eight-connected component. Without further explanation, an eight-connected component is referred to as a connected component hereinafter.

[0079] A connected component is a data structure representing a set of eight connected pixels in the height map 56. A connected component may represent one or more human objects of interest. The attributes of a connected component include height, position, size, etc. Table 1 provides a list of attributes associated with a connected component. Each attribute has an abbreviated name and a description enclosed in a pair of brackets. Attributes will be referenced below by their abbreviated names.

[0080]

[0081] A number of predicate operators are applied to a subset of properties of a connected component to check whether the subset of properties satisfies a specific condition. The component predicate operators include:

[0082] IsNoisy, which checks whether a connected component is too small to be considered a valid object detection 58. A connected component is considered "noisy" if at least two of the following three conditions are true: 1) its det_minSize is less than two-thirds of the specified minimum human body size, which can be configured in the range of [9, 36] inches; 2) its det_area is less than four-ninths of the area of ​​a circle with a diameter equal to the specified minimum body size; and 3) the product of its det_minSize and det area is less than the product of the specified minimum human body size and the specified minimum body area.

[0083] IsPointAtBoundaries checks whether a square window centered at the current point and having sides equal to the specified local maximum search window size intersects the boundaries of the height map 56, or whether a connected component has more than a certain number of pixels in the dead zone 90. If this operation returns true, the point being checked is considered to be within the boundaries of the height map 56.

[0084] NotSmallSubComponent, which checks if a subcomponent in the second pass of component detection is not small. It returns true if its detrninxize is larger than the specified minimum human head size or its det_area is larger than the specified minimum human head area.

[0085] BigSubComponentSeed, which checks if the subcomponent seed in the second pass of component detection is large enough to stop the grouping operation. It returns true if its detrninxize is larger than the specified maximum human head size or its det_area is larger than the specified maximum human head area.

[0086] SmallSubComponent, which checks if a subcomponent in the second pass of component detection is small. It returns true if its detrninxize is smaller than the specified minimum human head size or its der area is smaller than the specified minimum human head area.

[0087] 2.1.2 Background reconstruction

[0088] The background represents the static scenery in the field of view 44 of the image capture device 20 and is constructed from height maps 56. The background building process monitors each pixel of each height map 56 and updates the background height map. If a pixel has the same non-zero height value for a specified percentage (e.g., 70%) of the time, the pixel may be considered part of the static scenery.

[0089] 2.1.3 First pass component detection

[0090] The first-pass components are computed by applying a variation of the eight-connected image labeling algorithm to the runs of the height map 56. The properties of the first-pass components are computed according to the definitions in Table 1. Predicate operators are also applied to the first-pass components. Those first-pass components whose "IsNoise" predicate operator returns "true" are ignored and not passed to the second-pass component detection stage of object detection.

[0091] 2.1.4 Second pass of object detection

[0092] In this stage, the height map local maxima are obtained from the first pass components in the following steps, which will be considered as candidate human detections.

[0093] First, for each first-pass component, find all eight connected subcomponents whose pixels have the same height. Clear the deigrouped attributes of all subcomponents to prepare for subcomponent grouping, and set the deCID attribute of each subcomponent to the ID of the corresponding first-pass component.

[0094] Next, try to find the highest ungrouped local maximum subcomponent that satisfies the following two conditions: (1) the subcomponent has the highest height among all ungrouped subcomponents of the given first-pass component, or, if there are multiple ungrouped subcomponents with the same highest height, has the largest area among all ungrouped subcomponents of the given first-pass component; and (2) the subcomponent is taller than all its neighboring subcomponents. If such a subcomponent exists, use it as the current seed and proceed to the next step for further subcomponent grouping. Otherwise, return to step 1 to process the next first-pass component in the row.

[0095] Third, if the BigSubComponentSeed test returns true for the current seed, the subcomponent is considered a potential human object detection. Set the subcomponent's det grouped flag to mark it as grouped and continue with step 2 to find a new seed. If the test returns false, proceed to the next step.

[0096] Fourth, try to find a subcomponent next to the current seed that has the highest height and meets all three of the following conditions: (1) it is connected to the current seed; (2) its height is less than the height of the current seed; and (3) it is not connected to a higher third subcomponent and it passes the NotSmallSubComponent test. If more than one subcomponent meets all of the above conditions, choose the one with the largest area. When no subcomponent meets the criteria, set the deigrouped property of the current seed to "grouped" and go to step 2. Otherwise, proceed to the next step.

[0097] Fifth, calculate the distance between the center of the current seed and the center of the subcomponent found in the previous step. If the distance is less than the specified detection search range or the current seed passes the SmallSubComponent test, group the current seed with the subcomponent and update the current seed's attributes accordingly. Otherwise, set the current seed's det_grouped attribute to "grouped". Return to step 2 to continue the grouping process until no further grouping is possible.

[0098] 2.1.5 Merging detections with close locations

[0099] Because the image capture device 20 is installed on the ceiling of the facility entrance ( Figure 1), so the human subject of interest is identified by a local maximum in the height map. Sometimes, more than one local maximum detection is generated from the same human subject of interest. For example, when a human subject raises both of its hands at the same time, two closely located local maxima may be detected. Therefore, it is necessary to merge closely located local maxima.

[0100] The steps in this phase are as follows.

[0101] First, search for the closest local maximum detection pair. If the distance between the two closest detections is greater than the specified detection merging distance, stop and exit the process. Otherwise, proceed to the next step.

[0102] Next, check and process the two tests according to the following conditions, in the order given. Once one condition is met, ignore the remaining conditions and proceed to the next step:

[0103] a) If one but not all detections are in the background, ignore the detections in the background as they are most likely static objects (local maxima in the foreground have higher priority than local maxima in the background);

[0104] b) if one but not all detections touch the edge of the height map 56 or the dead zone, remove the detections that touch the edge of the height map 56 or the dead zone (complete local maxima have higher priority than incomplete local maxima);

[0105] c) if the difference between the det_maxHeight of the detections is less than the specified person height variation threshold, remove the detections with significantly smaller 3D volume (e.g. the product of det_maxHeight and det_masArea of ​​one connected component is less than two-thirds of the product of another connected component (strong local maxima have higher priority than weak local maxima);

[0106] d) If the difference between the maximum heights of the detections is greater than one foot, then: if the detection with the larger height of the two detections is less than the specified maximum person height, remove the detection with the smaller det_maxHeight, or if this detection's maximum height is greater than the specified maximum person height, remove the detection with the larger det_maxHeight (local maxima with plausible heights have higher priority than local maxima with unlikely heights);

[0107] e) remove detections whose det area is twice as small as the det area of ​​another detection (a small local maximum close to a large local maximum is more likely to be salt and pepper noise);

[0108] f) If the distance between the two detections is less than the specified detection search range, merge the two detections into one (two local maxima are equally good and close to each other);

[0109] g) If the distance between two detections is greater than or equal to the specified detection search range, keep both detections (two local maxima are equally good and not too close to each other). Update the det.,closestDet attribute of each detection with the ID of the other detection.

[0110] Then, return to step 1 to find the next closest detection pair.

[0111] The remaining local maximum detections after the above merging process are defined as candidate object detections 58, which are then matched against a set of existing tracks 74 for track extension, or used to initiate a new track if no match is found.

[0112] 2.2 Object Tracking

[0113] Object Tracking Figure 1 The objects detected in the object detection process (block 108) are used to extend an existing track 74 or create a new track 80. Some short, broken tracks are also analyzed for possible track repair operations.

[0114] To count human objects using object trajectories, zones 82 are depicted in height map 56. Gate zone 84 represents the gate area around the entrance to facility 23. Interior zone 86 represents the interior area of ​​the facility. Trajectories 76 that traverse from gate zone 84 to interior zone 86 have a potential "in" count. Trajectories 76 that traverse from interior zone 86 to gate zone 84 have a potential "out" count. If a trajectory 76 traverses zone 82 multiple times, there can only be one potential "in" or "out" count, depending on the direction of the most recent zone traversal.

[0115] like Figure 5 As shown, the process of object tracking 110 includes the following stages: 1) analysis and processing of old tracks (box 120); 2) first-pass matching between tracks and object detections (box 122); 3) suboptimal positioning of unpaired tracks (box 124); 4) second-pass matching between tracks and object detections (box 126); and 5) track update or creation (box 128).

[0116] Object tracks 76 may be used to determine whether a human object is entering or leaving a facility, or to derive properties such as the speed and direction of movement of the tracked human object.

[0117] Object tracks 76 can also be used to eliminate false human object detections, such as static signs around entrance areas. If an object detection 58 is not moving and its associated track 76 has been stationary for a relatively long time, the object detection 58 will be considered part of the background and its track 76 will be treated differently than normal tracks (e.g., the counts created by the track will be ignored).

[0118] Object tracking 110 also exploits the color or grayscale intensity information in the frames 52, 53 to search for the best match between the track 76 and the object detection 58. Note that color or intensity information is not carried to the disparity map 50 or the height map 56.

[0119] The same technology used in object tracking can also be used to determine how long a person has been in a checkout line.

[0120] 2.2.1 Properties of object trajectories

[0121] Each track 76 is a data structure generated from the same object being tracked in both the time domain and the space domain and contains, in addition to a set of related attributes, a list of 4-tuples (x, y, t, h), where h, x and y represent the height and position of the object in the field of view 44 at time t. (x, y, h) is defined in a world coordinate system, where the plane formed by h, x and y is parallel to the ground and the h axis is perpendicular to the ground. Each track can only have one position at any time. In addition to the list of 4-tuples, the track 76 also has a set of attributes defined in Table 2 and will be referenced later by the abbreviated name of the attribute in parentheses:

[0122]

[0123]

[0124]

[0125] 2.2.2 Trajectory-related prediction operations

[0126] A number of predicate operators are defined to obtain the current state of the track 76. The predicate operators are applied to a subset of attributes of the track 76 to check whether the subset of attributes satisfies a certain condition. The predicate operators include:

[0127] IsNoisyNow, which checks whether the track is jumping back and forth locally at the current time. Specifically, a track 76 is considered noisy if a track point with a fixed number of frames in the past (specified as the noisy track duration) satisfies one of the following conditions:

[0128] a) The range (trkrange) of trajectory 76 is less than the specified noisy trajectory range, and the negative distance (trk_negDist) is greater than two-thirds of the positive distance (trk_posDist) or the negative step (trk_negNum) is greater than two-thirds of the positive step (trk_posNum);

[0129] b) The range of trajectory 76 (trkrange) is less than half the specified noisy trajectory range, and the negative distance (trk_negDist) is greater than one-third of the positive distance (trk_posDist) or the negative step (trk_negNum) is greater than one-third of the positive step (trk_posNum).

[0130] WholeTrackIsNoisy: Track 76 may be noisy at one time and not noisy at another time.

[0131] This check is used when trajectory 76 was created a while ago, and the entire trajectory 76 is considered noisy if one of the following conditions is true:

[0132] a) The range (trkrange) of trajectory 76 is less than the specified noisy trajectory range, and the negative distance (trk_negDist) is greater than two-thirds of the positive distance (trk_posDist) or the negative step (trk_negNum) is greater than two-thirds of the positive step (trk_posNum);

[0133] b) The range of trajectory 76 (trkrange) is less than half the specified noisy trajectory range and the negative distance (trk_negDist) is greater than one-third of the positive distance (trk_posDist) or the negative step (trk_negNum) is greater than one-third of the positive step (trk_posNum).

[0134] IsSameTrack checks whether the two tracks 76, 77 are likely caused by the same human subject. This test must meet all three of the following conditions to return true: (a) the two tracks 76, 77 overlap in time by a minimum number of frames specified as the maximum track timeout; (b) the range of the two tracks 76, 77 is above a threshold specified as the span of valid count tracks; and (c) at any moment in time, the distance between the two tracks 76, 77 must be less than the specified minimum human width.

[0135] IsCountIgnored: When a track 76 crosses the counting zone, it may not be created by the human subject of interest. The count of the track is ignored if one of the following conditions is met:

[0136] Invalid trajectory: the absolute difference between trk_exits and trk_enters is not equal to one.

[0137] Small trace: trkrange is smaller than the specified minimum count trace length.

[0138] Unreliable merged track: trkrange is less than the specified minimum background count track length and one of the following conditions exists: trk_mergedTracks is equal to trk_smallSearches, or trk_backgroundCount exceeds 80% of the lifetime of track 76, or track 76 crosses the region boundary multiple times.

[0139] High object test: trk_highPtSteps is greater than half of trk_totaISteps.

[0140] Child test: trk_lowPtSteps is greater than 3 / 4 of trk_totaISteps, and trk_maxTrackHt is less than or equal to the specified minimum human height.

[0141] Shopping cart test: trk_voteFollowing is greater than 3, trk_minFollowingDist is greater than or equal to 80% of trk_maxFollowingDist, and trk_maxTrackHt is less than or equal to the specified shopping cart height.

[0142] False track test: trk_voteMirrorTrack is greater than 60% of the lifetime of track 76 and trk_maxMirrorTrackDist is less than two-thirds of the specified maximum person width or trk_totalVoteMirrorTrack is greater than 80% of the lifetime of track 76.

[0143] 2.2.3 Trajectory Update Operation

[0144] refer to Fig.12 , when there is a best matching human object detection 58 in the current height map 56, each track 76 is updated with new information about its position, time, and height. First, the trk_state of the track 76 is set to 1 (block 360).

[0145] Next, for the current frame, the height is obtained by using a median filter on the latest three heights of the trajectory 76 and the new position 56 is calculated by averaging the latest three positions of the trajectory 76 (block 362).

[0146] Third, for the current frame, the noise status is checked using the trajectory predicate operator IsNoisyNow. If true, the past specified number of frames are marked as noisy. In addition, the noise-related attributes of the trajectory 76 are updated (box 364).

[0147] Fourth, the span of the trace 76 is updated (block 366).

[0148] Fifth, the counts carried by the trace 76 are collected (block 374) if one of the following conditions is met:

[0149] a) track 76 was not noisy at the beginning, but it became noisy for longer than the specified stationary track timeout (block 368); or

[0150] b) Track 76 was not in the background initially, but has become in the background for longer than the specified stationary track timeout (block 370).

[0151] Finally, the current zone information is updated (block 372).

[0152] 2.2.4 Trajectory prediction calculation

[0153] When looking for the best matching detection 58, it is helpful to use the predicted position of the track 76. The predicted position is calculated by linearly extrapolating the position of the track 76 over the past three seconds.

[0154] 2.2.5 Analysis and processing of old tracks

[0155] This is the first stage of object tracking. An active track 88 is a track 76 that was created or extended in the previous frame using a human object detection 58. When there is no best matching human object detection 58 for a track 76, the track 76 is considered inactive.

[0156] This stage mainly processes tracks 76 that have been inactive for a certain period of time or marked for deletion in the previous frame 56. Track analysis is performed on tracks 76 that have been inactive for a long time to decide whether to group them with existing tracks 74 or mark them for deletion in the next frame 56. If track 76 has been marked for deletion in the previous frame 56, or track 76 is inactive and was created a short period of time ago, then track 76 is deleted. If the count of the track 76 that is about to be deleted should not be ignored according to the IsCountIgnored predicate operator, the count of track 76 is collected.

[0157] 2.2.6 First-pass matching between trajectories and detections

[0158] After all tracks 76 have been analyzed for grouping or removal, this stage searches for the best match between the human object detection 58 (ie, the set of local maxima found in the object detection stage) and the tracks 76 that have not been removed.

[0159] First, each possible track 76 and detection 58 pair is checked and placed into a candidate list if all of the following conditions are met:

[0160] 1) the track 76 is active, either it must be long enough (e.g., has more than three points), or it just became inactive a short time period ago (e.g., it has less than three frames);

[0161] 2) when track 76 has not moved too far (e.g., the span of track 76 is less than the specified minimum human head size and track 76 has more than 3 points), the smaller of the distances from the center of detection 58 to the last two points of track 76 is less than two-thirds of the specified detection search range;

[0162] 3) If detection 58 is in the background, the maximum height of detection 58 must be greater than or equal to the specified minimum human height;

[0163] 4) If detection 58 is neither in the background nor near a dead zone or height map boundary, and track 76 was neither in the background nor noisy in the previous frame, and a first distance from detection 58 to the predicted position of track 76 is less than a second distance from detection 58 to the end position of track 76, then use the first distance as the matching distance. Otherwise, use the second distance as the matching distance. The matching distance must be less than the specified detection search range;

[0164] 5) The difference between the maximum height of detection 58 and the height of the last point of trajectory 76 must be less than a specified maximum height difference; and

[0165] 6) If the last point is off track 76 or detection 58 is in the background, or detection 58 is close to a dead zone or height map boundary, the distance from track 76 to detection 58 must be less than the specified background detection search range, which is typically less than the threshold used in condition (4).

[0166] The candidate list is sorted in ascending order according to the distance from the detection 58 to the track 76 or the height difference between the detection 58 and the track 76 (if the distances are the same).

[0167] The sorted list contains pairs of detections 58 and trajectories 76 that are not paired. Traverse the entire sorted list from the beginning and check each pair. If the detection 58 or trajectory 76 of the pair has been marked as "paired", ignore the pair. Otherwise, mark the detection 58 and trajectory 76 of the pair as "paired".

[0168] 2.2.7 Searching for suboptimal positions of unpaired trajectories

[0169] Due to the sparsity of the disparity map 50 and the height map 56, some human objects may not produce local maxima in the height map 56 and may therefore be missed in the object detection process 108. In addition, the desired local maximum may be suppressed by the adjacent higher local maximum from the taller object. Therefore, some human object tracks 76 may not always have corresponding local maxima in the height map 56. This stage attempts to solve this problem by searching for suboptimal positions of tracks 76 that do not have corresponding local maxima in the height map 56 at the current time. Tracks 76 that have been paired with detections 58 in the previous stage may also go through this stage to adjust their positions if the distance from the end point of the track 76 to the detection it is paired with is much larger than its past steps. In the following description, the track 76 currently going through this stage is referred to as track A. The search proceeds as follows.

[0170] First, referring to FIG7 , if track A is deemed unsuitable for a suboptimal position search operation (i.e., it is inactive, or it is in the background, or it is close to the border or dead zone of the height map 56, or its height in the last frame is less than the minimum human height (block 184)), then stop the search process and exit. Otherwise, proceed to the next step.

[0171] Second, if track A has moved a few steps (block 200) (e.g., three steps) and is paired with a detection (called detection A) that is not in the background and whose current step is much larger than its maximum moving step in the past period of time specified by the track timeout parameter (blocks 202, 204) (block 186), then proceed to the next step. Otherwise, stop the search process and exit.

[0172] Third, around the end point of trajectory A, a position with the maximum height sum in a predefined window is searched within the range defined by the maximum moving step of the trajectory and this position is called the best point A (box 188). If some detections 58 are deleted during the merging process of closely located detections in the object detection stage and trajectory A is long in the spatial domain or time domain (for example, the span of trajectory A is greater than the specified noisy trajectory span threshold, or trajectory A has more than three frames) (box 190), the detection closest to the end point of the trajectory is also found. If the distance between the detection and the end point of trajectory A is less than the specified detection search range (box 206), the position with the maximum height sum is searched around the deleted component and is called the best point AI (box 208). If neither the best point A nor the best point AI exists, the search process is stopped and exited. If both the best point A and the best point AI exist, the one with the larger height sum is selected. The selected best point is called the suboptimal position of trajectory A. If the maximum height at the suboptimal position is greater than the predefined maximum human height (box 192), the search is stopped and exited. If there are no current detections around the suboptimal position (block 194), create a new detection 58 at the suboptimal position (block 214) and stop searching. Otherwise, find the detection 58 closest to the suboptimal position and call it detection B (block 196). If detection B is the same detection as detection A in step 2 (block 198), update the position of detection A with the suboptimal position (block 216) and exit searching. Otherwise, proceed to the next step.

[0173] Fourth, referring to Figure 8, if detection B has not been paired with track 76 (block 220), proceed to the next step. Otherwise, the paired track of detection B is called track B and one of the following operations is performed in the given order before exiting the search:

[0174] 1) When the suboptimal position of track A and detection B are from the same parent component (e.g., supported by the same first-pass component) and the distance between track A and detection B is less than half of the specified maximum human width, create a new detection 58 at the suboptimal position (block 238) if all three of the following conditions are met: (i) the difference between the maximum height at the suboptimal position and the maximum height at detection B is less than the specified human height error range; (ii) the difference between the sum of the heights at the two positions is less than half of the larger one; (iii) the distance between them is greater than the specified detection search range and the trk_range values ​​of both track A and track B are greater than the specified noisy track offset. Otherwise, ignore the suboptimal position and exit;

[0175] 2) If the distance between the suboptimal position and detection B is greater than the specified detection search range, create a new detection 58 at the suboptimal position and exit;

[0176] 3) If trajectory A is not observable in both the temporal and spatial domains (block 226), then the suboptimal position is ignored;

[0177] 4) If track B is not observable in both the temporal and spatial domains (block 228), separate track B from detection B and update the position of detection B with the suboptimal position (block 246). Mark detection B as the closest detection of track A;

[0178] 5) Find the best point of track B around its end position (block 230). If the distance between the best point of track B and the suboptimal position is less than the specified detection search range (block 232) and the best point of track B has a larger height sum, replace the suboptimal position with the best point of track B (block 233). If the distance between the best point of track B and the suboptimal position is greater than the specified detection search range, create detection 58 at the best point of track B (block 250). If detection A exists, update the position of detection A with the suboptimal position.

[0179] Fifth, if the suboptimal position and detection B are not under the support of the same first-pass component, proceed to the next step. Otherwise, if their distance is greater than half the specified maximum person width, create a new detection at the suboptimal position, otherwise ignore the suboptimal position and mark detection B as the closest detection for track A.

[0180] Finally, if their distance is greater than the specified detection search range, a new detection 58 is created at the suboptimal position and detection B is marked as the closest detection of track A (box 252). Otherwise, if the height sum at the suboptimal position is greater than the height sum at detection B, the end position of track A is updated with the suboptimal position (box 254), otherwise detection B is marked as the closest detection of track A.

[0181] 2.2.8 Second-pass matching between trajectories and detections

[0182] After the previous stage, some new detections may be added and some paired detections 72 and trajectories 76 become unpaired again. This stage finds the best match between the current unpaired detections 72 and trajectories 76 in the following steps.

[0183] For each track 76 and detection 58 pair that remains unpaired, the pair is put into the candidate list if all five of the following conditions are met:

[0184] 1) Track 76 is active (block 262 in FIG. 9 );

[0185] 2) when the track has not moved too far (e.g., the span of track 76 is less than the minimum head size and track 76 has more than three points (block 276)), the distance from detection 58 to the end point of track 76 (block 274) is less than two-thirds of the specified detection search range (block 278);

[0186] 3) If detection 58 is in the background (block 280), the maximum height of detection 58 must be greater than or equal to the specified minimum human height (block 282);

[0187] 4) the difference between the maximum altitude and the altitude of the last point of the trajectory 76 is less than a specified maximum altitude difference (block 284);

[0188] 5) If the last point of track 76 or detection 58 is in the background (box 286) or detection 58 is close to a dead zone or height map boundary (box 288), the distance from detection 58 to track 76 must be less than the specified background detection search range; or if not, the distance from detection 58 to track 76 must be less than the specified detection search range (box 292).

[0189] The candidate list is sorted in ascending order according to the distance from the detection 58 to the track 76 or the height difference between the two if the distances are the same (block 264).

[0190] The sorted list contains pairs of detections 58 and trajectories 76 that were not paired at all at the beginning. The entire sorted list is then traversed from the beginning and each pair is checked. If the detection 58 or trajectory 76 of the pair has been marked as "paired", the pair is ignored. Otherwise, the detection 58 and trajectory 76 of the pair are marked as "paired" (box 270).

[0191] 2.2.9 Track update or creation

[0192] After the second matching pass, perform the following steps to update old tracks or create new tracks:

[0193] First, refer to Fig.10 For each pair of trajectory 76 and detection 58, the trajectory 76 is updated with information from the detection 58 (blocks 300, 302).

[0194] Second, if the maximum height of the detection 58 is greater than the specified minimum human height and the distance between the detection 58 and the closest track 76 to the detection 58 is greater than the specified detection search range, a new track 80 is created (blocks 306, 308) for each detection 58 that does not match the track 76. When the distance is less than the specified detection merging range and the detection 58 and the closest track 76 are supported by the same first-pass component (i.e., the detection 58 and the track 76 are from the same first-pass component) (blocks 310, 320), the trk_lastCollidingTrack of the closest track 76 is set to the ID of the newly created track 80 (if any).

[0195] Third, each unpaired track 77 is marked as inactive (box 324). If this track 77 has a marked closest detection and the detection 58 has a paired track 76, the trk_lastCollidingTrack attribute of the current track 77 is set to the track ID of the paired track 76 (box 330).

[0196] Fourth, for each active track 88, a search is made for the closest track 89 that moves in a direction that is at most thirty degrees from the direction of the active track 88. If the closest track 89 exists, the track 88 is considered to be closely followed by another track, and the attributes of the track 88 related to the "shopping cart test" are updated in preparation for the "shopping cart test" when the track 88 is to be deleted later (block 334).

[0197] Finally, for each active track 88, the closest track 89 is searched. If the distance between the two is less than the specified maximum human width and the track 88 has the closest detection of the mark or its height is less than the specified minimum human height, the track 88 is considered to be a less reliable false track. The attributes associated with the "false track" are updated to prepare for the "false track" test when the track 88 is to be deleted later (box 338).

[0198] Therefore, all existing tracks 74 are either extended or marked as inactive, and new tracks 80 are created.

[0199] 2.2.10 Trajectory Analysis

[0200] Whenever a track 76 is to be deleted, the track analysis is applied. When a track 76 is not paired with any detection within a specified time period, it will be deleted. This happens when the human subject moves out of the field of view 44 or when the track 76 is interrupted due to poor disparity map reconstruction conditions (such as very low contrast between the human subject and the background).

[0201] The goal of trajectory analysis is to find those trajectories that are likely to be continuations of some trajectories that are about to be deleted and merge them. Trajectory analysis starts with the oldest trajectory and can be recursively applied to newly merged trajectories until no trajectories can be further merged. In the following description, the trajectories that are about to be deleted are called seed trajectories, and the other trajectories are called current trajectories. The steps of trajectory analysis are as follows:

[0202] First, if the seed track is noisy when active (box 130 in Figure 6), or its trkrange is less than the specified merging track span (box 132), or its trk_lastCollidingTrack does not contain a valid track ID and it was created less than the specified merging track time period ago (box 134), then stop and exit the track analysis process.

[0203] Second, if the "is the same track" predicate operation on the active track returns true, each active track created before the specified merge track time period is checked and the active track is merged with the seed track.

[0204] Third, if the current trajectory satisfies all three of the following initial test conditions, proceed to the next step. Otherwise, if there is a best-fit trajectory (the definition and search criteria of the best-fit trajectory will be described in the next step), the best-fit trajectory is merged with the seed trajectory (box 172, 176). If there is no best-fit trajectory, if the seed trajectory has been merged with at least one trajectory in this operation, the seed trajectory is retained (box 178), otherwise the seed trajectory is deleted (box 182). Then, exit the trajectory analysis.

[0205] The initial test conditions used in this step are: (1) the current trajectory has not been marked for deletion and has been active for a sufficiently long time (e.g., more than three frames) (box 142); (2) the current trajectory is continuous with the seed trajectory (e.g., it was created within a specified maximum trajectory timeout of the end point of the seed trajectory) (box 144); (3) if the two trajectories are spatially shorter (e.g., the trkrange attribute of both trajectories is less than the noisy trajectory length threshold), then the two trajectories should move in the same direction based on the relative offset of the trk_start and trk_end attributes of each trajectory (box 146).

[0206] Fourth, merge the seed track and the current track. If the current track has collided with the seed track (i.e., the trk_lastCollidingTrack of the current track is the trk_ID of the seed track), return to the last step. Otherwise, proceed to the next step.

[0207] Fifth, if both of the following conditions are met, proceed to the next step, otherwise return to step 3: (1) if either track is at the boundary according to the "at the boundary" check (box 148), then the two tracks should move in the same direction; and (2) at least one track is not noisy when merged (box 150). The noisy condition is determined by the "is noisy" predicate operator.

[0208] Sixth, one of the two thresholds that are about to appear is used for distance checking. A first threshold is specified for normal and clean tracks (box 162), and a second threshold is specified for noisy tracks or tracks in the background. If the seed track or the current track is unreliable (e.g., at the boundary, or either track is noisy, or the trkrange of both tracks is less than the specified noisy track length threshold and at least one track is in the background) (box 160), the second threshold is used (box 164), otherwise the first threshold is used. If the shortest distance between the two tracks during the overlapping time of the two tracks is less than the threshold (box 166), the current track is marked as the best fit track of the seed track (box 172). If the seed track i) does not have a best fit track yet or ii) the current track is closer to the seed track than the existing best fit track (box 170), go to step 3.

[0209] 2.2.11 Trajectory Merging

[0210] This operation merges two tracks into one and assigns the merged track attributes derived from both tracks. Most attributes of the merged track are the sum of the corresponding attributes of the two tracks, with the following exceptions:

[0211] refer to Fig.11 The trk_enters and trk_exits attributes of the merged trajectory are the sum of the corresponding attributes of the trajectory plus the counts caused by zone crossings from the end ozone trajectory to the start of the other trajectory, which compensates for lost zone crossings in the time gap between the two trajectories (box 350).

[0212] If a time point has multiple positions after merging, the final position is the average (block 352).

[0213] The trk_start attribute of the merged trajectory has the same trk_start value as the newer of the two trajectories being merged, and the trk_end attribute of the merged trajectory has the same trk_end value as the older of the two trajectories being merged (block 354).

[0214] The buffered original height and original position of the merged trajectory are the buffered original height and original position of the older of the two trajectories being merged (block 356 ).

[0215] Another embodiment of the present disclosure may be adapted for use in a counting system, which may be adapted for use in a variety of environments, such as a retail environment. Fig.13 , an example of a counting system deployed in a retail environment 1301 is shown. The retail environment 1301 may represent a shopping mall 1301 having a plurality of retail units 1302. Each retail unit 1302 may include a stereoscopic optical sensor unit 1303. For example, the stereoscopic optical sensor unit 1303 may be located at ceiling height above an interior entrance of the respective retail unit 1302. The stereoscopic optical sensor unit 1303 includes sensors and logic for counting each individual that visits the retail unit 1302. The stereoscopic optical sensor unit 1303 will also report such counting data to a user.

[0216] In retail environment 1301, users can evaluate counting data, for example, for analyzing statistics such as sales conversion rate and visitor traffic. However, one problem is to obtain accurate counting data from visitors entering and leaving a given retail unit 1302. More specifically, distinguishing between adult visitors and child visitors of a given retail unit 1302 is usually concerned when obtaining useful information for customer analysis. For example, a user may want to evaluate the habits of adult visitors of a retail unit 1302, and if child visitors are included in the counting, accurate insights into this habit may not be obtained. As another example, a user may want to evaluate the habits of child visitors of a retail unit 1302 separately. However, due to the relatively significant variability of the detection height of the tracked object, it may be a challenge for a counting system to distinguish between adult visitors and child visitors. The current method of doing so may be inaccurate and involve computational complexity.

[0217] Embodiments presented herein disclose techniques for obtaining relatively accurate height measurement estimates of tracked objects that indicate an individual. More specifically, the techniques process a set of height data of the individual collected over time. As further described herein, the techniques determine a relatively accurate height estimate for the tracked individual. For example, a k-means clustering algorithm can be performed to determine multiple clusters from the set of height data. The estimate can be derived from the densest cluster. This estimate can then be evaluated against a height threshold and used to determine whether the individual corresponds to an adult or a child. The individual can then be counted as an adult or a child. Advantageously, embodiments presented herein disclose techniques for accurately determining the height of a tracked object in a manner that requires less computational complexity.

[0218] Reference now Fig.14, a simplified diagram further illustrating that the stereoscopic optical sensor unit 1303 is located at an entrance 1404 of a location (e.g., retail unit 1302) is shown. A perspective view of the exterior of the stereoscopic optical sensor unit 1303 is shown at Fig.14 1402 and 1403. Illustratively, the stereo optical sensor unit 1303 includes a housing 1402 and a camera lens 1403. The illustrative embodiment provides two camera lenses 1403 positioned horizontally apart from each other. When multiple camera lenses 1403 are used, the stereo optical sensor unit 1303 can measure depth, or if the distance from the stereo optical sensor unit 1303 and the ground is known, the height can be measured.

[0219] Illustratively, the camera in the stereo optical sensor unit 1303 captures image data of individuals 1405 entering and leaving its field of view. Logic in the stereo optical sensor unit 1303 processes the image data. For example, the stereo optical sensor unit 1303 generates a height map of the scene captured by the field of view and identifies individuals to include as count data based on an evaluation of the generated height map.

[0220] Reference now Fig.15 , a block diagram depicting the hardware components of the stereo optical sensor unit 1303 is shown. As shown, the stereo optical sensor unit 1303 includes, but is not limited to, a central processing unit (CPU) 1502, a stereo optical camera 1504, a network interface 1506, a memory 1508, and a storage device 1510. Each of these components can be interconnected via an interconnect bus 1512. Note that Fig.15 Examples of hardware components that may be included in a given stereoscopic optical sensor unit 1303 are provided. Of course, those skilled in the art will recognize that the stereoscopic optical sensor unit 1303 may include additional components not discussed herein. Furthermore, the components of the stereoscopic optical sensor unit 1303 may also be provided in different Fig.15 For example, some components (such as sensors) may be wirelessly connected to the stereo optical sensor unit 1303 using a wireless communication protocol such as Bluetooth.

[0221] CPU 1502 retrieves and executes programming instructions stored in memory 1508, and stores and retrieves application data residing in storage 1510. Bus 1512 is used to transfer programming instructions and data between CPU 1502, storage 1510, network interface 1506, and memory 1508. Note that CPU 1502 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc. Memory 1508 is generally included to represent random access memory. Storage 1510 may be a disk drive storage device. Although shown as a single unit, storage 1510 may be a combination of fixed storage devices and / or removable storage devices, such as a fixed disk drive, a removable memory card, or an optical storage device, a network attached storage (NAS), or a storage area network (SAN).

[0222] The stereo optical camera 1504 includes multiple sensor lenses (e.g., camera lenses 1403) for capturing image data that can be temporarily maintained by the storage device 1510. The memory 1508 includes program code logic for directing the stereo optical camera 1504 to capture image data. The program code can also (e.g., using the above-mentioned techniques) identify individuals in the image data to count the data, and generate height map data for each individual and process the height map data (e.g., to distinguish between adult individuals and child individuals). The network interface 1506 can be implemented as any communication circuit, device, or collection thereof that can enable communication between the optical sensor unit 1303 and other devices over a network.

[0223] Reference now Fig.16 , a block diagram of the component arrangement of the counting system 1601 is shown. Illustratively, the counting system 1601 may include one or more stereo optical sensor units 1303. Each stereo optical sensor unit 1303 may include two camera lenses 1403. Each stereo optical sensor unit 1303 communicates with each other. For example, as shown, the stereo optical sensor units 1303 are connected to each other using data cables 1604. Doing so allows each stereo optical sensor unit 1303 to transmit data (e.g., image data, height data, count data, etc.) between each other. In addition, the stereo optical sensor unit 1303 can be connected to a computing device 1607 (e.g., via a cable 1606 or wirelessly). The computing device 1607 can be a management console that receives image data and other statistical data (e.g., height data, count data, etc.) generated by a given stereo optical sensor unit 1303. Note that although Fig.16Each component of the counting system 1601 is depicted as being connected to each other via a data cable (eg, data cables 1606, 1606), but those skilled in the art will recognize that the components may also be wirelessly connected to each other using some wireless standard (eg, WiFi, Bluetooth, etc.).

[0224] During configuration of the counting system 1601, the height of the stereoscopic optical sensor units 1303 from a surface (e.g., the floor of a retail unit) is measured to establish an accurate relationship between a given sensor unit 1303 and the surface. Doing so allows the stereoscopic optical sensor units 1303 to reliably detect and track individuals in a scene (e.g., an image area captured by the optical sensor in the field of view).

[0225] The counting system 1601 is also configured to define counting areas. In some embodiments, the counting areas correspond to areas within the captured image data that represent the physical area that individuals pass through to be counted by the system 1601. Fig.17 , shows an example counting area 1700. Illustratively, two counting zones (Zone 1 and Zone 2) are defined. Counting zones are preconfigured spatial boundaries that individuals cross to be counted by system 1601. As individuals cross Zone 1 and Zone 2, counting system 1601 counts entries and exits of each zone separately. For an individual to be counted as entering, counting system 1601 may determine that the individual passed through Zone 1 before passing through Zone 2. For an individual to be counted as leaving, counting system 1601 may determine that the individual passed through Zone 2 before passing through Zone 1. Fig.17 As shown, individual A has passed through zone 1 before passing through zone 2. In this case, counting system 1601 counts individual A as entering (e.g., counting system 1601 can increment a counter associated with the entry and individual B). Fig.17 As further shown, individual B is shown approaching and passing through zone 2 and then passing through zone 1. If so, counting system 1601 counts individual B as leaving (e.g., counting system 1601 may increment a counter associated with leaving and individual B).

[0226] In some embodiments, the user can calibrate the counting system 1601 to correct for lens distortion (e.g., in the stereo optical sensor unit 1601). Once calibrated, the counting system 1601 can better determine the height of the tracked object, given that the distance of the stereo optical sensor unit 1601 from the surface is known. Other information can be used for height calculations, such as lens focal length, distance between lenses, etc. In some cases, two images of the scene are recorded simultaneously by each camera and the images can be used to derive disparity information between any two corresponding points in the two images, where both points refer to the same physical point in the coordinate system. Once the disparity map is created, a height map is calculated to reflect the height of each point in the scene visible from the sensor. After these operations, a first height image is obtained, where each pixel represents a point in the scene and the pixel value represents the height of the point from the surface.

[0227] Reference now Fig.18 , an example illustration 1800 of a grid view of an 8-connected view is shown. Objects of interest for the counting system may include individuals moving through the scene. Individuals may be identified in the generated height map using blob detection. Typically, blob detection involves classifying pixels having related attribute values ​​(e.g., RGB values, height values, etc.) that are close to each other into groups. The stereoscopic optical sensor unit 1303 may create pixel groups from group members having height values ​​within a specified range. For example, a specified range for classifying a given object may include pixels having non-zero height values ​​that are close to each other. Continuing with this example, pixels having zero height pixels that are close to each other may correspond to a background (e.g., a floor of a retail unit).

[0228] Graph 1800 is an example of a non-zero height, 8-connected blob. In general, a non-zero height, 8-connected blob is a group of pixels where each member is an 8-neighborhood pixel of another. In addition, in general, if two pixels share an edge or vertex, the pixels are 8-neighborhood pixels of each other. Illustratively, each square on the grid of graph 1800 represents a pixel. The 8-connected blob of graph 1800 is a group of white pixels. Each white pixel has a non-zero height value and represents an object tracked by counting system 1601. Black pixels have a zero height value and represent the background of counting system 1601.

[0229] From an observational point of view, the spots can correspond to the shapes of the individuals. For example, now refer to Fig.19, an example illustration 1900 of a top view of a blob representation of an individual is shown. The black areas of the illustration 1900 represent zero height value areas and the non-black areas indicate blob representations. More specifically, the non-black areas represent pixels with height values. The counting system 1601 can track the blob within the counting zone. The counting system 1601 can count the individuals represented by the blob at the end of the track (e.g., when the individual likes the counting zone or at the end of a specific duration).

[0230] The height of an individual at each image frame can be evaluated as the maximum pixel height value within the blob representing the individual. Along the trajectory of the blob, the counting system 1601 (e.g., via the stereo optical sensor unit 1303) can collect a set of height data from each image frame during tracking. Due to various factors, such as posture changes, object occlusion, calculation errors, etc., the height data may vary along the trajectory. However, the counting system 1601 can apply a clustering algorithm to the height data to improve the height accuracy calculation.

[0231] Reference now Fig. 20 and Fig.21 , describes a method 2000 for estimating the height of a tracked object (e.g., a monitored individual entering and exiting a retail unit). For example, the counting system 1601 can perform the method 2000 in operation. In some embodiments, a single stereo optical sensor unit 1303 of the counting system 1601 can perform the method 2000. In addition, a computing system (e.g., computing device 1607) can perform the method 2000.

[0232] As shown, method 2000 begins at block 2000, where counting system 1601 detects an individual entering a counting zone. Counting system 1601 may detect the individual using the techniques described above. In block 2004, counting system 1601 captures one or more images of the individual. In block 2006, counting system 1601 generates a height map of the scene for each image. Doing so allows counting system 1601 to distinguish between background (e.g., pixels with zero height values) and objects of interest, such as individuals. Counting system 1601 also identifies local maxima within each height map (e.g., the maximum height value over pixels in the scene).

[0233] In block 2008, the counting system 1601 detects spots corresponding to the tracked objects in each height map. The counting system 1601 identifies a current point in time for the height map. For example, the counting system 1601 may use timestamp data maintained by the stereo optical sensor unit 1303 corresponding to the time when the stereo optical sensor unit 1303 captured the image. In block 2010, the counting system 1601 tracks the spots through the counting zone for a specified duration. The specified duration may be a period of time for a particular counting zone. In block 2012, the counting system 1601 determines whether the spot track is complete. For example, spot tracking may be complete once the tracked object is no longer in the scene. As another example, the spot track may be complete once the specified duration has passed.

[0234] In block 2014, the counting system 1601 obtains a set of height data of the tracked spot within a specified duration. The set of height data may correspond to the height values ​​of the spot tracked through the captured image. For example, each height value may correspond to the maximum height value of the tracked spot in a given image.

[0235] In box 2016, counting system 1601 applies clustering techniques to the height data set. The result of doing so is to generate multiple clusters. In some embodiments, counting system 1601 can apply a k-means clustering algorithm to generate multiple clusters. Each cluster includes multiple height values ​​from the height data set. In practice, generating three clusters has been proven to be effective. However, other numbers of clusters can also be considered. Similarly, other clustering algorithms can also be applied to height data.

[0236] The counting system 1601 continues to apply the clustering technique to the height data to provide stability in each cluster (e.g., to reduce the variation in cluster size with each successive iteration of the clustering technique). In block 2018, the counting system 1601 determines whether the clusters are stable. If not, the method 2000 returns to block 2016 to continue the execution of the iterative clustering technique.

[0237] Once stable, in box 2020, the counting system 1601 identifies the cluster with the largest number of height values. In box 2016, the counting system 1601 calculates the average height value of the identified cluster. The average height value of the cluster with the largest number of height values ​​corresponds to an estimate of the tracked object representing the individual. This calculation can be evaluated to distinguish whether the tracked object corresponds to an adult individual or a child individual. For example, in box 2018, the counting system 1601 determines whether the calculated value exceeds a specified threshold indicating whether the tracked object corresponds to an adult. If not, in box 2020, the counting system 1601 registers the individual as a child for counting purposes. Otherwise, if the average height value exceeds the threshold, in box 2022, the counting system 1601 registers the individual as an adult.

[0238] The present disclosure is not limited to the embodiments disclosed herein, and it will be appreciated that many modifications and embodiments can be designed by those skilled in the art. Therefore, the appended claims are intended to cover all such embodiments and modifications that fall within the true spirit and scope of the present disclosure.

Claims

1. A computer-implemented method for evaluating altitude data, the method comprising: obtaining a plurality of height maps, wherein each height map of the plurality of height maps corresponds to an image of a scene including an object for a specified duration; detecting a spot corresponding to an object in each of the plurality of height maps; obtaining, by execution of one or more processors and from the plurality of height maps, a plurality of height values ​​for the object, wherein each height value in the plurality of height values ​​corresponds to a height value of the tracked spot in one of the plurality of height maps; Generating a plurality of clusters of height values ​​from the plurality of height values, each cluster including at least one of the plurality of height values, at least one cluster including a plurality of height values; identifying a cluster of the plurality of clusters having a greatest number of the plurality of height values ​​relative to one or more remaining clusters of the plurality of clusters; and A height of the object is estimated based on the identified cluster of the plurality of clusters.

2. The computer-implemented method of claim 1 , further comprising: A determination is made as to whether the estimated height of the object exceeds a specified threshold.

3. The computer-implemented method of claim 2, further comprising: Upon determining that the estimated height of the subject exceeds a specified threshold, the subject is classified as an adult individual.

4. The computer-implemented method of claim 2, further comprising: Upon determining that the estimated height of the object does not exceed a specified threshold, the object is classified as a child individual.

5. The computer-implemented method of claim 1, wherein: Estimating the height of the object includes calculating an average height value of a portion of the plurality of height values ​​included in the identified cluster of the plurality of clusters.

6. The computer-implemented method of claim 1, wherein: The object corresponds to an individual, and the method further comprises: detecting said individual entering a predefined area; capturing a plurality of images of the individual via one or more sensors; and For each image, a correlated height map of the plurality of height maps is generated and local maxima are identified.

7. The computer-implemented method of claim 1, wherein: Generating the plurality of clusters includes applying a k-means clustering algorithm to the plurality of height values ​​of the object.

8. The computer-implemented method of claim 1, wherein: Generating the plurality of clusters includes applying a k-means clustering algorithm to the plurality of height values ​​of the object to create three clusters, the three clusters corresponding to the plurality of clusters.

9. The computer-implemented method of claim 7 or 8, further comprising: The k-means clustering algorithm is iterated until each of the plurality of clusters is stable.

10. The computer-implemented method of claim 1, wherein: The plurality of height values ​​of the object are obtained from an image captured by an image capture device.

11. A system for evaluating altitude data, comprising: one or more processors; as well as A memory storing program code, which, when executed on the one or more processors, causes the system to perform the following operations: obtaining a plurality of height maps, wherein each height map of the plurality of height maps corresponds to an image of a scene including an object for a specified duration; detecting a spot corresponding to an object in each of the plurality of height maps; obtaining a plurality of height values ​​of the object from the plurality of height maps, wherein each of the plurality of height values ​​corresponds to a height value of the tracked spot in one of the plurality of height maps; Generating a plurality of clusters of height values ​​from the plurality of height values, each cluster including at least one of the plurality of height values, at least one cluster including a plurality of height values; identifying a cluster of the plurality of clusters having a greatest number of the plurality of height values ​​relative to one or more remaining clusters of the plurality of clusters; and A height of the object is estimated based on the identified cluster of the plurality of clusters.

12. The system of claim 11, wherein: The program code further causes the system to perform the following operations: A determination is made as to whether the estimated height of the object exceeds a specified threshold.

13. The system of claim 12, wherein: Upon determining that the estimated height of the object exceeds a specified threshold, the program code further causes the system to: The subject was classified as an adult individual.

14. The system of claim 12, wherein: Upon determining that the estimated height of the object does not exceed a specified threshold, the program code further causes the system to: The subject is classified as a child individual.

15. The system of claim 11, wherein: The program code causes the system to estimate the height of the object at least in part by calculating an average height value for an identified cluster of the plurality of clusters.

16. The system of claim 11, wherein: The program code causes the system to generate the plurality of clusters at least in part by applying a k-means clustering algorithm to the plurality of height values ​​of the object to create three clusters, the three clusters corresponding to the plurality of clusters.

17. The system of claim 11, wherein: The plurality of height values ​​of the object are obtained from an image captured by an image capture device.

Images