Irregular biometric boarding

Abstract

A method for controlling access to at least one tracked object is disclosed, the method including the steps of obtaining or receiving a series of two-dimensional images in which at least one tracked object is presumably taken, and also location data for the at least one tracked object, assigning a unique tracking identifier to the at least one tracked object, determining a trajectory of the at least one tracked object from the location data, determining whether there is a discontinuity in the trajectory or in data calculated from the trajectory, and if a discontinuity is detected, obtaining or receiving one or more new images of the at least one tracked object and assigning a new unique tracking identifier to the at least one tracked object, and if a discontinuity is detected, determining whether access should be granted based on at least one of the one or more new images, or if no discontinuity is detected, based on at least one image from the series of two-dimensional images.

Description

[Technical Field]

[0001] The present disclosure relates to biometric boarding. In particular, the present disclosure relates to improvements in the assignment of passenger identifiers to facilitate passenger boarding that is not limited to a strict, prescriptive process. [Background technology]

[0002] Biometric boarding typically involves an orderly process whereby passengers are lined up in a single file. Passengers are photographed one by one in an orderly queue, and their photographs are compared to a database of authorized passengers. In an orderly queue, spacing between passengers is often maintained by airport staff or by the passengers following instructions from staff or signage.

[0003] If two passengers in a queue are standing too close to each other, the biometric identification system's facial capture algorithms can sometimes become confused, causing the facial capture algorithm to attempt to assign the same identifier to both passengers. Once the first passenger boards, the matching system is unable to match the second passenger to valid boarding data. While the above confusion does not occur all the time, when it does occur, it can lead to what is known as a process exception, causing significant delays in the overall boarding process.

[0004] Therefore, row spacing must be maintained at a distance large enough to avoid the boarding system confusing the "target" passenger identity with that of another passenger who has already boarded. However, requiring spacing between queuing passengers tends to double boarding time. In most situations, maintaining the process also requires significant manual effort by staff.

[0005] Therefore, with existing regular biometric boarding processes, the achievable process efficiency is limited by the sophistication of capture technology and the efficiency with which airline staff and signage can overcome unexpected or undesirable passenger behavior. The more passengers are required to be spaced apart, the slower the boarding process will be.

[0006] All known biometric boarding systems suffer from these same limitations, particularly their inability to distinguish between close and distant faces (face coverage is not accurate due to age or race differences), as well as the lack of identity tracking over time (someone who momentarily looks away from the camera may be replaced by someone looking at the camera after them).

[0007] Ideally, irregular boarding (still single file) would minimize operation time (no walking time delays) and increase automation (reduced requirement for strict boarding processes and well-trained personnel). Colloquially, this problem is known as the "stacking problem", i.e., biometrically addressing queues of passengers who tend to stand too close to each other and "pile up".

[0008] To avoid the identification problem, some vendors resort to placing biometric cameras perpendicular to the row direction so that passengers are likely to turn toward or appear in front of the camera one by one. However, this placement is suboptimal for biometric capture because passengers will have more difficulty aligning themselves with such a camera than with a camera aligned with their own direction of movement. This results in significant processing delays as the target passenger positions themselves at an angle to the passenger stream.

[0009] One solution is to incorporate a three-dimensional (3D) camera to obtain "depth" data, i.e., the distance between the passenger whose face is captured and the 3D camera. The face capture system can, for example, ignore any faces that are outside a certain distance threshold. However, this solution still requires that a certain distance between passengers be maintained to minimize stacking problems.

[0010] Solutions which further minimise stacking problems are desirable. Where any prior art is referred to herein, it is to be understood that such reference does not constitute an admission that the prior art forms part of the common general knowledge in the art in Australia or any other country. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] AU2019272041 Summary of the Invention [Means for solving the problem]

[0012] In one aspect, a method for controlling access to at least one tracked object is disclosed, the method including the steps of obtaining or receiving a series of two-dimensional images of at least one tracked object that are presumably taken, and also location data for the at least one tracked object; assigning a unique tracking identifier to the at least one tracked object; providing a trajectory of the at least one tracked object from the location data; determining whether there is a discontinuity in the trajectory or data calculated from the trajectory, and if a discontinuity is detected, obtaining or receiving one or more new images of the at least one tracked object and assigning a new unique tracking identifier to the at least one tracked object; and if a discontinuity is detected, determining whether access should be granted based on at least one of the one or more new images, or if no discontinuity is detected, based on at least one image from the series of two-dimensional images.

[0013] In some embodiments, determining whether there is a discontinuity includes determining whether a distance discontinuity condition or a velocity discontinuity condition is satisfied by a time series of distance data or velocity data obtained from the trajectory.

[0014] In some forms, the discontinuity condition is whether the difference between two data samples in the time series is 1) above a threshold, or 2) greater than or equal to a threshold.

[0015] In some embodiments, the threshold value depends at least in part on the amount of time that elapsed between the times two data samples were taken.

[0016] In some embodiments, the method further includes providing a time series of a statistic or metric calculated from the trajectory or two-dimensional image, and determining whether a statistic or metric discontinuity condition is satisfied by the time series of the statistic or metric.

[0017] In some embodiments, the method further includes checking whether a statistical or metric discontinuity condition is met before determining whether there is a discontinuity in the trajectory or data calculated from the trajectory.

[0018] The method may be a biometric access control method. The biometric access control method may further be a facial biometric control method, wherein the at least one tracked object is a facial area of ​​a person. The statistic or metric may be a facial area size or a biometric score calculated from the two-dimensional image.

[0019] In some forms, the method includes applying an object detection algorithm to the three-dimensional image to detect one or more objects, each object being one of the at least one tracked object.

[0020] In some embodiments, the position data comprises at least depth data.

[0021] In a second aspect, a method for counting the number of times a checkpoint passage occurs by one or more tracked objects is disclosed. The method includes processing each passage by the tracked object and acknowledging the passage by the tracked object. For each tracked object, the processing includes acquiring or receiving a series of two-dimensional images inferred to be taken of the tracked object, as well as position data for the tracked object; assigning a unique tracking identifier to the tracked object; determining a trajectory of the tracked object from the position data; and determining whether there is a discontinuity in the trajectory or data calculated from the trajectory, and if a discontinuity is detected, assigning a new unique tracking identifier to one or more obtained new images of the tracked object. Determining the number of times a checkpoint passage occurs includes determining the number of different unique tracking identifiers assigned.

[0022] In some forms, approving passage includes determining whether passage should be allowed based on a new unique tracking identifier and at least one of the one or more new images if a discontinuity is detected, or based on an existing tracking identifier and at least one image from the series of two-dimensional images if no discontinuity is detected. In determining the number of times checkpoint passage has occurred, only tracking identifiers assigned to tracked objects that are approved for passage are counted.

[0023] In a third aspect, a method is disclosed for tracking a count of the number of times a checkpoint has been crossed by one or more tracked objects, the method including the steps of acquiring or receiving a series of two-dimensional images of each tracked object presumably taken, along with position data for the tracked objects, assigning a unique tracking identifier to the tracked object, determining a trajectory of the tracked object from the position data, determining whether there is a discontinuity in the trajectory or data calculated from the trajectory, and if a discontinuity is detected, acquiring or receiving one or more resulting new images of the tracked object and assigning a new unique tracking identifier to the tracked object, determining whether passage should be permitted based on at least one of the one or more new images, or if no discontinuity is detected, based on at least one image from the series of two-dimensional images, and if passage is determined to be permitted, incrementing a count.

[0024] In a fourth aspect, a method of controlling or monitoring access to a premises or means of transportation having one or more entry points, or one or more exit points, or both, is disclosed, comprising implementing at each entry point, at each exit point, or both, the method of the first or second aspect as described above.

[0025] In a fifth aspect, a biometric access control method is disclosed, including a method according to any of the previous aspects.

[0026] The method may be a facial biometric access control method, where the tracked object is the facial area of ​​a person, for example in the form of a passenger wanting access to a moving vehicle.

[0027] The statistic or metric may be a facial area or biometric score calculated from a two-dimensional image.

[0028] The method may include applying an object detection algorithm to the three-dimensional image to detect one or more objects, each object being a tracked object.

[0029] In a sixth aspect, a computer-readable medium having stored thereon machine-readable instructions, the machine-readable instructions being adapted, when executed, to perform a method according to any of the aspects set forth above, is disclosed.

[0030] In a seventh aspect, an access control system is disclosed, comprising a processor configured to execute machine-readable instructions that, when executed, are adapted to perform a method according to any of the aspects described above.

[0031] In some embodiments, the system includes an image capture device, the device including a two-dimensional camera and a three-dimensional camera.

[0032] In use, the image capture device may be disposed in front of a queue of objects to be processed by the access control system.

[0033] The access control system may be a biometric boarding system.

[0034] In an eighth aspect, a method is disclosed for controlling or monitoring access by one or more subjects standing in a queue by processing each subject in turn using a biometric access control system, the biometric access control system including an image capture device facing directly towards the queue. The provided biometric access control system may be according to the seventh aspect described above.

[0035] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0036] [Figure 1] FIG. 1 is a schematic diagram of a biometric boarding system according to one embodiment. [Figure 2] FIG. 1 is a schematic diagram of an example process for generating trajectories from a track list to be analyzed. [Figure 3] FIG. 3(1) shows an example of a depth history of a tracked object, including discontinuities, and FIG. 3(2) shows an example of a time series of calculated facial area sizes of a tracked object, including discontinuities. [Figure 4] FIG. 10 is a diagram that schematically depicts an example of a process utilized by the discrete analysis module. [Figure 5] FIG. 10 is a diagram illustrating an example discontinuity detection algorithm. [Figure 6] FIG. 1 is a diagram illustrating a biometric boarding device. [Figure 7] FIG. 1 is a perspective view of an example "pod" type boarding device. [Figure 8] FIG. 1 depicts a boarding queue with a camera positioned directly in front of the queue. [Figure 9] FIG. 1 illustrates an example interface provided by the system. [Figure 10] 10A-10C depict example displays that may be shown on a passenger interface. DETAILED DESCRIPTION OF THE INVENTION

[0037] In the following detailed description, reference is made to the accompanying drawings, which form a part of the detailed description. The exemplary embodiments depicted in the drawings and described in the detailed description are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented. It will be readily understood that the aspects of the present disclosure, as generally described herein and illustrated in the drawings, can be arranged, substituted, combined, separated, and designed into a wide variety of different configurations, all of which are contemplated by the present disclosure.

[0038] Aspects of the invention disclosed herein may be utilized for biometric boarding of passengers onto an aircraft or other means of transportation such as a train, cruise ship, etc. It may also be utilized in applications involving monitoring and determining a subject's biometric information at a gate, lounge, door, or restricted area. Also, while many applications involve the use and determination of facial biometric information, other biometric information (e.g., hand, fingerprint, retina, etc.) may be used or determined. For brevity, various aspects of the invention will be described in the context of boarding passengers using facial biometric information.

[0039] Facial biometric boarding involves capturing one or more facial images from a passenger and then performing a match between the captured images and a reference passenger database to obtain the passenger data needed to board the passenger. Typically, an algorithm captures multiple images and then attempts to select the best image from the captured images, which will be used to perform the match. Such best images, if they match the passenger admitted to boarding, may be stored in a local or remote storage location and accessed at a later time. For example, facial images of a passenger may be captured and processed using the system disclosed herein upon deplaning or arriving at the gate to see if they match any of the stored images used for matching purposes upon boarding.

[0040] In many situations, multiple passengers will be instructed to form a queue, so that biometric boarding can be done for each passenger individually with each biometric boarding aircraft. Doing this as quickly as possible and without introducing errors is important for both logistical and economic reasons. Aspects and embodiments disclosed herein provide technological tools that facilitate reducing boarding times, as will be described herein.

[0041] The invention described herein is suitable for incorporation into biometric boarding or entry applications to reduce the likelihood that the biometric application will be unable to separate different passengers, which is often caused by passengers standing too close to each other. Thus, the invention does not require passengers to be as far apart from each other as in prior art processes, but can tolerate an irregular process with minimized likelihood of timeout issues occurring.

[0042] In biometric boarding, the biometric camera works such that passengers within the camera's field of view are "tracked" over time. This helps reduce the occurrence of error events that require the passenger to repeat the identification and boarding process. By "tracking" the passenger over time, multiple images can be acquired, allowing bad facial images unsuitable for the biometric process to be discarded. The inventors have found that a "good enough" facial image can actually be quickly replaced with a "better" facial image before a facial match occurs. This increases the probability of a good facial match and decreases the probability of no match. This saves processing time, as the inventors noted in testing where facial image capture operates at 10-15 frames per second (fps), creating more than enough images for facial matching algorithms that typically operate at 0.5-5.0 fps.

[0043] In a prior application by the inventors disclosed in AU2019272041, the contents of which are incorporated by reference, an apparatus and system are described that uses both two-dimensional (2D) and three-dimensional (3D) data to facilitate biometric boarding.

[0044] The invention disclosed herein further uses 2D and 3D information from camera systems to track objects. It uses discontinuity analysis to resolve identity tracking failures that sometimes occur when passenger spacing is not enforced. In particular, the time series of tracked passenger history is analyzed to detect specific discontinuity patterns that occur when there is a tracking failure.

[0045] Thus, in general terms, the present invention involves generating a time series of position data for a tracked passenger (i.e., a tracked face) and determining whether there has been a discontinuity in the tracked position. If a discontinuity is detected, it is treated as an indication that an error condition has occurred, e.g., the biometric boarding system is attempting to board a passenger who it determines has already boarded.

[0046] As will be explained, once such an error is detected, it is addressed by re-acquiring the subject's facial image (prior to biometric matching) so as to re-assign the subject (or its tracked object) a new unique tracking identifier.

[0047] The present invention aims to solve the problem where a person at the front of a queue leaves the field of view of the biometric camera (i.e. leaves to board the aircraft) and another person nearby at the back moves into the same spatial region in the field of view occupied by the first person.

[0048] It is possible that the person at the front of the queue will only temporarily leave the view of the biometric camera, for example if they look away or move temporarily, rather than leaving to board a flight. The biometric camera itself has some occlusion handling and requires a stable face before tracking begins. So if they "look away" for a short time, the person standing behind them will not be considered a tracked face until some time has passed, and the person at the front will retain their tracking identifier during the occlusion.

[0049] However, if the person at the front of the line disappears from the camera's field of view (e.g., bending down to tie their shoelaces) long enough for a stable face of the person behind them to be captured, a discontinuity will be detected if the person behind them is standing close enough so that their face is visible in the same spatial region of the camera's field of view. The next face in the field of view (of the person behind them) will then be tracked with a new tracker identifier.

[0050] In this case, the process fails and the following may happen: - While the first, lead person is out of sight, the second person will be tracked, authenticated, and instructed to board. Once the lead person is back in sight, that person will typically again get a new unique tracker identifier. - The first person has already "boarded" based on the biometrics of the person behind them (the second person), but when they return to the camera's field of view, the system will tell the passenger they are "already on board." - If the first person "boards" based on the biometrics of the second person behind them and then leaves without returning to their field of view, and the second person moves forward, the system will attempt to capture and match the second person's face again and report that they are "already on board." - In the hypothetical situation where the person behind is in the same spatial position as the front position and the tracking identifier does not change, the same discontinuity detector described in this application will result in the generation of a new tracking identifier and subsequently the generation of a new tracking face, and the passenger will receive an "already boarded" notification.

[0051] In this way, process consistency is maintained at the expense of operational time, where the "already on board" notification prompts airline / airport staff to handle the exception.

[0052] Furthermore, there are two additional protections envisaged for the biometric boarding process: - Displaying seat numbers on screen to provide additional manual passenger checks; and - There should be personnel at the desk in front of passengers, trained to spot unusual scenarios and able to reset the tracking system if necessary.

[0053] Embodiments of the present invention will now be described in more detail. Figure 1 conceptually depicts a biometric boarding system 100 according to one embodiment of the present invention. The system 100 is adapted to process biometric data and to control whether a subject should board. The system 100 includes a processing module 102, which may reside on a server computer or on a computing means provided on a device housing hardware for implementing the system 100.

[0054] The processing module 102 is also adapted to receive first data 104 including a two-dimensional (2D) image and second data 106 providing position information of the imaged object. The two-dimensional image 104 is an image of the object, and the second data 106 provides information regarding the position of the imaged object relative to a device capturing the image. In a facial biometric boarding application, the imaged object is a facial image of the subject. The second data 106 may be depth (z-axis) information, or it may be three-dimensional (3D) x, y, z position information. The position information may be captured by a three-dimensional (3D) camera. Depending on the specific embodiment, the processing module 102 may also access additional measurement data 108, such as point cloud data, infrared or laser scanning data.

[0055] The processing module 102 includes a detection module 110, a tracking module 112, and a discontinuity analysis module 114. The first data 104 and the second data 106 are continuously acquired at the time resolution of the acquisition device and provided to the detection module 110. The detection module 110 is adapted to detect relevant objects, in this case faces, in the input image frames. The tracking module 112 is adapted to track the detected faces and record their positions over time to generate trajectory data 116, which is a time series of position data for each of the detected faces.

[0056] Each series of image data 104 is assumed to be of the same subject and will be provided to a biometric matching module 118. Biometric matching is then performed to find a match in a passenger reference database to confirm the boarding eligibility of the subject being processed. However, before biometric matching occurs in the biometric matching module 118, the system 100 first verifies whether there may be an error in the assumption that the series of image data 104 were in fact taken from the same person.

[0057] This is done by analyzing the trajectory data 116 to see if there are any discontinuities in the trajectory 116 of the tracked object (eg, face).

[0058] The detection of a discontinuity in the tracking list then triggers the system 100 to resolve the identity of the tracked object associated with the tracking list.

[0059] FIG. 2 conceptually illustrates one embodiment of a process 200 for generating a trajectory to be analyzed. In this embodiment, the first data 104 is image data provided by a high-resolution 2D camera, and the second data 106 is image data provided by a 3D camera. The two cameras can be used interchangeably or together to collect position data of a tracked object. In the example proposed by the inventors, a 3D camera with a frame rate of 10-20 frames per second provides z-axis (and optionally x-axis and y-axis) information to the processing module 102. Once the object is determined to be correctly oriented, a higher-resolution 2D camera takes a still image of the object, which will be used for biometric matching in the biometric matching module 118. Because the facial position (from the camera's perspective) remains relatively constant despite small movements of the passenger, strict synchronization of the 2D and 3D cameras is not required. One constraint is that the camera's sampling rate must be high enough so that passenger movement does not result in erroneous position measurements. Experiments have found that a lower limit for the sampling rate is approximately 5 fps.

[0060] In step 202, for each frame of high-resolution 2D image data 104, an object detection algorithm is applied, constraining it to a 3D region calculated using the 3D image data 106 determined to contain an object. Co-registration of the 2D object detection data with the 3D data provides 3D position information for the object—here, face—region. This generates a list of high-resolution images of detected objects (e.g., high-resolution face images) and their positions at each instant in time.

[0061] In step 204, position information is recorded to be provided to the tracking module in step 206. There are several ways in which the position information can be recorded. A series of images extracted from the 3D image data may be recorded, and the tracking module then analyzes the images to determine a trajectory of movement of the object image in the series of images, for example, by analyzing relative pixel displacements of the object image. Alternatively, a 2D camera may be triggered to take images at regular intervals and / or in response to real-time analysis of the 3D images when it is determined that any movement has occurred. Thus, the stored data may be 3D images, 2D images, or simply position information of specific features of the images, or a combination thereof. The stored data may be any data related to the images that can be used to track movement in the series of images.

[0062] In step 206, data at t=tn is provided to the tracking module 112 to update 206 the tracking data to generate 208 a track list of objects (faces) with trajectories.

[0063] In step 208, the track list of objects with trajectories 116 will be provided to the discontinuity analysis module 114. Typically, there will be one track list, especially in scenarios where the objects (passengers) are single-file. The discontinuity analysis module 114 determines if there have been discontinuities in any of the trajectories and removes the object from the stack if necessary. The detection of discontinuities may be based on changes between data samples meeting or exceeding a threshold, or on another statistical attribute of the data, such as detection speed or specificity. This prevents the biometric boarding system from stalling due to confusion over the identity of the subjects.

[0064] FIG. 3(1) illustrates an example depth history of a tracked object, including discontinuities. The horizontal axis shows a series of times, from t=t1 to t=t8. The vertical axis shows depth information calculated from the trajectory and for the facial images acquired at t=t1 through t8. The depth information is the object's distance (measured in millimeters or "mm") from the image capture device at each instant the image is acquired. From the first time frame or instance (t=t1) to the sixth time frame (t=t6), there is a decrease in distance as subject A walks toward the image capture device. Subject A is biometrically matched to the passenger list (a unique tracker identifier is created for subject A as described above), and at t=t7, subject A turns around and walks away. Subject B is standing nearby behind subject A and is now visible in the field of view (FOV) of the image capture device before subject A completely leaves the FOV. The image capture device therefore captures a blurred image of subject A walking away, as well as a facial image of subject B. The system then treats the image of subject B as that of subject A. The captured data for subject B is now included in the same track trajectory as subject A. However, because subject B is standing some distance behind subject A, the distance data jumps from approximately 330 mm at t=t6 to approximately 770 mm at t=t7. A track trajectory discontinuity is recorded as occurring at t=t7. A notification is generated that a new object is now likely in the image capture device's FOV, and image processing resumes, creating a new tracker identifier for subject B and no longer treating the image of subject B as that of subject A, but recognizing it as that of the new object.

[0065] Figure 3(2) shows the history of facial area size calculated from the tracked face in each image sample in the sequence of images contained in the tracking data. The horizontal axis shows the time sequence from t=t1 to t=t10. The vertical axis shows the size of the facial area (in square millimeters or "mm 2 "). The raw calculated facial area data points are represented by squares. The circles represent data points that result from applying a filter to the raw data, e.g., to smooth the data and remove insignificant discontinuities (e.g., those below a variance threshold). At t=t6, approximately 6300 mm2 Approximately 5800mm 2 There is a slight decrease in the facial area of ​​the detected image to t = t5. This decrease is small enough that it does not trigger a declaration that there is a discontinuity. The small decrease can be attributed to the movement of the target occupant between t = t5 and t = t6. However, at t = t7, there is a decrease of about 2000 mm 2 There is a larger decrease in facial area in the detected image to t, which is greater than the expected change in facial area size for the same person (i.e., the threshold). Therefore, a discontinuity in facial area size is detected at t=t. Figure 3(2) therefore shows an example in which the discontinuity at t=t is also (or alternatively) confirmed using facial area discontinuity detection, supporting the trajectory data shown in Figure 3(1).

[0066] The threshold amount of change in facial area size required for a discontinuity to be detected may be determined using several factors, such as statistics related to facial size, expected movement speed, and sampling rate (i.e., expected time lapse between two image samples).

[0067] 4 schematically depicts one example of a process 400 utilized by the discontinuity analysis module 114. At the beginning of process 400, the system has not detected any discontinuities and no track lists have been processed (stage 402). At this stage, each track list is presumed to contain a series of image data and trajectories relating to the same object and has one unique track identifier assigned to it.

[0068] The discontinuity analysis module 114 will process 404 each track list in turn to detect 406 whether there are any discontinuities in the track list. As explained, any detected discontinuities are an indication of a stacking error, requiring the object to be de-stacked by assigning it a new unique track identifier.

[0069] If there are no available track lists or the system has processed all of the available track lists without detecting a discontinuity (408), the discontinuity analysis module 114 terminates its processing (410) without assigning new unique track identifiers to any of the tracked objects (426).

[0070] If there is at least one available track list (412), the system processes the next available list by extracting its trajectory (step 414) to determine changes in 3D position over time. The trajectory is analyzed by a discontinuity detection algorithm (416) to detect (418) the presence of discontinuities in the data. The discontinuities may be in the tracked object's trajectory, or in another statistic or metric related to the tracked object, or in a combination.

[0071] If no discontinuity is detected (420), the system processes the next track list (404). If a discontinuity is detected (422), that track list will be added to a record (424). A record is provided to store data about any tracked object for which a discontinuity is detected. The record may be a database or a linked list, or it may have another data structure. The system then continues to process the next track list (404).

[0072] If there are no more tracking lists to process (408), the system will check the records described above to see if any of the processed lists are identified as being for a tracked object containing a data discontinuity. Each tracked object contained in the record will be assigned a new tracking identifier to disambiguate the tracked object's identity. The new tracking identifier will be different from the tracking identifier associated with any previously boarded passenger.

[0073] In a preferred embodiment, the assignment of a new tracking identifier to a tracked object will also trigger the image capture device to capture new images of the tracked object, which will be associated with the newly assigned tracking identifier and passed to biometric matching module 118 (see FIG. 1).

[0074] In the context of biometric boarding, the system would now recognize that the newly assigned tracking identifier did not biometrically match any subject in the reference list and could perform a check to allow or deny boarding by the tracked subject.

[0075] 5 schematically depicts an example discontinuity detection algorithm (500). At step 502, the discontinuity detection algorithm receives as input a trajectory obtained from a track list. In this particular embodiment, an algorithm is provided to evaluate discontinuities in a time series of depth data associated with acquired images. Thus, at step 504, a time series of depth data is determined or extracted from the trajectory. The amount of data extracted may be limited to recent history, for example, a very recent time period. The time period may be a few seconds, for example, 3 seconds.

[0076] In step 506, the data determined from step 504 is smoothed by applying a filter. This may be done to remove the effects of noise, but it is not a required step. If this step is included, the filter applied is preferably suitable for preserving discontinuities, which would be expected to be high frequency components in the signal. As such, the filter may need to be specifically designed to suit the type of signal being processed while retaining the ability to detect discontinuities.

[0077] At step 508, a dissimilarity score or discontinuity score is calculated at each time sample. This may simply entail calculating the difference between the data between two consecutive time samples. At step 510, the algorithm determines whether the dissimilarity score satisfies a discontinuity condition. The determination may be based on a threshold, where the algorithm may say that if the dissimilarity score satisfies the discontinuity condition, it exceeds the threshold, or alternatively, determines that a discontinuity has occurred if the dissimilarity score is at least equal to the threshold. For example, the calculations at steps 508 and 510 are: Δt n =Xt n -Xt n-1 It looks like this, and in the formula, Δt n is time t=t n is the dissimilarity score at time t, Xt is the value of the data sample at time t, and Xt n-1 is time t=t n-1 is the value of the data sample at

[0078] Time t n The discontinuity condition at Δt n ≧threshold.

[0079] The threshold may depend on the expected noise range in the data, or the expected position change of the person's position between two or more image frames (and thus depend on the frame rate of the camera), or a combination of both.

[0080] Furthermore, the threshold may be several times the average depth of a person's head, or at least the average depth, knowing that it is highly unlikely that the second person in a row will have a head depth less than the first person behind them.

[0081] An alternative discontinuity condition could be the rate of change (i.e., velocity or speed) of distance z. If the sampling rate / frame rate of a 3D camera is measured in time samples tn, then a rate of change of distance z greater than, for example, 120 mm per frame or 120 mm / s or some other suitable value could mean that there is a discontinuity.

[0082] If the discontinuity condition is met (516), the algorithm outputs a result indicating that a discontinuity in the tracking data is detected (518). If the discontinuity condition is not met (512), the algorithm outputs a result indicating that a discontinuity in the tracking data is not detected (514). The algorithm ends at this point for the currently processed track list (520).

[0083] The above algorithm can also be used to detect discontinuities in information other than depth data, depending on the input provided to the discontinuity detection algorithm. For example, if the algorithm receives 2D images as input, the algorithm may determine the area of ​​the detected object in each of the 2D images in the tracking list (e.g., the facial area size in FIG. 3(2)) and detect discontinuities in the time series of areas by detecting whether the size of the area changes abruptly between samples. The area may alternatively be already calculated and then provided to the algorithm as input. For example, a discontinuity condition may be whether the change is outside the expected amount of variation due to expected position changes in frame rate, data noise, or slight changes due to the expected range of movement by the subject.

[0084] As another example, the algorithm may receive as input a biometric score calculated for each image in the tracking list, or the algorithm may be configured to determine a score when it receives an image as input. The algorithm can then determine whether there is a discontinuity in the time series of biometric scores.

[0085] In doing so, the captured facial images are compared for similarity to detect when one sample is no longer sufficiently similar to the previous sample. For example, a discontinuity condition may be whether the change between two consecutive data samples is outside the expected amount of variation, noise, or other acceptable within-subject variability.

[0086] More generally, therefore, the discontinuity determination algorithm may be used to determine discontinuities in trajectories, depths, or other statistics associated with images included in a track list.

[0087] In the above embodiment, tracked objects are assigned new unique tracking identifiers if their trajectories are determined to contain discontinuities.

[0088] However, in further embodiments, discontinuities may instead be determined based on other statistics or metrics about the captured image, such as object area, or facial biometrics if the captured object is a face.

[0089] In further embodiments, a combination of detecting discontinuities in the trajectory and detecting discontinuities in other statistics or metrics may be used. For example, a discontinuity algorithm may be required to verify that discontinuity conditions for both the trajectory and the size of the object area (e.g., the depth of the tracked face and the size of the tracked face) are satisfied for the tracked object in order to declare that a discontinuity exists for the tracked object. That is, in some embodiments, the system may apply a multi-modal constraint to its discontinuity detection algorithm. Before declaring that a discontinuity exists for the tracked object, the system may first verify that the discontinuity condition for the trajectory is satisfied, and then verify whether one or more discontinuity conditions for the other metrics or statistics are satisfied.

[0090] 6 schematically depicts a biometric boarding device 600. The biometric boarding device 600 includes a processing device 602, which may be a central processing unit or another processor, configured to execute machine instructions to implement one or more systems of the above-described embodiments. The machine instructions may be stored in a memory device 604 co-located with the processing device 602, or they may reside partially or entirely in one or more remote memory locations accessible by the processing device 602. The processing device 602 also accesses a data storage 606, which is adapted to contain data to be processed and, possibly, to at least temporarily store results from the processing.

[0091] There may also be a communications module 608 so that the device can access the data provided wirelessly or communicate data or results over a communications network 612 to a computer at a remote location, such as a monitoring station or cloud storage 613 .

[0092] Box 610 conceptually indicates that the components therein may be provided in the same physical device or housing. Box 610 is shown with dashed lines to indicate that one or more components may instead be provided separately, or that other components (such as image capture devices, or input and / or output devices) may also be physically included in the same device, or both.

[0093] The processing device 602 is in communication, either wired or wireless, with a data capture device 614 to capture the data required for processing.

[0094] As mentioned previously with respect to the system module, in a preferred embodiment for facial biometric boarding, the data capture device 614 is configured to capture 2D images of sufficient resolution to enable facial recognition, and also 3D images to provide trajectory data.

[0095] 7 shows an example "pod" type boarding device 700. It should be noted that this is merely an illustrative example and that the present invention may be embodied in other types of devices. Devices may also generally have different physical shapes or have different hardware requirements depending on the specific purpose for which the device is provided (e.g., in an airport context - baggage drop or check-in, customs control, boarding).

[0096] The boarding device 700 includes a device housing 702 and a biometric authentication system 704. The biometric authentication system 704 comprises a first image capture device 706 and a second image capture device 708. The first image capture device 706 may be in the form of a depth camera or a stereo 3D camera. The second image capture device 708 may be in the form of a high-resolution 2D camera. A display 710 may also be included. Optionally, a laser projector or infrared (IR) sensor / emitter 712 is provided in the biometric authentication system 704 or elsewhere within the device 700. The laser projector or IR sensor / emitter may optionally be located behind the display 710.

[0097] A set of illumination means or lights 714 may be mounted on the front 701 of the device 700. A laser projector or point cloud generating device (not shown) may also be provided to assist in mapping and detection of objects. Optionally, the device 700 may be a portable device that can be used to determine a person's identity. The point cloud generating device projects a group or multiple light-based dot points (including, but not limited to, infrared LED projected dots) onto the upper torso region of a user or passenger for the purpose of region of interest detection, biometric analysis, and facial topographic mapping by the system. If a point cloud generating device is included, data from the point cloud generating device will also be provided to the biometric authentication system, and the biometric analysis or topographic mapping data may also be used for discrete analysis.

[0098] The embodiments described herein therefore provide a system that can facilitate irregular boarding, i.e., where subjects (passengers) do not strictly adhere to the required distance separation of the regular boarding process. The tracking process also allows the system to maintain separate tracking identifiers for a group of passengers despite slight occlusions or motion blur that may occur in the images.

[0099] The described system thus allows for minimizing the amount of biometric processing that needs to be done by only sending good images of subjects with unique tracking identifiers, without confusing them with neighbors. It should be noted that while such confusion does not occur often, even a small percentage of such occurrences will result in large outliers in the data (time required to board a passenger) due to the additional processing steps required to retry or manually process the passenger.

[0100] The ability of a system that requires using discontinuities in the trajectory of a tracked target (or, more precisely, a tracked object) that occur because of the inherent measurable physical distance between two people depends on a sufficiently fast sampling rate. Thus, the system's ability to resolve confusion between two targets that are too close together will also depend on the sampling rate. A higher sampling rate will enable the system to separate targets that are close together.

[0101] The described system therefore offers technical advantages over known systems that require the boarding process to be controlled by the use of trained staff or signage processes to ensure spacing is enforced, as well as software that minimizes potentially unnecessary identification and departure control system operations.

[0102] In the prior art, to minimize stacking problems, biometric cameras are sometimes placed at an angle to the queue of passengers or customers, and each passenger is required to turn away from the queue direction and walk or otherwise move toward the camera to minimize the possibility of confusion with the faces of other passengers in the queue. However, requiring each passenger to turn toward the camera incurs a time cost. In contrast, the present system, as described, is designed to minimize stacking errors using discontinuity detection rather than angular placement of the camera relative to the queue.

[0103] Because the solution provided by the present system does not require such an angled placement, the camera for the present system can be placed directly in front of the queue of passengers waiting to board. FIG. 8 depicts this direct placement, which one skilled in the art would understand is enabled by the system described herein with reference to FIGS. 1-7. As shown in FIG. 8, camera 802 (i.e., data capture device) is placed directly in front of passenger queue 804, so that passenger 806 being processed does not have to turn away from the direction of the queue (arrow 812) to enter camera 802's field of view 810. While other passengers 808 waiting in queue 804 may also turn toward camera 802 as they proceed or wait, the system could handle the detection of another face within field of view 810, using a discontinuity detection algorithm to resolve stacking errors if necessary.

[0104] The system may provide an interface, which may be a graphical user interface, to be displayed on a screen and viewed by passengers, for example, to provide boarding instructions. The interface may also be provided on a separate display device for viewing by staff members to monitor the live boarding process, to review the completed boarding process, or for other purposes, such as training. The interface shown to staff may fully or partially mirror that (if any) shown to passengers. It may include interface portions that display other data not shown to passengers, such as one or more of the following non-limiting examples: passenger trajectories, values ​​of certain metrics being monitored to detect discontinuities, unique tracking identifiers, or statistics such as number of people boarded, current average or total elapsed processing time, successful processing, and load factor.

[0105] 9 depicts an example interface 900. The interface 900 includes a first display field 902 that may show an image of the passenger being processed, captured by a camera at a particular time or all the time. The first display field 902 may mirror the interface shown to the passenger (see FIG. 10 ), in which case the display field 902 may also display instructions to the passenger, which may be in the form of text or graphics (e.g., animations).

[0106] In FIG. 9 , a first display field 902 shows an image of a passenger. Box 904 (dotted line) marks the area of ​​the image that is being processed by an algorithm to detect the presence of a face. Box 906 marks the area of ​​the image where a face is detected. In this example, the system tracks the location of the center of the detected face and displays a trajectory 908 of the tracked face. In some embodiments, the trajectory 908 will be displayed in a different color or visual style depending on the operation of the access control system. For example, the trajectory 908 may be displayed in a default color or style once tracking of an object begins. However, if the system detects an inconsistency in the tracking identifier associated with the trajectory—i.e., a discontinuity is detected—it will change to a different color or style to provide a visual indication of the error.

[0107] The interface 900 may display a time series 910 of the three-dimensional position or depth (i.e., distance to the camera) of the tracked face. Additionally or alternatively, the interface 900 may display at least one other time series 912 that is a time series of values ​​of another metric (such as face area size) determined over time if the algorithm has identified a discontinuity in the values ​​for that metric.

[0108] A warning display or warning field 914 may be included to display a warning if the algorithm detects an error. The interface 900 may include one or more data display areas 916, which may be in the form of a display portion, window, or access field or button for displaying or providing access to particular data. The data display 916 may be a passenger identifier display that shows the unique tracking identifier of the passenger currently being processed. The data display 916 may also provide statistics regarding individual, overall, or average processing times for passengers.

[0109] The above portions of the interface do not all need to be provided at the same time, for example, a dashboard may allow a user (e.g., airport staff or system developers) to select which interface portions to include for showing on the device screen.

[0110] Figures 10(1)-10(4) depict various displays that may be shown on the passenger interface for viewing by the passenger. Figure 10(1) shows a screen from which the passenger is instructed to walk toward the camera. Figure 10(2) shows a progress screen instructing the passenger to look at the biometric camera while a photograph of the passenger is taken. A camera view of the passenger may be shown inside a progress circle 1002. Figure 10(3) shows a screen confirming the passenger has been matched and prompting the passenger to board. A seat number 1004 is shown, providing the passenger with an opportunity to confirm that it matches their assigned or selected seat as shown on their boarding pass. Other information, such as the passenger's name, may also be shown for verification purposes. Figure 10(4) shows a screen displayed when the biometric boarding process is unable to board the passenger for any reason and shows instructions directing the passenger to seek assistance from staff.

[0111] The disclosed embodiments therefore aim to provide a tangible improvement in turnaround time by facilitating the irregular boarding process.

[0112] It will be appreciated that the above system is not limited to being deployed at boarding. It may alternatively or additionally be used within a facility, or at another checkpoint (such as an exit) for a vehicle or transportation mode. A facility with multiple exit or entry points may each have separate devices (whether they are "pod" type devices or in other forms) that implement an instance of the above process.

[0113] It should further be noted that in applications where the actual identities of individuals entering or leaving an area are not verified, but rather it is the number of people within an area that is tracked, the identification aspect of the above process and system may be omitted or disabled. By looking at how many different "tracking identifiers" there are, the process can determine the number of different people who have been processed, from regular or "irregular" queues, to enter the area, exit the area, or generally move through a checkpoint.

[0114] For example, a device embodying the above system that has been used to board passengers onto an aircraft can be used again at an exit checkpoint to "de-board" passengers. Processing de-boarding passengers may include queuing the passengers to be processed again by the system. New images of the passengers captured at the processing checkpoint may be matched against a passenger database. The passenger database may be a reference database comprising "best images" captured while the passenger was being processed by the boarding system that biometrically match the passenger and are then selected to board. The "best image" reference data may be stored locally on the device, such that biometric matching of the newly captured image and the "best image" reference can occur even when there is no connectivity to a central passenger reference database.

[0115] The device may have its biometric identification module completely disabled and simply count the number of different passengers who disembark or move through an exit checkpoint (e.g., at a gate or other restricted arrival area within an airport), in which case an alert may be issued if the number of people arriving from the aircraft differs from the number of people who originally boarded the aircraft.

[0116] Modifications and variations may be made in the foregoing without departing from the spirit or scope of the present disclosure. For example, in the above, embodiments utilizing biometric matching determine whether to grant access or approve passage by a subject (e.g., a passenger) by matching to see if there is a match between the subject and a reference database. However, the same inventive concepts may be applied to situations where the system determines when to deny access or passage. For example, the system would still use the same discontinuity detection to attempt to reduce stacking error. However, the biometric matching module could attempt to match the image of the passenger or subject being processed against a "no access" list, and if a match is found on the list, access by that passenger or subject could be denied. By simply counting the unique identifiers assigned to subjects (or "faces," in the case of facial recognition) that are granted or have gained access, the system can keep track of how many subjects have been approved.

[0117] As mentioned above, the system requires 2D images with a high enough resolution for face or object detection, as well as position data for tracking the detected face (or object). Although the use of two capture devices is described in the embodiments, this does not form a requirement that constrains the spirit and scope of the present invention. For example, if both the 2D image data and the spatial position data come from a 3D camera, and the 3D camera can capture images with a high enough resolution to allow face or object detection and matching, the same inventive concept would still be met, and the method implemented by the system would still function in the same way.

[0118] In the claims that follow and in the foregoing description of the invention, unless the context requires otherwise by express language or necessary implication, the word "comprise" or variations such as "comprises" or "comprising" are used in the inclusive sense, i.e., to specify the presence of stated features in various embodiments of the invention but not to exclude the presence or addition of further features. [Explanation of symbols]

[0119] 100 Biometric Boarding System 102 Processing Module 104 First Data 106 Second Data 108 Measurement Data 110 Detection Module 112 Tracking Module 114 Discontinuity Analysis Module 116 Trajectory Data 118 Biometric Matching Module 600 Biometric Boarding Device 602 Processing Device 604 Memory Devices 606 Data Storage 608 Communication Module 610 Box 612 Communication Network 613 Cloud Storage 614 Data Capture Device 700 "Pod" type boarding device 701 Front 702 Device housing 704 Biometric Authentication System 706 First Image Capture Device 708 Second Image Capture Device 710 Display 712 Laser Projector or Infrared (IR) Sensor / Emitter 714 Lighting means or lights 802 Camera Column 804 806 passengers 808 passengers 810 field of view 812 Arrow 900 Interface 902 First Display Field 904 Enclosure 906 Box 908 Trajectory 910 Time Series 912 Time Series 914 Warning display or warning field 916 Data display area 1002 Progress Circle 1004 seat number

Claims

1. 1. A method for controlling entry into a predetermined area for at least one tracked object, comprising: acquiring or receiving a series of two-dimensional images acquired by a 2D camera in which the at least one tracked object is presumably captured, and also position data for the at least one tracked object based on image data acquired by a 3D camera, wherein when the image data acquired by the 3D camera determines that the at least one tracked object is in the correct orientation, the series of two-dimensional images are acquired and position data for the at least one tracked object in the series of two-dimensional images is acquired; assigning a unique tracking identifier to the at least one tracked object; determining a trajectory of the at least one tracked object from the position data; determining whether there is a discontinuity in the trajectory or data calculated from the trajectory, and if a discontinuity is detected, acquiring or receiving one or more new images of the at least one tracked object after detecting the discontinuity and assigning a new unique tracking identifier to the at least one tracked object; if a discontinuity is detected, based on at least one of the one or more new images, or if no discontinuity is detected, based on at least one image from the series of two-dimensional images, matching the at least one tracked object with objects in a reference list to determine whether entry into a predetermined area should be permitted; A method comprising:

2. The method of claim 1 , wherein determining whether there is a discontinuity comprises determining whether a distance discontinuity condition or a velocity discontinuity condition is satisfied by a time series of distance or velocity data obtained from the trajectory.

3. 3. The method of claim 2, wherein the distance discontinuity condition or the velocity discontinuity condition is 1) whether the difference between two data samples in the time series exceeds a threshold, or 2) whether it is equal to or greater than a threshold.

4. The method of claim 3 , wherein the threshold value depends at least in part on the elapsed time between the times the two data samples were taken.

5. 5. The method of claim 1, further comprising providing a time series of a statistic or metric calculated from the trajectory or the two-dimensional image, and determining whether a statistic or metric discontinuity condition is satisfied by the time series of the statistic or metric.

6. The method of claim 5 , further comprising the step of verifying whether the statistical or metric discontinuity condition is satisfied before determining whether there is a discontinuity in the trajectory or data calculated from the trajectory.

7. A method for controlling entry to a predetermined area by biometric authentication, comprising the method of any one of claims 1 to 6, wherein the at least one tracked object is a biometric.

8. The method of claim 7 , being a facial biometric control method, wherein the at least one tracked object is a facial area of ​​a person.

9. 9. The method of claim 8, which indirectly relies on claim 5, wherein the statistic or metric is a facial area size or a biometric score calculated from the two-dimensional image.

10. A method according to any one of claims 1 to 9, wherein the step of acquiring or receiving the series of two-dimensional images and also position data relating to the at least one tracked object includes a step of applying an object detection algorithm to the three-dimensional images to detect one or more objects, each object being one of the at least one tracked object.

11. The method of claim 1 , wherein the position data comprises at least depth data.

12. 1. A method for counting the number of times a checkpoint crossing occurs by one or more tracked objects, comprising: processing passages by each tracked object and acknowledging passages by said tracked object; and processing a passage by each tracked object and acknowledging the passage by the tracked object, acquiring or receiving a series of two-dimensional images of the tracked object, the series of two-dimensional images being acquired by a 2D camera, and also position data for the tracked object based on image data acquired by a 3D camera, wherein when the image data acquired by the 3D camera indicates that the tracked object is correctly oriented, the series of two-dimensional images are acquired and position data for each of the tracked objects in the series of two-dimensional images is acquired; assigning a unique tracking identifier to the tracked object; determining a trajectory of the tracked object from the position data; determining whether there is a discontinuity in the trajectory or data calculated from the trajectory, and if a discontinuity is detected, acquiring one or more new images of the tracked object after detecting the discontinuity and assigning a new unique tracking identifier to the tracked object; Including, A method wherein determining the number of times the checkpoint crossing has occurred includes counting the number of different unique tracking identifiers assigned to the tracked object.

13. 13. The method of claim 12, wherein the step of authorizing passage comprises matching the tracked object to targets in a reference list to determine whether passage should be allowed based on at least one of the one or more new images if a discontinuity is detected, or based on at least one image from the series of two-dimensional images if no discontinuity is detected.

14. 1. A method for counting the number of times a checkpoint crossing occurs by one or more tracked objects, comprising: acquiring or receiving a series of two-dimensional images, acquired by a 2D camera, in which at least one tracked object is presumably captured, and also position data for at least one portion of the at least one tracked object based on image data acquired by a 3D camera, wherein when the image data acquired by the 3D camera determines that the at least one tracked object is in the correct orientation, the series of two-dimensional images are acquired and position data for the at least one tracked object in the series of two-dimensional images is acquired; assigning a unique tracking identifier to the at least one tracked object; determining a trajectory of the at least one tracked object from the position data; determining whether there is a discontinuity in the trajectory or data calculated from the trajectory, and if a discontinuity is detected, acquiring or receiving one or more new images of the at least one tracked object after detecting the discontinuity and assigning a new unique tracking identifier to the at least one tracked object; if a discontinuity is detected, based on at least one of the one or more new images, or if no discontinuity is detected, based on at least one image from the series of two-dimensional images, matching the at least one tracked object with targets in a reference list to determine whether it should be allowed to pass; incrementing said count if it is determined that passage is permitted; A method comprising:

15. 15. A method of controlling or monitoring entry to a premises or means of transportation having one or more entry points, or one or more exit points, or both, comprising implementing the method of any one of claims 1 to 14 at at least one of the entry points, at least one of the exit points, or both.

16. A method for controlling entry to a predetermined area by facial biometric authentication, according to any one of claims 1 to 15, wherein the tracked object is a facial area of ​​a person.

17. 16. A computer readable medium having stored thereon machine readable instructions, the machine readable instructions being adapted, when executed, to perform the method of any one of claims 1 to 15.

18. 16. A system for controlling entry to a predetermined area, comprising a processor configured to execute machine-readable instructions that, when executed, are adapted to perform a method according to any one of claims 1 to 15.

19. 20. The system for controlling entry to a predetermined area according to claim 18, comprising image capture devices, said image capture devices comprising a two-dimensional image capture camera and a three-dimensional image capture camera.

20. 20. The system for controlling entry to a predetermined area according to claim 19, wherein, in use, the image capture device is disposed in front of a queue of objects to be processed by the system for controlling entry to the predetermined area.

21. 21. The system for controlling entry to a predetermined area according to any one of claims 18 to 20, which is a biometric boarding system, wherein the tracked object is a living body.

22. 21. A method of controlling or monitoring entry into a predetermined area by one or more subjects standing in a queue by processing each subject in turn using a biometric area entry control system, wherein the biometric area entry control system includes an image capture device pointing directly towards the queue, the tracked object is a biometric, and the biometric area entry control system is the system of any one of claims 18 to 20.

Images