DETECTION, COUNTING AND IDENTIFICATION OF OCCUPANTS IN VEHICLES

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention

[0001] The present disclosure relates to facial detection and recognition, and more particularly to facial detection and recognition for occupants in vehicles.2. Description of Related Art

[0002] At security check points, border crossings, high occupancy vehicle (HOV) lanes, and the like it, is desirable to know how many occupants are in each vehicle that passes. At a traditional checkpoint an officer can count occupants that are visible in a vehicle. In security applications, it can be desirable to know who the occupants of a vehicle are. An officer can verify this by inspection of identification documents such as a photo ID for each occupant of the vehicle. However, these techniques require each vehicle to stop for inspection before passing through.

[0003] The conventional techniques have been considered satisfactory for their intended purpose.

[0004] ES 2 395 105 discloses a system to automatically control occupancy of vehicles at a toll station, the system comprising means for acquiring images of a vehicle to be controlled, means for processing the acquired images, and means for detecting faces inside the vehicle and counting said faces. A discount is then applied to the toll according to the number of validated occupants.

[0005] US 2012 / 0106806 discloses a method for performing face recognition in a video. Face detection data in frames of input data are used to generate face galleries, which are labeled and used in recognizing faces throughout the video. Metadata that associates the video frame and a face are generated and maintained for subsequent identification. Faces other than those found by face detection may be found by face tracking, in which facial landmarks found by the face detection are used to track a face over previous and / or subsequent video frames. Once generated, the maintained metadata may be accessed to efficiently determine the identity of a person corresponding to a viewer selected face.

[0006] CA 3010922 discloses a passenger counting system including: a first photographing means for photographing a vehicle and acquiring a first image; a second photographing means for photographing the vehicle in a state of receiving only light of a first specific wavelength band and acquiring a second image; and a passenger counting device. The passenger counting device includes: an image separation means for acquiring a plurality of separate images based on the first image and the second image; and a passenger number determination means for determining the number of passengers of the vehicle based on the plurality of separate images.

[0007] US 2013 / 0147959 discloses a method for determining a pixel classification threshold for vehicle occupancy determination. An IR image of a moving vehicle is captured using a multi-band IR imaging system. A driver's face is detected using a face recognition algorithm. Multispectral information extracted from pixels identified as human tissue of the driver's face is used to determine a pixel classification threshold. This threshold is then used to facilitate a classification of pixels of a remainder of the IR image. Once pixels in the remainder of the image have been classified, a determination can be made whether the vehicle contains additional human occupants other than the driver. An authority is alerted in the instance where the vehicle is found to be traveling in a HOV / HOT lane requiring two or more human occupants and a determination has been made that the vehicle contains an insufficient number of human occupants.

[0008] However, there is a need for improved systems and methods for detecting, counting, and identifying occupants in vehicles. The present invention provides a solution for this need.SUMMARY OF THE INVENTION

[0009] In an aspect of the present invention, there is provided a system according to appended claims 12-14.

[0010] In another aspect of the present invention, there is provided a method according to appended claims 1-11.

[0011] These and other features of the system and method of the present invention will become more readily apparent to those skilled in the art from the following detailed description of the preferred embodiments taken in conjunction with the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] So that those skilled in the art to which the present invention appertains will readily understand how to make and use the devices and methods of the present invention without undue experimentation, preferred embodiments thereof will be described in detail herein below with reference to certain figures, wherein: Fig. 1 is a schematic side elevation view of an exemplary embodiment of a system constructed in accordance with the present disclosure, showing multiple camera systems with an approaching vehicle; Fig. 2 is a schematic plan view of the system of Fig. 1, showing the positions of the camera systems; Fig. 3 is a schematic view of one of the camera systems of Figs. 1 and 2, showing the imaging sensor; and Fig. 4 is a schematic view of a method in accordance with the subject disclosure, showing a process using images acquired by the system of Fig 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Reference will now be made to the drawings wherein like reference numerals identify similar structural features or aspects of the present invention. For purposes of explanation and illustration, and not limitation, a partial view of an exemplary embodiment of a system in accordance with the present invention is shown in Fig. 1 and is designated generally by reference character 100. Other embodiments of systems in accordance with the present invention, or aspects thereof, are provided in Figs. 2-4, as will be described. The systems and methods described herein can be used for automated counting and identification of occupants in vehicles.

[0014] The system 100 for detecting occupants in a vehicle 102 includes a controller 104 and a plurality of camera systems 106, 108, and 110 that are external to the vehicle 105 in the vehicle approach area 112. Each camera system 106, 108, and 110 is operatively connected to the controller 104. A trigger 114 in the vehicle approach area 112 is operatively connected to the controller 104 to detect an approaching vehicle 105 and to control the camera systems 106, 108, and 110 to acquire images of the approaching vehicle 105. The controller 104 includes machine readable instructions configured to cause the controller 104 to perform any method as disclosed herein. As shown in Figs 1 and 2, each camera system 106, 108, and 110 is in a different location for acquiring images with sensors viewing the vehicle from different respective angles.

[0015] With reference now to Fig. 3, camera system 106 includes an imaging sensor 116, a pulsed illumination device 118, and a processor 120 operatively connecting the imaging sensor 116 to the pulsed illumination source 118 for synchronizing an illumination pulse from the pulsed illumination device 118 with exposure of the imaging sensor 116. The illumination device 118 can be located on camera as in camera system 106 shown in Fig. 3, or can be located off-camera as in camera systems 108 and 110 shown in Figs. 1-2. The camera system 106 include a lens 122 optically coupled to the imaging sensor 116, an optical bandpass filter 124 operatively connected to filter light passing through the lens 122 to the imaging sensor 116. The camera system 106 also includes a linear polarization filter 126 operatively connected to filter light passing through the lens 122 to the sensor 116, e.g., to reduce glare from glass windshields and windows of the vehicle 105. Imaging sensors 108 and 110 can include the same components as camera system 106.

[0016] With reference now to Fig. 4, a method of detecting occupants in a vehicle includes detecting an oncoming vehicle, e.g., detecting oncoming vehicle 105 using trigger 114 as shown in Fig. 1. When trigger 114 detects an oncoming vehicle 105, it signals the controller 104. Controller 104 then commands the camera systems 106, 108, and 110 to acquire a plurality of images of occupants in the vehicle 105. Each camera system 106, 108, and 110 can acquire a respective image 128, 130, and 132, forming a set 200 of acquired images as shown in Fig. 4. Controller 104 can illuminate the vehicle 105 with a respective pulse of illumination from each respective illumination device 118 for each image acquired, wherein each pulse of illumination is performed at a different time to reduce shadows cast onto the occupants while acquiring the plurality of images.

[0017] The method includes having controller 104 perform automated facial detection on the plurality of images 128, 130, and 132, and to add a facial image for each face detected to a gallery 202 of facial images for the occupants of the vehicle 105. For the image 128, three faces are detected and four faces are detected from each of images 130 and 132. Controller 104 performs automated facial recognition on the facial images of gallery 202 to group the facial images into groups 134, 136, 138, and 140 based on which occupant is in the respective facial images, as indicated by facial recognition groupings 204 in Fig. 4. While multiple images are shown in Fig. 4 in each of the groups 134, 136, 138, and 140, it should be understood that, in an alternative example not covered by the claims, the groups 134, 136, 138, and 140 need not ever actually contain multiple images in each group. For example during facial recognition, each time a new image of a given individual is identified, controller 104 can decide whether the new image is better than the previous best image of the individual (based on facial detection confidence scores, facial offset angle, or the like, as described below), and if so the new image replaces the previous image in the respective group. In this way each group 134, 136, 138, and 140 only ever includes one image.

[0018] Facial detection and facial recognition need not necessarily be performed one after another, but instead can be performed together on the fly. One of the sensors 120 can be a primary sensor, e.g., the sensor 120 of camera system 106, that acquires a primary image, e.g., image 128, of occupants in the vehicle 105. The faces detected in primary image 128 can serve as references in the gallery 202 for facial recognition for subsequent ones of the images 130 and 132 of occupants in the vehicle. The controller 104 can add a new face to the gallery 202 each time a detected face in a subsequent one of the images 130 and 132 does not match with a face already in the gallery 202. The controller 104 can iteratively compare faces detected in subsequent ones of the images 128, 130, and 132 and add each face detected to the gallery 202 that is not already in the gallery 202 until there is an image in the gallery 202 of each face detected by performing automated facial detection.

[0019] Whenever a face is detected for which there is already an image in the gallery 202, the best image of the face can be retained in the image gallery 202. Controller 104 selects a representative image 142, 144, 146, and 148 from each group 134, 136, 138, and 145 and outputs a set 206 of cropped selected images, one uniquely cropped selected image for each of the occupants. Set 206 includes no duplicate images, i.e. no more than one image is in set 206 for a given occupant, so duplicate images of a given occupant need be stored or displayed. The controller 104 can select the representative image 142, 144, 146, and 148 from each group 134, 136, 138, and 140 by selecting images based on corresponding confidence scores from the automated facial detection. It is also contemplated that controller 104 can select the representative image 142, 144, 146, and 148 from each group 134, 136, 138, and 140 by selecting images based on which image in the group has least facial offset angle from line of sight of the imaging sensor 120 which acquired the respective image. This selection process can be run on the fly with facial detection and facial recognition to winnow the gallery 202 down to the set 206.

[0020] The controller 104 can determine how many occupants are in the vehicle 105 by counting the groups 134, 136, 138, and 140. In this example, there are four groups 134, 136, 138, and 140 indicating there are four occupants in the vehicle 105. If groups 134, 136, 138, and 140 are conflated down to the set 206 on the fly as described above, then the groups 134, 136, 138, and 140 can be counted indirectly by simply counting the final cropped images in set 206 to determine how many occupants are in the vehicle 105.

[0021] The controller 104 can output the number of occupants in the vehicle 105, and can provide other output actions as needed. For example, controller 104 can initiate a response, e.g., via the output device 150, upon determining an improper number of occupants in the vehicle. For example, if controller 104 determines there are not enough occupants in a vehicle in an ROV lane, controller 150 can use the output device 150 to output an alert on a visual display, sound an audible alarm, close a physical barrier, transmit a citation, mail a citation, update a database, and / or dispatch an officer.

[0022] It is also contemplated that with the set of images 206, controller 104 can run the final cropped facial images through a facial recognition database, either locally or remotely, to check for matches between the occupants and known individuals in the database. If a match is found, e.g., one of the occupants in the vehicle 105 is on a watch list, the controller 104 can initiate an output response, e.g., using output device 150, such as outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, updating a database, and / or dispatching an officer.

[0023] While shown and described herein in an exemplary context where there are n=3 camera systems and m=4 occupants in the vehicle 105, those skilled in the art will readily appreciate that any suitable number n of camera systems can be used, and any suitable number m of occupants in a vehicle can be counted / identified without departing from the scope of the invention as defined by the claims.

[0024] The methods and systems of the present invention, as described above and shown in the drawings, provide for counting and identifying occupants in vehicles with superior properties including reliable, automated detection and identification of all occupants in a moving vehicle. While the apparatus and methods of the present invention have been shown and described with reference to preferred embodiments, those skilled in the art will readily appreciate that changes and / or modifications may be made thereto without departing from the scope of the invention as defined by the claims.

Claims

1. A method of detecting occupants in a vehicle (105) using a system (100) comprising a first camera system (106) and a second camera system (108), the first and second camera systems being external to the vehicle in a vehicle approach area (112), the method comprising: detecting an oncoming vehicle; using the first camera system and the second camera system, acquiring (200) a plurality of images (128, 130) of occupants in the vehicle in response to detection of the vehicle, wherein the first camera system acquires a first portion (128) of the plurality of images from a first angle, wherein the second camera system acquires a second portion (130) of the plurality of images from a second angle, and wherein the first angle is different from the second angle; characterized by: performing automated facial detection on the plurality of images and adding a facial image for each face detected in the plurality of images to a gallery (202) of facial images for the occupants of the vehicle, wherein the gallery of facial images comprises at least a first facial image of at least one of the occupants taken from the first angle and a second facial image of said at least one of the occupants taken from the second angle; performing automated facial recognition on the gallery of facial images to form facial groupings (134, 136, 138, 140), wherein each of the facial groupings comprises facial images of only one of the occupants, and wherein one of said facial groupings comprises said first and second facial images of said at least one of the occupants; selecting a representative facial image from each of the facial groupings; outputting the representative facial images selected from each of the facial groupings, each of the outputted representative facial images being representative of only one of the occupants; and counting the facial groupings to determine how many occupants are in the vehicle.

2. The method as recited in claim 1, wherein no multiple representative facial images of a given occupant are stored or displayed.

3. The method as recited in claim 1, wherein selecting the representative facial image from each of the facial groupings includes selecting the representative facial image based on corresponding confidence scores from the automated facial detection.

4. The method as recited in claim 1, wherein selecting the representative facial image from each of the facial groupings includes selecting the representative facial image based on which facial image in the facial grouping has least facial offset angle from line of sight of the camera system which acquired the respective image.

5. The method as recited in claim 1, further comprising running the selected representative facial images through a database to check for matches between the occupants and known individuals in the database.

6. The method as recited in claim 5, further comprising initiating a response upon finding a match in the database, wherein the response includes at least one of outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, and / or dispatching an officer.

7. The method as recited in claim 1, further comprising initiating a response upon determining an improper number of occupants in the vehicle, wherein the response includes at least one of outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, and / or dispatching an officer.

8. The method as recited in claim 1, further comprising illuminating the vehicle with a respective pulse of illumination for each image acquired, wherein each pulse of illumination is performed at a different time to reduce shadows cast onto the occupants while acquiring the plurality of images.

9. The method as recited in claim 1, wherein one of the first and second camera systems is a primary camera system that acquires a primary image of occupants in the vehicle, and wherein faces detected in the primary image serve as references in the gallery for facial recognition for subsequent ones of the images of occupants in the vehicle.

10. The method as recited in claim 9, further comprising adding a new face image to the gallery each time a detected face in a subsequent one of the images of occupants in the vehicle does not match with a face in a facial image already in the gallery.

11. The method as recited in claim 9, further comprising iteratively comparing faces detected in subsequent ones of the images of occupants in the vehicle and adding a facial image of each face detected to the gallery that is not already in the gallery until there is a facial image in the gallery of each face detected by performing automated facial detection.

12. A system (100) for detecting occupants in a vehicle comprising: a controller (104); a plurality of camera systems (106, 108, 110) external to the vehicle in a vehicle approach area (112), each camera system operatively connected to the controller, wherein the plurality of camera systems includes at least a first camera system and a second camera system configured to acquire images from a first angle and a second angle, respectively, the first angle being different from the second angle; and a trigger (114) in the vehicle approach area and operatively connected to the controller to detect an approaching vehicle (105) and control the camera systems to acquire images of the approaching vehicle; wherein the controller is configured to perform any one of claims 1 to 11.

13. The system as recited in claim 12, wherein each camera system includes an imaging sensor (116), a pulsed illumination device (118), and a processor (120) operatively connecting the imaging sensor to the pulsed illumination device for synchronizing a pulse of illumination from the pulsed illumination device with exposure of the imaging sensor.

14. The system as recited in claim 13, wherein each camera system includes a lens (122) optically coupled to the imaging sensor, an optical bandpass filter (124) operatively connected to filter light passing through the lens, and a linear polarization filter (126) operatively connected to filter light passing through the lens.