A vehicle cabin monitoring system and method

Abstract

Described herein are systems and methods of monitoring an environment such as a vehicle cabin. In one embodiment, a system (100) comprises one or more cameras (110) configured to capture images of the vehicle cabin in the visible and / or infrared wavelength range. The vehicle cabin comprises one or more reflective regions (152, 154, 156) having properties such that at least a portion of visible and / or infrared radiation is reflected therefrom. System (100) also comprises a processor (124) configured to process the images to identify an object within the vehicle cabin that is visible in reflections from the one or more reflective regions (152, 154, 156). The processor is further configured to receive input indicative of the geometry of the vehicle cabin and process images relating to the identified object visible in reflections in conjunction with the geometry of the vehicle cabin to determine characteristics of the object in three dimensions.

Description

A VEHICLE CABIN MONITORING SYSTEM AND METHODFIELD OF THE INVENTION

[0001] The present application relates to monitoring systems and in particular to image processing in monitoring systems to view areas of an environment not otherwise viewable directly by a camera.

[0002] Embodiments of the present invention are particularly adapted for vehicle cabin monitoring systems for monitoring a state of a driver and / or occupants of the vehicle. However, it will be appreciated that the invention is applicable in broader contexts and other applications.BACKGROUND

[0003] Camera monitoring systems are typically limited by the field of view of the camera(s). Conventional solutions to improve the field of view of a monitoring system include adding additional cameras with different fields of view, providing capability to pan or move one or more cameras or adding wide angle capability to a camera.

[0004] Vehicle cabin monitoring systems such as driver and occupant monitoring systems are unable to leverage many of these solutions due to strict size and cost restraints within vehicles.

[0005] Separately, many modern vehicles incorporate infrared reflective glass to reflect a portion of infrared radiation while transmitting visible light to improve thermal comfort within the vehicle cabin. Increasingly, vehicles are incorporating more glass surfaces within a vehicle cabin.

[0006] US Patent Application Publication 2023 / 0377352 to Timo et al. entitled “Methods and systems for determining one or more characteristics inside a cabin of a vehicle" discloses a system for detecting occluded objects in a vehicle through reflections. However, this system does not capture sufficient three dimensional information to be able to accurately characterize objects in three dimensions.

[0007] Similarly US Patent Application Publication 2021 / 0016736 to Naghizadeh et al. entitled “3D time of flight active reflecting sensing systems and methods” discloses identifying objects in an enclosed space using a time of flight camera and reflective objects. Without knowledge of the enclosed space (e.g. a vehicle cabin), the system has no knowledge of thegeometry of the environment. As a result, reconstructing accurate three dimensional information of the objects is difficult. Furthermore, this system requires a time of flight camera which is typically, cost prohibitive for internal vehicle monitoring applications.

[0008] Any discussion of the background art throughout the specification should in no way be considered as an admission that such art is widely known or forms part of common general knowledge in the field.SUMMARY OF THE INVENTION

[0009] Embodiments of the present invention leverage reflections to extend the field of view of a camera monitoring system and / or to provide a second optical path for three dimensional reconstruction. In the context of a vehicle cabin, the inventors have identified that infrared reflections off cabin features can be used to image objects within the vehicle cabin not directly visible to a camera.

[0010] In accordance with a first aspect of the present invention, there is provided a method of monitoring an environment in the form of an interior of a vehicle cabin, the method comprising the steps: receiving image data indicative of one or more images captured of the vehicle cabin in the visible and / or infrared wavelength range from one or more imaging devices, the vehicle cabin comprising one or more reflective regions having properties such that at least a portion of visible and / or infrared radiation is reflected therefrom; processing the images to identify an object within the vehicle cabin, wherein one or more of the objects are visible in reflections from the one or more reflective regions; receiving input indicative of the geometry of the vehicle cabin; and processing images relating to the identified object visible in reflections in conjunction with the geometry of the vehicle cabin to determine characteristics of the object in three dimensions.

[0011] In some embodiments, at least some of the objects visible in reflections from the one or more reflective regions are located at positions within the vehicle cabin not directly visible by the one or more imaging devices.

[0012] In some embodiments, the one or more imaging devices comprise a two dimensional image sensor. In other embodiments, the one or more imaging devices comprisea three dimensional imaging system such as a stereo camera system or a time of flight camera.

[0013] In some embodiments, the one or more reflective regions include one or more regions or features of the vehicle cabin. The one or more reflective regions may include a rear window of the vehicle. The one or more reflective regions may include one or more side windows of the vehicle. The one or more reflective regions may include an inside surface of a sunroof of the vehicle. The one or more reflective regions may include one or more vehicle cabin lights. The one or more reflective regions may include one or more reflective elements installed in the vehicle cabin. The one or more reflective elements may include one or more cabin trim elements.

[0014] In some embodiments, the step of processing the images comprises detecting regions of reflection within the captured images and designating image pixels corresponding to the regions of reflection as reflective regions. In some embodiments, the step of processing the images comprises performing a distortion compensation process on pixels corresponding to the reflective regions.

[0015] In some embodiments, the one or more imaging devices includes a camera mounted on or adjacent a vehicle instrument cluster. In some embodiments, the one or more imaging devices include a camera mounted on or adjacent a vehicle rearview mirror. In some embodiments, the one or more imaging devices include an occupant monitoring camera mounted to an interior ceiling of the vehicle cabin.

[0016] The input indicative of the geometry of the vehicle cabin may comprise a CAD model of the vehicle. In some embodiments, the method further comprises the step of determining the location and orientation of the predetermined reflective regions in the CAD model of the vehicle.

[0017] In some embodiments, the one or more imaging devices comprises an image sensor with a two dimensional array of photosensitive pixels. In some embodiments, the one or more imaging devices comprises a three dimensional camera system configured to generate three dimensional image data.

[0018] In some embodiments, the method comprises the step of detecting a target object in the images that corresponds to an object visible in a reflection from the one or more reflective regions.

[0019] In some embodiments, the method comprises the step of comparing characteristics of the target object with that of the corresponding target object visible in the reflection to extract three dimensional location information of the target object in the environment.

[0020] In some embodiments, the one or more reflective regions are identified by a machine learning classification system that classifies images of the environment.

[0021] In some embodiments, at least one of the one or more reflective regions comprise a reflective region of eyewear worn by a subject within the environment.

[0022] In some embodiments, the objects within the environment comprises a mobile device.

[0023] In accordance with a second aspect of the present invention, there is provided a monitoring system comprising: one or more cameras configured to capture images of an interior of a vehicle cabin in the visible and / or infrared wavelength range, the vehicle cabin comprising one or more reflective regions having properties such that at least a portion of visible and / or infrared radiation is reflected therefrom; a processor configured to: process the images to identify an object within the vehicle cabin that is visible in reflections from the one or more reflective regions; receive input indicative of the geometry of the vehicle cabin; and process images relating to the identified object visible in reflections in conjunction with the geometry of the vehicle cabin to determine characteristics of the object in three dimensions.

[0024] In some embodiments, the one or more cameras include an image sensor capable of imaging radiation in both the visible and infrared wavelength ranges.

[0025] In some embodiments, the one or more cameras are controlled to image the environment under more than one image exposure rate.

[0026] In some embodiments, the processor is adapted to perform a hyper-resolution multi frame image analysis on image frames having different exposure periods to improve image resolution within the one or more reflective regions.

[0027] In some embodiments, the one or more cameras include a driver or occupant monitoring camera.

[0028] In some embodiments, the one or more reflective regions include one or more regions or features of the vehicle cabin.

[0029] In some embodiments, the system comprises one or more reflective elements installed in the vehicle cabin and wherein the one or more reflective regions include regions of reflection from the one or more reflective elements. The reflective elements may be embedded within a cabin trim of the vehicle cabin. The reflective elements may include an infrared reflective mirror.

[0030] In some embodiments, the one or more reflective regions are at predetermined positions within the environment.

[0031] In some embodiments, the one or more cameras include a composite lens having a first focal region with a long focal length at the predetermined positions corresponding to one or more of the reflective regions and a second focal region with a focal length shorter than that of the first focal region to image a broader region of the environment.

[0032] In some embodiments, the composite lens comprises a metasurface having subwavelength features to provide the first and second focal regions.

[0033] In accordance with a third aspect of the present invention, there is provided a method of monitoring an environment, the method comprising the steps: receiving image data indicative of images captured of the environment in the visible and / or infrared wavelength range from one or more imaging devices, determining one or more reflective regions within the environment having properties such that at least a portion of visible and / or infrared radiation is reflected therefrom; and processing the images to determine a presence and / or characteristics of objects visible within the one or more reflective regions.

[0034] In some embodiments, the method comprises the step of processing the images to determine a presence and / or characteristics of objects visible within a region outside the one or more reflective regions.

[0035] In some embodiments, the method comprises the step of identifying objects imaged both in a reflective region and a region outside the one or more reflective regions.

[0036] In some embodiments, the method comprises the step of comparing the location of the identified object in the reflective region to the location of the identified object in the region outside the one or more reflective regions.

[0037] In some embodiments, the step of determining reflective regions within the environment comprises determining coordinates of the reflective region in a predefined frame of reference.

[0038] In accordance with a fourth aspect of the present invention, there is provided a method of determining a three dimensional position of an object in an image, the method comprising the steps: i. receiving one or more images of a scene containing an object to be characterized; ii. detecting an object directly visible in the image; iii. detecting the object in a reflection visible in the image; iv. determining one or more corresponding pairs of features of the directly visible object and the object in the reflection; and v. mapping the pairs of features to a common reference frame using known geometry of the scene to determine the three dimensional position of the pairs of features.

[0039] In some embodiments, the method further comprises the steps: repeating steps iv and v for a plurality of pairs of features; and combining the three dimensional positions of the pairs of features to form a three dimensional reconstruction of the object.

[0040] In some embodiments, the method comprises the step of calculating a distortion of the reflection and applying this to pixels corresponding to the object visible in the reflection.

[0041] In some embodiments, the method comprises the steps: projecting a structured light pattern onto the scene; anddetecting characteristics of an imaged object based on deformations detected in the structured light pattern in processed images.

[0042] Embodiments of the invention allow a monitoring system to view areas of an environment that would not otherwise be visible directly to a camera or cameras. In the case of vehicle cabin monitoring, embodiments of the invention provide for viewing regions of the vehicle cabin such as passenger compartments behind the front seats and the rear cargo area. Furthermore, imaging an object both directly and via reflection provides for two optical imaging paths. With appropriate image processing, the multiple imaging paths can be used to triangulate the objects to extract distance and also to reconstruct the objects in three dimensions to extract three dimensional characteristics from the objects.BRIEF DESCRIPTION OF THE FIGURES

[0043] Example embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:Figure 1 is a schematic perspective view of the interior of a vehicle having a camera monitoring system installed in a rearview mirror;Figure 2 a schematic plan view of the vehicle of Figure 1 having the rearview mirror of Figure 1 installed therein;Figure 3 is a schematic front view of a rearview mirror according to an embodiment of the invention;Figure 4 is a schematic functional view of the main components of a camera monitoring system;Figure 5 is a perspective view of the cabin of the vehicle of Figures 1 and 2 as viewed from a camera of the rearview mirror of Figure 1 ;Figure 6 is a process flow diagram illustrating the primary steps in a method of monitoring an environment;Figure 7 is a schematic plan view of the vehicle of Figures 1 to 3 illustrating a scenario where a passenger’s head is imaged by directly by a camera and through a reflection off a side window of the vehicle;Figure 8 is a schematic diagram of a system over and coordinate frames for calculating a three dimensional reconstruction of an object or image;Figure 9 is a schematic illustration of a backward ray projection for a fisheye lens;Figure 10 is a schematic illustration of a ray calculation of a reflection off a convex mirror surface; andFigure 1 1 is a schematic illustration of a triangulation process.DESCRIPTION OF THE INVENTION

[0044] Embodiments of the present invention will be described with reference to a vehicle monitoring system such as a driver or occupant monitoring system for monitoring vehicle occupants within a vehicle cabin. In these embodiments, the environment being monitored includes the vehicle cabin. However, it will be appreciated that the invention is applicable to other scenarios and for monitoring other non-vehicle scenes or environments.System overview

[0045] Referring initially to Figures 1 to 4, the main components of an occupant monitoring system 100 will be described. System 100 is configured for capturing images of a vehicle driver 102 and / or occupants during operation of a vehicle 104. System 100 is further adapted for performing various image processing algorithms on the captured images such as facial detection, facial feature detection, facial recognition, facial feature recognition, facial tracking or facial feature tracking, such as tracking a person’s eyes. Example image processing routines are described in US Patent 7,043,056 to Edwards et al. entitled “Facial Image Processing System” and assigned to Seeing Machines Pty Ltd (hereinafter “Edwards et al”), the contents of which are incorporated herein by way of cross-reference.

[0046] As best illustrated in Figures 1 and 2, system 100 is incorporated into a rearview mirror 200 of vehicle 104. In other embodiments, system 100 may be incorporated into a vehicle dash adjacent an instrument display. In further embodiments, system 100 may be incorporated into other regions of vehicle 104 such as in an overhead system located on a ceiling / roof of vehicle 104.

[0047] Rearview mirror 100 is mounted in the conventional location within vehicle 104 at a central upper region of the front windshield. Mirror 100 includes a substantially horizontally elongate body 105 mounted to vehicle 104 at one or more mounting points 106, as shown in Figure 2. Mounting point 106 may be adapted to allow mirror 100 to be pivotally moveable. Body 105 preferably takes the form of a protective housing formed of a rigid material such as a plastics material. Body 105 supports an electrically controllable reflective device 108 that isadapted to selectively filter and reflect light incident onto mirror 100 in a manner described below.

[0048] System 100 includes an imaging camera 1 10 that is positioned on or in rearview mirror 200 and oriented to capture images of at least the driver’s eyes or face in the infrared wavelength range to identify, locate and track one or more human facial features. Camera 110 may also be positioned to image part or all of the cabin of vehicle 104 in addition to the driver’s face. In some embodiments, camera 1 10 may include a wide angled lens or fisheye lens to image the scene in wide angle. Although only a single camera is illustrated, it will be appreciated that system 100 may comprise a plurality of cameras disposed at different locations and orientations to view the environment from different perspectives.

[0049] Camera 1 10 is mounted to or adjacent to body 105 and comprises an image sensor oriented to capture two or three dimensional images of the interior of the vehicle. Camera 1 10 may be a conventional CCD or CMOS based digital camera having a two-dimensional image sensor comprising a two dimensional array of photosensitive pixels. Optionally, the camera may have the capability to determine range or depth (such as through one or more phase detect elements). The photosensitive pixels are preferably capable of sensing electromagnetic radiation in both the visible and infrared wavelength ranges. Camera 1 10 may also be a three dimensional camera such as a time-of-flight camera or other scanning or range-based camera capable of imaging a scene in three dimensions. In other embodiments, camera 110 may be replaced by a pair of like cameras operating in a stereo configuration and calibrated to extract depth.

[0050] Although camera 1 10 is preferably configured to image in both the visible and infrared wavelength ranges, it will be appreciated that, in alternative embodiments, camera 1 10 may image in only the infrared range or the visible wavelength ranges. To image in both the visible and infrared wavelength range, camera 1 10 may include a RGB-IR image sensor having pixels capable of sensing in the red, green, blue and IR wavelength regions. In some embodiments, camera 1 10 may include a wide-angled lens. In some embodiments, camera 110 may include an event camera.

[0051] As shown in Figures 1 and 2, camera 1 10 is preferably oriented such that the image sensor captures a field of view including the majority of the vehicle cabin, including driver 102, a rear window 1 12 of vehicle 104, passengers 116, 1 17 and 1 18 (to perform occupant / cabin monitoring) and one or more side window 120 and 122.

[0052] Figure 5 illustrates an exemplary perspective view of camera 1 10 viewing the interior of vehicle 104, including rear window 1 12, driver 102, passengers, 1 16, 1 17 and 1 18 and side windows 120 and 122. Axis C represents the longitudinal axis of vehicle 104 as shown in Figure 2.

[0053] Referring now to Figure 4, there is illustrated a schematic system level view of system 100. A processor 124 is configured to process the captured images and may perform various other functions such as generating a control signal for controlling a transmittance of electrically controllable reflective device 108. Processor 124 acts as the central processor for system 100 and is configured to perform a number of functions as described below.

[0054] Processor 124 is preferably contained within body 105. However, in other embodiments, processor 124 is located separate to body 105 and connected electrically or wirelessly to mirror 100 via a communications interface. In one embodiment, the operation of controller 124 is performed by an onboard vehicle computer system which is connected to camera 110 and light sources 144A and 144B. Processor 124 may be implemented as any form of computer processing device or portion of a device that processes electronic data, e.g., from registers and / or memory to transform that electronic data into other electronic data that, e.g., may be stored in registers and / or memory. As illustrated in Figure 4, processor 124 includes a microprocessor 126 (or multiple microprocessors, integrated circuits or chips operating in conjunction with each other), executing code stored in memory 128, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and other equivalent memory or storage systems as should be readily apparent to those skilled in the art.

[0055] Microprocessor 126 of processor 124 functionally includes a vision processor 130 and a device controller 132. Vision processor 130 and device controller 132 represent functional elements which are both performed by microprocessor 126. However, it will be appreciated that, in alternative embodiments, vision processor 130 and device controller 132 may be realized as separate hardware such as microprocessors in conjunction with custom or specialized circuitry.

[0056] Vision processor 130 is configured to process the captured images to perform various image processing functions described below, such as region of interest detection, brightness comparisons, glare detection, image distortion correction, object detection, three dimensional reconstruction and driver / occupant monitoring routines. In general, thedriver / occupant monitoring is performed based on infrared wavelength information received from the image sensor of camera 1 10 while brightness comparison and glare detection is performed based on visible wavelength information received from the image sensor of camera 110.

[0057] Device controller 132 is configured to control camera 1 10 and to generate a control signal for controlling a transmittance of electrically controllable reflective device 108. Control of electrically controllable reflective device may be performed based on input from vision processor 130 in a manner similar to that described in US Patent Application Publication 2024 / 0059220 entitled “Auto Dimming Mirror1’ and assigned to Seeing Machines Limited. The contents of US Patent Application Publication 2024 / 0059220 are incorporated herein by way of cross-reference.

[0058] In some embodiments, vision processor 130 is configured to process the captured images to perform the driver monitoring; for example to determine a three dimensional head pose and / or eye gaze position of the driver 102 within the monitoring environment. To achieve this, vision processor 130 utilizes one or more eye gaze determination algorithms. This may include, by way of example, the methodology described in Edwards et al. Vision processor 130 may also perform various other functions including determining attributes of the driver 102 such as eye closure, blink rate and tracking the driver’s head motion to detect driver attention, sleepiness or other issues that may interfere with the driver safely operating the vehicle.

[0059] The raw image data, gaze position data and other data obtained by vision processor 130 is stored in memory 128.

[0060] In some embodiments, the algorithms performed by vision processor 130 may be implemented as rules in a rule-based system or a weighted coefficients of a trained machine learning classifier such as implemented in a neural network. The neural network or other machine learning classifier model may be hosted as a network of nodes on microprocessor 126 and fed data stored in memory 128. Object detection and characterization may be performed by training a machine learning classifier to identify a predetermined set of objects.

[0061] Additional components of mirror 100 may also be included within the common housing of body 105 or may be provided as separate components according to otheradditional embodiments. Throughout this specification, specific functions performed by vision processor 130 or device controller 132 may be described more broadly as being performed by processor 124.

[0062] Finally, referring to Figures 1 and 3, mirror 100 optionally includes light sources 144A and 144B that are adapted to illuminate driver 102 and / or other occupants 1 16- 1 18 with infrared radiation. Light sources 144A and 144B may comprise Vertical Cavity Surface Emitting Lasers (VCSEL), Light Emitting Diodes (LED) or other light sources. This illumination is timed to occur during predefined image capture periods when camera 1 10 is capturing an image, so as to enhance the driver’s face to obtain high quality images of the driver’s face or facial features. This illumination by light sources 144A and 144B is advantageous for driver and occupant monitoring systems.

[0063] As illustrated in Figure 1 , camera 1 10 is preferably disposed on a lower flange 1 19 of mirror 100, together with light sources 144A and 144B. However, in other embodiments, camera 110 may be disposed at other locations on mirror 100 such as behind electrically controllable reflective device 108. In some embodiments, camera 1 10 may be located on a separate region of the vehicle cabin that is close to or adjacent to mirror 100. In these embodiments, it is preferable for camera 1 10 to be located as close to mirror 100 as possible so that the image sensor of camera 1 10 to capture visible light reflections that would result in glare to the driver 102.

[0064] Device controller 120 may be configured to control camera 110 and to selectively actuate light sources 144A and 144B in a sequenced manner in sync with the exposure time of camera 1 10. In some embodiments, the light sources 144A and 144B may be controlled to activate alternately during even and odd image frames to perform a strobing sequence. Other illumination sequences may be performed by device controller 132, such as L,L,R,R,L,L,R,R... or L,R,0,L,R,0,L,R,0... where “L” represents a left mounted light source, “R” represents a right mounted light source and “0” represents an image frame captured while both light sources are deactivated. Light sources 144A and 144B are preferably electrically connected to device controller 132 but may also be controlled wirelessly by controller 132 through wireless communication such as Bluetooth™ or WiFi™ communication.

[0065] Thus, during operation of vehicle 104, device controller 132 activates camera 1 10 to capture images of the face of driver 102 in a video sequence. Light sources 144A and 144B are activated and deactivated in synchronization with consecutive image frames captured bycamera 110 to illuminate the driver during image capture. Working in conjunction, device controller 132 and vision processor 130 provide for capturing and processing images of the driver to obtain driver state information such as drowsiness, attention and gaze position during an ordinary operation of vehicle 104.

[0066] In some embodiments, a light source such as one or both of light sources 144A and 144B are adapted to project a structured light pattern onto the scene being imaged. Structured light is a method of projecting a known pattern (often a grid or stripes) of light onto a scene. The deformation of this known pattern when it strikes surfaces within the scene allows for precise 3D shape measurement and surface characterization during the image processing stage. Generating a structured light pattern is possible in a number of ways such as with a physical mask, optical elements such as lenses, micromirror devices or liquid crystal devices. In some embodiments, one or more dielectric metasurface materials may be configured to generate a structured light pattern. This method is described in US Patent 1 1 ,431 ,889 entitled “High performance imaging system using a dielectric metasurface" and assigned to Seeing Machines Limited. The contents of this document are incorporated herein by way of crossreference.Image processing method

[0067] Referring now to Figure 6, there is illustrated a method 600 of monitoring an environment such as a vehicle cabin. Method 600 may be performed by vision processor 130 of system 100 and will be described with reference to system 100 of Figures 1 to 5. However, it will be appreciated that method 600 may be performed by other imaging systems. In particular, for cost and complexity reasons it is advantageous for method 600 to be performed by an existing imaging system such as an occupant monitoring. However, in some embodiments, method 600 is performed by a separate standalone imaging system.

[0068] Method 600 comprises the initial step 601 of receiving image data indicative of images captured of the environment in the visible and / or infrared wavelength range. These images are captured by one or more imaging devices such as driver / occupant monitoring camera 110 of system 100. The camera 1 10 may be a conventional camera having an image sensor with a two-dimensional array of pixels. Alternatively, camera 1 10 may comprise a time- of-flight or other three dimensional imaging device. As mentioned above, camera 1 10 may be mounted on or adjacent a vehicle rearview mirror, mounted on or adjacent a vehicle instrument cluster or may include an occupant monitoring camera mounted to an inside of theroof (or ceiling) of the vehicle. Although driver and occupant monitoring is frequently performed in the infrared wavelength range, camera 1 10 may also capture images in the visible wavelength range. The images are captured at frame rates and resolutions controlled by device controller 132 described above.

[0069] In the illustrated example of Figure 5, the environment being imaged includes the vehicle cabin, which comprises one or more reflective regions 152, 154 and 156 corresponding to reflective surfaces of rear window 1 12 and side windows 120 and 122. These reflective regions have properties such that at least a portion of infrared radiation is reflected therefrom. Such ‘native’ reflective regions are common in modern vehicles where surfaces such as glass windows and sunroofs comprise thermally insulating layers for enhanced thermal comfort to occupants. By ‘native’, it is intended to mean the reflective regions are formed from objects native or already found in a vehicle such as windows.

[0070] As described below, reflective regions may also be produced by reflections from dedicated ‘non-native’ reflective objects placed in the vehicle cabin. By ‘non-native’, it is intended to mean the reflective regions are formed from objects that are introduced or not typically found in a vehicle. Where a subject wearing eyewear (e.g. sunglasses) is imaged, reflective regions may also be defined from reflections from the eyewear. Although reflective regions 152, 154 and 156 are illustrated as being rectangular, this is for simplicity and it will be appreciated that reflective regions may take any shape.

[0071] In step 601 , although reflective regions 152, 154 and 156 are imaged, for simple object detection or classification, vision processor 130 need not have knowledge that these regions are in fact reflective regions. However, as will be described below, it is advantageous to have prior knowledge of the locations and characteristics of the reflective regions for performing more advanced processing. In general, reflective regions 152, 154 and 156 represent regions that allow camera 112 to image regions or objects within the scene that are not in a direct line of sight to camera 1 12.

[0072] At optional step 601 A, during image capture by the camera, one or more light sources may be configured to generate and project a structured light pattern onto the scene (in this case, the vehicle cabin). This structured light pattern is incident on the scene and objects therein and captured by camera 1 10. As described below, the deformation of the known structured light pattern can be extracted and used to determine three dimensional characteristics of objects in the scene.

[0073] At step 602, the images are processed by vision processor 130 to determine a presence and / or characteristics of objects within the environment. Objects may comprise any detected shapes or structures in the images determined through vision processing algorithms such as:• Image pre-processing to increase or decrease the volume of image data being processed;• Stereo image synthesis from images of cameras (paired with other cameras in a stereo arrangement) to determine three dimensional locations and sizes of objects imaged;• Object detection and recognition;• Motion detection;• Facial image processing, facial feature detection and facial recognition;• Region of interest analysis;• Eye gaze tracking;• Head pose estimation;• Extraction of geometric and topological information;• 3D Reconstruction;• Simultaneous location and mapping;• Scene recognition;• Optical character recognition;• Gesture and movement recognition;• Anomaly detection;• Behaviour and sentiment analysis;• High frame rate video construction;• Dynamic lighting adaptation: Adjusting to fast changes in illumination;• High frame rate video construction: Generating ultra-high-speed videos;• High-speed object detection;• Motion-based anomaly detection;• High-speed gesture recognition;• Neuromorphic processing: Processing visual data with neural networks;• Dynamic range enhancement;• Adaptive region of interest analysis: Dynamically focusing on parts of a scene with changes;• Structured light detection and reconstruction;• Rapid 3D scene reconstruction; and• Micro-scale event detection: Identifying rapid changes at micro levels;

[0074] Some or all of these algorithms may be implemented in the form of rules in a rulebased system or a weighted coefficients implemented by a machined learned model such as a neural network. The detection of objects at step 602 may coincide with a detection of the structured light pattern projected onto the scene at step 601 A.

[0075] By way of example, objects may comprise cabin regions or features such as seats, headrests, window regions, lights, doors. Objects may also comprise vehicle occupants, mobile devices such as smartphones, pets or foreign objects such as balls, bags, tools or equipment. At least some of the detected objects comprise objects detected as reflections from reflective regions 152, 154 and 156. Examples of reflective regions comprise:• Side windows 120 and 122;• An inside surface of a vehicle sunroof;• Rear window 1 12;• Vehicle cabin lights;• Cabin trim elements;• Reflective plastics surfaces such as B-pillars or C-Pillars;• One or more reflective elements installed in the vehicle cabin (described below).

[0076] Again, at this step, vision processor 130 still need not have knowledge that reflective regions 152, 154 and 156 are in fact regions of reflection. Rather, vision processor 130 maysimply be detecting a presence and / or characteristics of objects visible within the images and therefore within the vehicle cabin.

[0077] Reflective regions 152, 154 and 156 represent regions in which portions of the environment (e.g. vehicle cabin) are viewed by camera 1 10 through reflection. This is illustrated schematically in Figure 7 in which a head of passenger 1 18 is imaged both directly by camera 1 10 and also by a reflection off side window 122. Some of the objects detected as reflections from the one or more reflective regions may be located at positions within the environment not directly visible by camera 110. By way of example, camera 1 10 may image reflections off windows 120 or 122 so as to view objects hidden from direct view by vehicle seats. Similarly, camera 1 10 may image reflections off a glass or reflective sunroof so as to view objects within the footwells behind the seats. Camera 1 10 may also image reflections off rear window 1 12 so as to view objects within the trunk of vehicle 102.

[0078] Where the objective is to simply determine a presence and / or characteristics of objects, no further image processing is performed and method 600 ends at step 602. This provides for imaging objects within the vehicle cabin that would not otherwise be imaged directly by camera 110. By imaged ‘directly’, it is meant that an object or region is within a direct line of sight of camera 1 10. Vision processor 130 may not know which detected objects were detected by direct line of sight imaging of camera 1 10 or which objects are visible as reflections from one or more of the regions 152, 154 and 156 (representing reflections of real objects). However, as will be described below, having knowledge of where reflective regions 152, 154 and 156 lie and their associated reflective characteristics is advantageous in some embodiments.

[0079] Thus, in some embodiments, method 600 comprises the further step 603 of actively detecting regions of reflection within the captured images and designating image pixels corresponding to the regions of reflection as reflective regions 152, 154 and 156. Through this process, vision processor 130 may gain knowledge of which pixel regions correspond to reflections and therefore identify reflections of objects. Reflective regions may comprise regions of pixels in various shapes such as rectilinear shapes or curved shapes. Once a reflective region has been detected, vision processor 130 may perform additional processing on pixels of these regions to extract information regarding objects detected in these regions.

[0080] In some embodiments, method 600 comprises the optional step 604 of receiving input indicative of the geometry of the vehicle cabin (or environment in non-vehicleembodiments). This input may comprise a CAD model of the vehicle or other data providing position information of objects and features of the vehicle cabin. The CAD model may be stored in memory 128 and camera 110 may be calibrated with reference to the 3D data stored within the CAD model. By way of example, the three dimensional pose of camera 1 10 may be determined in a vehicle (or ‘world’) frame of reference through the method described in PCT Patent Application Publication WO 2018 / 000037 entitled “Systems and Methods for Identifying Pose of Cameras in a Scene", assigned to Seeing Machines Limited.

[0081] By way of example, a vehicle and camera coordinate system may be defined as follows:• World Coordinate System (WCS): (X_w, Y_w, Z_w) - A fixed global reference frame. This is where the final object shape and location will be defined.• Camera Coordinate System (CCS): (X c, Y c, Z c) - Origin C is usually the camera's effective optical center. The Z-axis often aligns with the optical axis.

[0082] Knowledge of the vehicle cabin geometry may also provide information on the contours of reflective surfaces such as windows and a sunroof. This information is valuable as contoured reflective surfaces will distort objects imaged in the reflections. By way of example, a reflective surface may be defined with a mirror coordinate system as follows:• Mirror Coordinate System (MCS): (X_m, Y_m, Z_m) - A local frame defined relative to the mirror (e.g., vertex at the origin) to simplify its mathematical description.

[0083] Where camera 1 10 comprises a three dimensional camera system (e.g. stereo system or time-of-flight sensor), three dimensional image data may be obtained about the environment (vehicle cabin). In these embodiments, it may not be necessary to rely on input indicative of the geometry such as a vehicle cabin geometry as objects can be imaged in three dimensions directly from camera 110.

[0084] Once calibrated, each sensor pixel of camera 110 may be designated as imaging a particular three dimensional position within the vehicle cabin relative to a vehicle or world frame of reference. Objects detected by camera 1 10 may then be located in two dimensions or three dimensions if camera 1 10 has capability of performing depth detection.

[0085] The CAD model information (or other input position data) input at step 604 or three dimensional image data from a three dimensional camera system may be used by visionprocessor 130 to determine the location and / or characteristics (e.g. orientation or curvature) of the reflective regions at step 603. This may include allocating some or all of the pixels of the reflective regions a three dimensional position, coordinates or address relative to a vehicle or world frame of reference or a camera frame of reference. The located reflective regions may be determined as regions of interest within the environment.

[0086] In some embodiments, the reflective regions may be identified by a machine learning classification system (e.g. neural network) that classifies images of the environment. The machine learning classification system may be trained with images of objects visible directly and in reflections so that the classification system can accurately distinguish objects viewed in reflections from direct imaged objects.

[0087] In some embodiments, a light source such as light sources 144A and 144B are adapted to produce polarized light and the image sensor of camera 1 10 is adapted to be sensitive to polarization via one or more polarizing elements. In these embodiments, the polarizing properties of light reflected off reflective regions may be used to locate reflective regions and / or distinguish objects visible in reflective regions from objects imaged directly. In particular, polarization states change orientation upon specular reflection. For example, vertically linear polarized light changes to horizontally linearly polarized light upon specular reflection (and vice versa). Similarly, right hand circularly polarized light changes to left hand circularly polarized light upon specular reflection (and vice versa). These changes in polarization state may be detected at camera 1 10 and used to locate reflective regions and / or distinguish objects visible directly versus in reflective regions.

[0088] Once the locations of reflective regions 152, 154 and 156 are known, vision processor 130 is able to determine whether detected objects fall within one of the reflective regions or not. As such, in some embodiments, method 600 comprises the optional step 605 of detecting a target object in the images that corresponds to an object detected as a reflection from the one or more reflective regions. This process involves performing a comparison of the pixel address / coordinates (locations) corresponding to the target object with the known pixel addresses of the one or more reflective regions determined in step 603. If it is determined that a target object falls within a reflective region, then the target object is determined to be a reflection of an actual object within the vehicle cabin.

[0089] In some embodiments, step 605 of detecting a reflected target object does not require specific position knowledge of the reflective regions. In some embodiments, visionprocessor 130 is capable of discriminating reflections from direct-imaged regions via algorithms such as a trained classifier algorithm. The machine learning classification system may be trained with images of objects visible directly and in reflections so that the classification system can accurately distinguish objects viewed in reflections from direct imaged objects through known object distortion or the like. In other embodiments, the position data from the CAD model received at step 604 may be used to determine whether an object is imaged directly or in a reflection. By way of example, referring to Figure 7, if the object position is determined to be at a location 180 outside the vehicle cabin based on the CAD model data, then vision processor 130 may determine that this object has been imaged as a reflection.

[0090] In some embodiments, at optional step 606, vision processor 130 performs further processing on reflective regions, particularly when objects are detected in these regions. By way of example, at step 606, vision processor 130 may perform distortion compensation such as an inversion process on pixels corresponding to the reflective regions. This inversion process counters the inversion that occurs upon reflection off a surface and can be used to reconstruct an estimate of the object’s shape and / or orientation. As reflective regions 152, 154 and 156 correspond to surfaces that may be non-planar (e.g. concave or convex), in some embodiments, step 606 comprises vision processor 130 performing other distortion compensation processes on pixels corresponding to the reflective regions 152, 154 and 156.

[0091] This distortion compensation may be based on known characteristics of the reflective region such as the curvature of the corresponding reflective surface from a CAD model. These characteristics may be provided or derived from the input of the geometry of the vehicle cabin (or environment in general) at optional step 604. Full three dimensional reconstruction of objects imaged in a reflective region must take into account not only the curvature of the reflective surface but also any distortion by the camera. For example, if the camera comprises a fisheye lens, that lens will naturally distort any imaged objects.

[0092] As mentioned above, object detection and characterization may be performed by training a machine learning classifier to identify a predetermined set of objects. In these embodiments, distortion compensation may be taken into account in the training of the machine learning classifier. For example, the machine learning classifier may be trained on data with representative distortions of objects that may be detected. This may involve feedingthe machine learning classifier images of objects visible in reflections and which contain different degrees of distortion.

[0093] In sophisticated monitoring systems, the camera(s) will include full capability to image in three dimensions and extract depth or distance information. However, this adds cost and complexity to the monitoring system. Where camera 1 10 does not include capability for estimating depth, the technique of using reflections can be used to determine characteristics such as depth or distance from the camera to the object and / or the shape and size of an object. This is possible as, in some cases, a target object may be visible to camera 1 10 both directly (line of sight) and as indirectly a reflection in a reflection region.

[0094] The dual-imaging of an object may be performed, at step 607 by vision processor 130 performing a shape or object matching algorithm to detect similar characteristics of the objects with a sufficient degree of confidence or accuracy. This processing may also include inputting the images into a machine learning classification model (such as a neural network) trained to compare objects viewed from different orientations and / or with different degrees of distortion. In this scenario, where two views of an object are available, vision processor 130 may be capable of performing analysis on the two versions of the imaged object to extract a higher level of information about the target object. As such, in some embodiments, method 600 comprises the optional step 608 of comparing characteristics of the target object with that of the corresponding target object detected as a reflection.

[0095] By way of example, a passenger of a vehicle can be imaged directly by a camera and also in a reflection from a reflection region of the vehicle such as the sunroof or side windows. By comparing image data of the passenger from both the direct image path and the reflected image path, information about the passenger size and weight can be estimated. This can be useful information for the purpose of vehicle control information such as the smart deployment of airbags. This information can be enhanced when combined with vehicle CAD model data or three dimensional image data to extract distance information from a calibrated camera.

[0096] Where depth information is not already available, through comparison of the characteristics of the two versions of the objects, three dimensional location information of the target object in the environment can be obtained using an object triangulation analysis (similar to stereo imaging). This dual imaging of objects improves the accuracy of object characteristic estimation because a second view of an object helps confirm / correct the estimates made fromthe first view. Furthermore, visibility is different in each view as the optical paths are different. For example, as mentioned above, polarization properties of light received directly from an object and via a specular reflection of an object may be different. In addition, reflection regions such as thermal glass windows of a vehicle have filtering properties for infrared and visible light so different levels of light may be imaged along the different optical paths.

[0097] By way of example, as illustrated in Figure 7, a target object is a passenger’s head, which can be viewed directly from the front and from the side as a reflection. In this scenario, vision processor 130 has two views of the head and can perform processing to determine a size or shape of the head to estimate whether the head belongs to an adult or a child. Vision processor 130 can also accurately determine a distance to the head from camera 1 10.

[0098] This processing may include rendering the captured images of the head onto a 3D head model based on detected features such as the eyes, nose and mouth. This processing may also include inputting the images into a machine learning classification system (such as a neural network) trained to detect objects such as heads from different views.

[0099] At step 610, three dimensional reconstruction of objects detected in reflections is performed. This reconstruction may be based on the information obtained in step 606 or from step 608. The three dimensional reconstruction includes determining object location within a known reference frame, compensating for distortions and camera calibration and extracting characteristics such as the size, location and three dimensional shape of the object.

[0100] Example calculations for three dimensional image or object reconstruction is set out below. These calculations are based on a single fisheye lens camera viewing the object directly and via a reflection from a mirror surface of known shape. The known shape may come from the vehicle CAD model for example. This technique relies on triangulation principles applied to corresponding features identified in both views within the same image. As such, it results from information determined in step 608 of method 600. However, with the information on cabin geometry known from step 604, it is possible to perform three dimensional reconstruction directly from the reflection image of an object combined with the known cabin geometry, which includes location and curvature of a reflective surface.

[0101] 1 . Coordinate Systems:• World Coordinate System (WCS): (X_w, Y_w, Z_w) - A fixed global reference frame. This is where the final object shape and location will be defined.® Camera Coordinate System (CCS): (X_c, Y_c, Z c) - Origin C is usually the camera's effective optical center. The Z-axis often aligns with the optical axis.® Mirror Coordinate System (MCS): (X_m, Y_m, Z_m) - A local frame defined relative to the mirror (e.g., vertex at the origin) to simplify its mathematical description.

[0102] The system overview and coordinate frames are illustrated schematically in Figure 8.

[0103] 2. Known Geometric Relationships (Extrinsics):

[0104] The spatial relationship between the camera, the mirror, and the world (objects and features within the vehicle cabin) must be precisely known. This involves rotation matrices (R) and translation vectors (t):• Camera Pose: o Transformation from WCS to CCS: P_c = R_{cw} * P_w + t_{cw}. o This defines the camera's position and orientation in the world.• Mirror Pose: o Transformation from WCS to MCS: P_m = R_{mw] * P_w + t_{mw}. o This defines the mirror's position and orientation in the world. o The inverse transformations (R_{wc), t_{wc} and RJwm], t_{wm}) are needed as well. These are assumed to be known (e.g., from calibration).

[0105] 3. Camera Model (Fisheye Lens):

[0106] A calibrated fisheye camera model is described by projection and distortion models:• Projection Model: o Defines how an incoming ray angle 0 maps to a radial distance r on the image plane (e.g., Equidistant: r = f*0, Equisolid: r = 2f*sin(0 / 2), Orthographic: r = f*sin(0), Stereographic: r = 2f*tan(0 / 2)).• Intrinsic Parameters: o Focal length parameters (fx, fy in pixel units). o Principal point (ex, cy in pixel units), the intersection of the optical axis with thesensor.• Distortion Parameters: o Coefficients describing deviations from the ideal projection model. Typically includes:■ Radial distortion (k1 , k2, k3, k4,...)■ Tangential distortion (p1 , p2)• Backward Projection (Pixel -> 3D Ray): o Given a pixel coordinate p = (u, v) on the sensor, this function calculates the 3D ray in WCS that corresponds to this pixel.• Pixel to Normalized Distorted: o Convert (u, v) to normalized coordinates relative to the principal point, incorporating focal lengths: x_d = (u - ex) I fx, y_d = (v - cy) I fy.• Undistort: o Apply the inverse distortion model to (x_d, y_d) to get undistorted normalized coordinates (x', y'). o This step might require iterative methods, (x', y') = Undistort(x_d, y_d, k1 , k2,..., p1 , p2).• Normalized to Angle: o Calculate the radial distance r = sqrt(x'2+ y'2). o Use the inverse of the chosen projection model to find the angle of incidence 0 (e.g., for Equidistant 0 = r / f, adjusting for f vs fx.fy). o Calculate the azimuth <p = atan2(y', x').• Angle to 3D Vector (CCS): o Convert the angles (0, <p) into a 3D direction vector d_c in the Camera Coordinate System (CCS). o Typically: d_c = (sin(0) * cos(cp), sin(0) * sin(cp), cos(0)) (assuming Z is forward).• Vector to WCS: o Transform the direction vector d_c from CCS to WCS using the camera's rotation: D = R_{wc} * d_c.• Ray in WCS: o The final ray originates from the camera's position C_w (derived from t_{cw}) in WCS and travels along direction D: Ray = C_w + A * D, where A > 0 is the distance along the ray.

[0107] This process is illustrated schematically in Figure 9.

[0108] 4. Mirror Model:• Surface Equation: o The mirror's shape is known in MCS by polynomial curves. o This defines a surface, which can be represented implicitly as M_m(x_m, y_m, z_m) = 0 or explicitly z_m = Poly(x_m, y_m).• Surface in WCS: o Transform the mirror equation into WCS using the known mirror pose (RJmw], tjmw}). o Let the surface equation in WCS be M_w(P_w) = 0, where P_w = (X_w, Y_w, Z_w).• Surface Normal: o Define a function to calculate the outward-pointing unit normal vector N_m to the mirror surface at any given point P_m on the mirror (expressed in WCS). o This is typically found using the gradient of the surface equation: N = VM_w(P_m), followed by normalization: N_m = N / ||N||.

[0109] 5. Feature Detection and Matching:Capture an image containing both the direct view and the reflected view of the object.Detect salient feature points (e.g., corners, blobs using SIFT, SURF, ORB or other algorithms) in the image.• Establish Correspondence: o Identify pairs of pixel coordinates (p_d, pj) that correspond to the same physical point on the object, where p_d = (u_d, v_d) is from the direct view and pj = (uj, vj) is from the reflected view. o Matching descriptors and potentially geometric constraints (like catadioptric epipolar geometry) are used here. o Use methods like RANSAC to detect outliers and filter mismatches.[001 10] 6. Ray Generation per Feature Pair:[001 1 1 ] For each matched pair (p_d, pj):• Direct View Ray (Ray_d): o Take the direct view pixel p_d. o Use the Backward Projection function (Step 3) to compute the corresponding 3D ray Ray_d in WCS. It originates from the camera center C_w and has direction D_d. Ray_d: P(A_d) = C_w + A_d * D_d• Indirect (Reflected) View Ray (Rayj-): o Take the mirror view pixel 'p_i$. o Use the Backward Projection function (Step 3) to compute the initial 3D ray RayJ in WCS, originating from C_w with direction D i. This ray points towards the mirror. RayJ: P(AJ) = C_w + A J * D_i o Find Mirror Intersection Point (P_m): Find where RayJ intersects the mirror surface M_w(P_w) = 0. Substitute the ray equation into the mirror equation: M_w(C_w + A J * DJ) = 0. Solve this equation for A_i. Since the mirror is convex, there should be a unique, physically meaningful solution (positive A_i). Calculate the intersection point: P_m = C_w + A_i * DJ. o Calculate Normal at Intersection: Compute the unit surface normal N_m of the mirror at point P_m (Step 4). o Apply Law of Reflection: Calculate the direction D r of the reflected ray using the incident direction D i and the normal N_m: D r = D i - 2 * (D i •N_m) * N_m (Ensure DJ points towards the mirror). o Define Reflected Ray: The final reflected ray Ray_r originates from the reflection point P_m on the mirror surface and travels along direction D r. Ray_r: P(A_r) = P_m + A_r * D_r[001 12] Figure 10 schematically illustrates a reflection off a convex mirror surface.[001 13] 7. Triangulation:• For the feature pair (p_d, pj), there are now two 3D rays in WCS: Ray_d (origin C_w, direction D_d) and Ray_r (origin P m, direction D r). These two rays should ideally intersect at the actual 3D location of the object feature P_{obj}.• Find Point of Closest Approach: o Due to noise and calibration inaccuracies, the rays will likely be skew (not intersecting perfectly). o Find the points P_d_closest on Ray_d and P_r_closest on Ray_r such that the distance between them is minimized. o This involves solving a small linear system derived from the condition that the vector connecting P_d_closest and P_r_closest is orthogonal to both direction vectors D_d and D r.• Estimate 3D Point (P_{obj}): o The estimated 3D coordinate of the object feature is typically taken as the midpoint of the shortest segment connecting the two rays: P_{obj} = (P_d_closest + P_r_closest) 12[001 14] Figure 1 1 schematically illustrates the triangulation process.[001 15] 8. 3D Object Reconstruction:• Repeat Steps 5, 6, and 7 for many corresponding feature pairs identified across the object's visible surfaces in both views.• The collection of all computed P_{obj} points forms a 3D point cloud representing the object's shape and location in the World Coordinate System.• Optionally, apply surface reconstruction algorithms (e.g., Poisson reconstruction, ballpivoting, Delaunay triangulation) to the point cloud to generate a 3D mesh model of the object.[001 16] Summary of Mathematics:• Coordinate transformations (rotations, translations).• Fisheye camera projection and distortion models (and their inverses).• Vector geometry (dot products, normalization).• Analytic geometry (ray-surface intersection - solving polynomial equations potentially numerically).• Gradient calculation (for surface normals).• Linear algebra (solving for point of closest approach between skew lines).[001 17] Although method 600 is described in a particular order, it will be appreciated that the order of at least some of these steps can be varied. By way of example, step 606 may be performed as part of or immediately following step 602 or step 603.[001 18] It will be appreciated that the quality of reflected images will depend on factors such as ambient light within the vehicle cabin and also the brightness of the environment outside the vehicle cabin. As such, in some embodiments, camera 110 may be controlled by device controller 132 to image the vehicle cabin under more than one image exposure period and / or illumination setting by a light source or sources such as light sources 144A and 144B. In particular, the exposure rate and / or illumination timing / intensity of light sources may be dynamically controlled based on detection of ambient light conditions, either within the vehicle or outside the vehicle (or both). In some embodiments, the ambient light is detected by camera 110. In some embodiments, the ambient light may be detected by one or more ambient light detectors (not shown), which provide input to device controller 132. Control algorithms may be implemented with feedback between vision processor 130 and device controller 132 so as to enhance or maximise reflections from reflective regions 152, 154 and 156.[001 19] In some embodiments, vision controller 130 and device controller 132 together perform a hyper-resolution multi frame image analysis on image frames having differentexposure periods. This process may act to improve image resolution within the one or more reflective regions and enhance object detection within those regions.

[0120] In method 600 described above, the locations of the reflective regions need not be prior known as they can be determined from vehicle geometry information, three dimensional image information and / or using a machine learning classifier. However, in other embodiments, the locations and shapes of the reflective regions are prior known and stored in memory. In this way, when vision processor 130 processes images captured by camera 1 10 and detects one or more objects, vision processor 130 can determine whether or not the objects are detected as reflections from the one or more reflective regions.

[0121] In some embodiments, the environment may be embedded with one or more dedicated reflective elements which give rise to reflective regions. In the illustrated embodiment of a vehicle cabin, one or more reflective elements may be installed in the vehicle cabin at particular locations and orientations in the view of camera 1 10. The reflective elements may be located at unknown locations or at predetermined known positions within the environment. The reflective elements may be partially or wholly reflective for one or both infrared or visible wavelengths of light.

[0122] The reflective regions then include regions of reflection from the one or more reflective elements. By way of example, the reflective elements may be embedded within a cabin trim of the vehicle cabin such as in the vehicle’s B-pillars or C-pillars. The reflective elements may include infrared reflective mirror elements such as metal or plastics. In one particular example, convex mirrors may be placed at key locations within the vehicle cabin (e.g. on cabin walls) to reveal occupants or objects behind the seats, and / or view the cabin and occupants from alternative viewpoints.

[0123] A similar approach may be performed in other non-vehicle environments by installing mirrors or other reflective elements in locations such that they can view regions of the environment hidden from view of cameras. This is particularly useful for video surveillance systems.

[0124] It will be appreciated that the reflective regions may be small regions and thus only provide small or low resolution images of reflected objects. In some embodiments, camera 110 may be modified to improve the imaging of objects imaged in reflections. For example, camera 1 10 (or other cameras within the environment) may include a compositelens that can image different regions of the environment with different focal lengths / powers. In particular, when the environment and locations of the reflective regions are known, a composite lens can be designed which has a first focal region with a long focal length at the predetermined positions corresponding to one or more of the reflective regions. The composite lens also has a second focal region with a focal length shorter than that of the first focal region to image a broader region of the environment.

[0125] The longer focal length region effectively creates a zoomed image of the reflective regions to improve the information quality received by the camera. At the same time, the shorter focal length region can image the whole vehicle cabin field of view. In this manner, the broader vehicle cabin can be imaged to perform driver and occupant monitoring through the second focal region while one or more of the reflective regions can be imaged with higher resolution through the higher focal power of the first focal region. The focal properties of the composite lens must be taken into account when performing image processing of objects in the captured images.

[0126] In some embodiments, the composite lens comprises a metasurface material. This metasurface material preferably has subwavelength features to provide the first and second focal regions. The metasurface may be formed by a dielectric material having a periodic subwavelength structure such as a diffraction grating.

[0127] One particular application of the present invention is the detection of a mobile device by a driver of a vehicle. Conventional driver monitoring systems are configured to image a face of a driver but not necessarily the driver’s lap. When a driver is using a mobile device while driving, that mobile device is typically out of view of the driver monitoring system and cannot be detected. However, using the present invention, where the driver is wearing eyewear such as sunglasses and using a mobile device, the camera of the driver monitoring system may be able to view the mobile device in reflections off the eyewear. Detection of a mobile device is important in driver monitoring systems as such use is a major source of distraction to drivers. Upon detection of a mobile device in use by the driver, appropriate alerts or countermeasures may be employed by the driver monitoring system or ancillary systems.INTERPRETATION

[0128] The term “infrared” is used throughout the description and specification. Within the scope of this specification, infrared refers to the general infrared area of the electromagnetic spectrum which comprises near infrared, infrared and far infrared frequencies or light waves.

[0129] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "processing," "computing," "calculating," “determining”, analyzing” or the like, refer to the action and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

[0130] In a similar manner, the term “controller” or "processor" may refer to any device or portion of a device that processes electronic data, e.g., from registers and / or memory to transform that electronic data into other electronic data that, e.g., may be stored in registers and / or memory. A “computer” or a “computing machine” or a "computing platform" may include one or more processors.

[0131] Reference throughout this specification to “one embodiment”, “some embodiments” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases “in one embodiment”, “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.

[0132] As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

[0133] In the claims below and the description herein, any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements / features that follow, but not excluding others. Thus, the term comprising, when used in the claims, should not be interpreted as being limitative to the means or elements or stepslisted thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which comprises or that comprises as used herein is also an open term that also means including at least the elements / features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.

[0134] It should be appreciated that in the above description of exemplary embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single embodiment, Fig., or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this disclosure.

[0135] Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the disclosure, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0136] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0137] The method described above are outlined in a particular order. However, it will be appreciated that the order of steps of the method may, in some embodiments, be varied.

[0138] Embodiments described herein are intended to cover any adaptations or variations of the present invention. Although the present invention has been described and explained in terms of particular exemplary embodiments, one skilled in the art will realize that additional embodiments can be readily envisioned that are within the scope of the present invention.

Claims

What is claimed is:1 . A method of monitoring an interior of a vehicle cabin, the method comprising the steps: receiving image data indicative of one or more images captured of the vehicle cabin in the visible and / or infrared wavelength range from one or more imaging devices, the vehicle cabin comprising one or more reflective regions having properties such that at least a portion of visible and / or infrared radiation is reflected therefrom; processing the images to identify an object within the vehicle cabin that is visible in reflections from the one or more reflective regions; receiving input indicative of the geometry of the vehicle cabin; and processing images relating to the identified object visible in reflections in conjunction with the geometry of the vehicle cabin to determine characteristics of the object in three dimensions.

2. The method according to claim 1 wherein at least some of the objects visible in reflections from the one or more reflective regions are located at positions within the vehicle cabin not directly visible by the one or more imaging devices.

3. The method according to any one of the preceding claims wherein the one or more reflective regions include one or more regions or features of the vehicle cabin.

4. The method according to claim 3 wherein the one or more reflective regions include a rear window of the vehicle.

5. The method according to any one of the preceding claims wherein the one or more reflective regions include one or more side windows of the vehicle.

6. The method according to any one of the preceding claims wherein the one or more reflective regions include an inside surface of a sunroof of the vehicle.

7. The method according to any one of the preceding claims wherein the one or more reflective regions include one or more vehicle cabin lights.

8. The method according to any one of the preceding claims wherein the one or more reflective regions include one or more reflective elements installed in the vehicle cabin.

9. The method according to claim 8 wherein the one or more reflective elements include one or more cabin trim elements.

10. The method according to any one of the preceding claims wherein the step of processing the images comprises detecting regions of reflection within the captured images and designating image pixels corresponding to the regions of reflection as reflective regions.1 1. The method according to claim 10 wherein the step of processing the images comprises performing a distortion compensation process on pixels corresponding to the reflective regions.

12. The method according to any one of the preceding claims wherein the one or more imaging devices include a camera mounted on or adjacent a vehicle instrument cluster.

13. The method according to any one of claims 1 to 1 1 wherein the one or more imaging devices include a camera mounted on or adjacent a vehicle rearview mirror.

14. The method according to any one of claims 1 to 1 1 wherein the one or more imaging devices include an occupant monitoring camera mounted to an interior ceiling of the vehicle.

15. The method according to any one of the preceding claims wherein the input indicative of the geometry of the vehicle cabin comprises a CAD model of the vehicle.

16. The method according to claim 15 comprising the step of determining the location and orientation of the predetermined reflective regions in the CAD model of the vehicle.

17. The method according to any one of the preceding claims wherein the one or more imaging devices comprises an image sensor with a two dimensional array of photosensitive pixels.

18. The method according to any one of claims 1 to 16 wherein the one or more imaging devices comprises a three dimensional camera system configured to generate three dimensional image data.

19. The method according to any one of the preceding claims comprising the step of detecting a target object in the images that corresponds to an object visible in a reflection from the one or more reflective regions.

20. The method according to claim 19 comprising the step of comparing characteristics of the target object with that of the corresponding target object visible in the reflection to extract three dimensional location information of the target object in the vehicle cabin.21 . The method according to any one of the preceding claims wherein the one or more reflective regions are identified by a machine learning classification system that classifies images of the environment.

22. The method according to any one of the preceding claims wherein at least one of the one or more reflective regions comprise a reflective region of eyewear worn by a subject within the vehicle cabin.

23. The method according to any one of the preceding claims wherein the objects within the vehicle cabin comprises a mobile device.

24. A monitoring system comprising: one or more cameras configured to capture images of an interior of a vehicle cabin in the visible and / or infrared wavelength range, the vehicle cabin comprising one or more reflective regions having properties such that at least a portion of visible and / or infrared radiation is reflected therefrom; and a processor configured to: process the images to identify an object within the vehicle cabin that is visible in reflections from the one or more reflective regions; receive input indicative of the geometry of the vehicle cabin; and process images relating to the identified object visible in reflections in conjunction with the geometry of the vehicle cabin to determine characteristics of the object in three dimensions.

25. The system according to claim 24 wherein the one or more cameras include an image sensor capable of imaging radiation in both the visible and infrared wavelength ranges.

26. The system according to claim 24 or claim 25 wherein the one or more cameras are controlled to image the vehicle cabin under more than one image exposure rate.

27. The system according to claim 26 wherein the processor is adapted to perform a hyperresolution multi frame image analysis on image frames having different exposure periods to improve image resolution within the one or more reflective regions.

28. The system according to any one of claims 24 to 27 wherein the one or more cameras include a driver or occupant monitoring camera.

29. The system according to claim 28 wherein the one or more reflective regions include one or more regions or features of the vehicle cabin.

30. The system according to claim 28 or claim 29 wherein the system comprises one or more reflective elements installed in the vehicle cabin and wherein the one or more reflective regions include regions of reflection from the one or more reflective elements.31 . The system according to claim 30 wherein the reflective elements are embedded within a cabin trim of the vehicle cabin.

32. The system according to claim 30 or claim 31 wherein the reflective elements include an infrared reflective mirror.

33. The system according to any one of claims 24 to 32 wherein the one or more reflective regions are at predetermined positions within the environment.

34. The system according to claim 33 wherein the one or more cameras include a composite lens having a first focal region with a long focal length at the predetermined positions corresponding to one or more of the reflective regions and a second focal region with a focal length shorter than that of the first focal region to image a broader region of the vehicle cabin.

35. The system according to claim 34 wherein the composite lens comprises a metasurface having subwavelength features to provide the first and second focal regions.

36. A method of monitoring an environment, the method comprising the steps: receiving image data indicative of images captured of the environment in the visible and / or infrared wavelength range from one or more imaging devices, determining one or more reflective regions within the environment having properties such that at least a portion of visible and / or infrared radiation is reflected therefrom; and processing the images to determine a presence and / or characteristics of objects visible within the one or more reflective regions.

37. The method according to claim 36 comprising the step of processing the images to determine a presence and / or characteristics of objects visible within a region outside the one or more reflective regions.

38. The method according to claim 37 comprising the step of identifying objects imaged both in a reflective region and a region outside the one or more reflective regions.

39. The method according to claim 38 comprising the step of comparing the location of the identified object in the reflective region to the location of the identified object in the region outside the one or more reflective regions.

40. The method according to claim 36 wherein the step of determining reflective regions within the environment comprises determining coordinates of the reflective region in a predefined frame of reference.41 . A method of determining a three dimensional position of an object in an image, the method comprising the steps: i. receiving one or more images of a scene containing an object to be characterized; ii. detecting an object directly visible in the image; ill. detecting the object in a reflection visible in the image; iv. determining one or more corresponding pairs of features of the directly visible object and the object in the reflection; and v. mapping the pairs of features to a common reference frame using known geometry of the scene to determine the three dimensional position of the pairs of features.

42. The method according to claim 41 further comprising the steps: repeating steps iv and v for a plurality of pairs of features; and combining the three dimensional positions of the pairs of features to form a three dimensional reconstruction of the object.

43. The method according to claim 42 comprising the step of calculating a distortion of the reflection and applying this to pixels corresponding to the object visible in the reflection.

44. The method according to any one of claims 1 to 23 or 36 to 43 comprising the steps: projecting a structured light pattern onto the scene; anddetecting characteristics of an imaged object based on deformations detected in the structured light pattern in processed images.

Images