Controller, control method, monitoring system in cabin and vehicle

Abstract

A controller for an in-cabin monitoring system of a vehicle, the in-cabin monitoring system comprising a controller, a time-of-flight device configured to perform time-of-flight measurements to obtain time-of-flight data, and a radar device configured to perform radar measurements to obtain radar data, the controller comprising circuitry, the circuitry is configured to: obtain time-of-flight data and radar data; performing object / passenger monitoring in the cabin based on the time-of-flight data and the radar data; detecting a region of interest in the cabin based on the time-of-flight data; and adjusting an operating mode of the in-cabin monitoring system based on the detected region of interest.

Description

Technical Field

[0001] The present disclosure generally relates to control and control methods of an in-cabin monitoring system for a vehicle, an in-cabin monitoring system, and a vehicle. Background Art

[0002] Currently, in-cabin monitoring systems for monitoring objects / passengers in the cabin of a vehicle are typically based on direct sensing, using RGB ("Red-Green-Blue"), IR ("Infrared"), iToF ("Indirect Time of Flight") and radar sensing systems.

[0003] For example, when the vehicle is a car, some systems employ driver monitoring and passenger detection and identification. Typically, only a single sensor has been used for driver monitoring, and another completely separate sensing system has been used for passenger detection, where, for example, "child left alone in car" represents a use case.

[0004] Radar technology is used, for example, in cabin monitoring systems to detect the presence of children (whether fully obscured or not) and to count the number of occupants, however its performance for these basic functions, as a stand-alone radar-based system, may depend on a priori knowledge of the seat position and footwell in the vehicle.

[0005] In some cases, as is generally known, radar solutions may have certain detection characteristics and may require complex algorithms to process all in-cabin data to provide all required functionality. For example, a certain distance between occupants may be assumed, while there may be situations where two occupants are closer to each other and therefore merged into one spot detection.

[0006] While the technology exists for in-cabin monitoring systems, it is often desirable to improve upon existing technologies. Summary of the invention

[0007] According to a first aspect, the present disclosure provides a control of an in-cabin monitoring system for a vehicle, the in-cabin monitoring system comprising a controller, a time-of-flight device configured to perform time-of-flight measurements to obtain time-of-flight data, and a radar device configured to perform radar measurements to obtain radar data, the controller comprising a circuit configured to:

[0008] Obtain flight time data and radar data;

[0009] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data;

[0010] detecting an area of ​​interest in the cabin based on the time-of-flight data; and

[0011] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0012] According to a second aspect, the present disclosure provides a control method for an in-cabin monitoring system for a vehicle, the in-cabin monitoring system comprising a controller, a time-of-flight device configured to perform time-of-flight measurement to obtain time-of-flight data, and a radar device configured to perform radar measurement to obtain radar data, the control method comprising:

[0013] Obtain flight time data and radar data;

[0014] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data;

[0015] detecting an area of ​​interest in the cabin based on the time-of-flight data; and

[0016] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0017] According to a third aspect, the present disclosure provides an in-cabin monitoring system for a vehicle, comprising:

[0018] a time-of-flight device configured to perform a time-of-flight measurement to obtain time-of-flight data;

[0019] a radar device configured to perform radar measurements to acquire radar data; and

[0020] A controller includes a circuit configured to:

[0021] Get time-of-flight data and radar data,

[0022] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data,

[0023] Detecting regions of interest in the cabin based on time-of-flight data, and

[0024] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0025] According to a fourth aspect, the present invention provides a vehicle, the vehicle comprising an in-cabin monitoring system, the in-cabin monitoring system comprising:

[0026] a time-of-flight device configured to perform a time-of-flight measurement to obtain time-of-flight data;

[0027] a radar device configured to perform radar measurements to acquire radar data; and

[0028] A controller includes a circuit configured to:

[0029] Get time-of-flight data and radar data,

[0030] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data,

[0031] Detecting regions of interest in the cabin based on time-of-flight data, and

[0032] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0033] Further aspects are set out in the dependent claims, the following description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Embodiments are described by way of example with reference to the accompanying drawings, in which:

[0035] Figure 1 schematically illustrates a side view of an embodiment of a vehicle including an in-cabin monitoring system;

[0036] Figure 2 An embodiment of a vehicle including an in-cabin monitoring system is schematically illustrated in a bird's-eye view;

[0037] Figure 3 A block diagram schematically illustrates an embodiment of an in-cabin monitoring system; and

[0038] Figure 4 The flowchart of an embodiment of a control method for a vehicle cabin monitoring system is schematically shown. DETAILED DESCRIPTION

[0039] In giving reference Figure 1 Before describing in detail the embodiments of the present invention, an overall explanation is given.

[0040] As mentioned at the outset, currently, in-cabin monitoring systems for monitoring objects / passengers in the cabin of a vehicle are typically based on direct sensing using RGB (“Red-Green-Blue”), IR (“Infrared”), iToF (“Indirect Time of Flight”) and radar sensing systems.

[0041] As mentioned above, some systems use driver monitoring and passenger detection and identification when the vehicle is a car, for example. Typically, only one single sensor is used for driver monitoring, and another completely independent sensing system is used for passenger detection, where, for example, "children are left alone in a car" represents a use case.

[0042] Driver monitoring currently mainly involves driver face monitoring, which is usually done using a high spatial resolution 2D IR camera to extract facial features and activity. The typical location for a camera monitoring system is the dashboard, assuming the face is clearly seen without occlusion. In addition to facial features, for example, driver monitoring systems monitor the driver's head orientation.

[0043] Another setup possibility includes a rear-mirror positioned camera with a wider field of view (FoV), which potentially reduces the spatial resolution of facial features but can provide extended information about the entire cockpit.

[0044] It has been recognized that, for example, a dashboard high-resolution camera can be used in conjunction with a larger FoV camera placed in the rearview mirror viewing position (or some other application-driven position), which can improve the performance of the system through sensor fusion.

[0045] In recent years, occupant and child detection systems have moved from indirect sensing (e.g. sensing of opening (rear) doors) to direct sensing, where for example radar sensors play a role due to their ability to detect small movements under occlusion (i.e. blocked by vision).

[0046] Such radar sensors are also used, for example, in cabin monitoring systems to monitor for the presence of children (whether fully obscured or not) and for counting the number of occupants, however, their performance as a standalone radar-based system on these basic functions may depend on a priori knowledge of the seat positions and footwells in the vehicle. In some cases, as is generally known, radar solutions may have certain detection characteristics and may require complex algorithms to process all cabin data to provide all required functionality. For example, a certain distance between occupants may be assumed, while there may be situations where two occupants are closer to each other and may therefore be merged into one blob detection (thus missing some occupants due to limited resolution and signal-to-noise ratio (SNR)).

[0047] Another example could be a child or animal sitting on an adult's lap. Furthermore, if the passenger is not moving and is, for example, sleeping, the presence of the passenger may be missed due to the unclear classification between an empty seat and a passenger who is not moving at all. Further, in some cases, radar-based solutions may not be optimized to detect the posture / shape and activity type of the occupant, which may become more important for future safety applications - for example: the car is moving and a child is doing something that may be unsafe in the back.

[0048] It has been recognised that, while the position of the seats is currently mostly fixed, the front seats may be movable and the rear seats may be removed and / or moved down so that the footwell area may also change. Further, it has been recognised that once the level of autonomy increases, such as level 3 and higher, it is expected that the seat positions will be more movable and the footwells will also change and, therefore, detecting and identifying areas of interest for monitoring becomes more important.

[0049] However, it has been further recognised that radar sensing equipment may be used to detect, for example, the slightest breathing or movement of a child beneath a blanket.

[0050] However, it has been recognized that in some cases, reflections of radar signals caused by small and slow movements may have limited SNR levels, which may require the application of specific processing techniques to extract information about, for example, the subtle breathing or movement of a child. This relatively low SNR, in addition to the possible presence of clutter (e.g., vibrations of the vehicle), may degrade the performance of the system.

[0051] Even though there are some typical so-called Doppler signatures for specific gestures that can be used to identify them (finger rotation, palm push, etc.), in some cases the information extracted from the radar signal alone may not be sufficient to provide the required level of certainty about the gesture performed.

[0052] Thus, it has been recognized that in some embodiments, in order to achieve higher performance levels and reduce the computing resources required to process radar signals, the 3D position and size of the target of interest (e.g., a person covered with a blanket or a child seat) should be known.

[0053] It has been recognized that in addition to improving performance compared to radar-only systems, the additional information from time-of-flight sensors should be used.

[0054] Therefore, in some embodiments, sensor fusion of radar signals and iToF ("indirect time of flight") 3D sensing technology is employed.

[0055] It has been recognized that improvements can be achieved by: (1) dynamically adapting the 3D region of interest, where, for example, gesture recognition should be performed, and (2) by providing additional information about the 3D structure of the hand and its position in the region of interest, which information can, for example, help distinguish gestures such as clockwise / counterclockwise finger rotation or finger tapping in and out, etc.

[0056] Furthermore, it has been recognized that in some embodiments, given specific information about a target of interest, the radar sensing device functionality should be optimized to optimize its device parameters and thus further improve the processing of data for identification and decision making. In some embodiments, radar internal adjustment parameters include: 1) field of view and / or 2) angular / range resolution, and / or 3) power mode and / or 4) frame rate.

[0057] Furthermore, in some embodiments, in addition to sensor fusion of ToF and radar data (eg, using the output of one device's processing to adjust the processing of the other device's data), adaptive adjustment of real-time sensing device parameters is employed.

[0058] In some embodiments, the in-cabin monitoring system covers complete in-cabin monitoring based on radar and time-of-flight data fusion, including child presence monitoring, passenger posture / activity / interaction monitoring, identification of obscured and unobstructed objects / passengers in the vehicle and activity monitoring for safety / crash protection purposes and comfort.

[0059] Therefore, some embodiments relate to control of an in-cabin monitoring system for a vehicle, the in-cabin monitoring system comprising a controller, a time-of-flight device configured to perform time-of-flight measurements to obtain time-of-flight data, and a radar device configured to perform radar measurements to obtain radar data, wherein the controller comprises circuitry configured to:

[0060] Obtain flight time data and radar data;

[0061] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data;

[0062] detecting an area of ​​interest in the cabin based on the time-of-flight data; and

[0063] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0064] Some embodiments relate to an in-cabin monitoring system for a vehicle, wherein the in-cabin monitoring system includes: a time-of-flight device configured to perform a time-of-flight measurement to obtain time-of-flight data; a radar device configured to perform a radar measurement to obtain radar data; and a controller, wherein the controller includes a circuit configured to:

[0065] Get time-of-flight data and radar data,

[0066] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data,

[0067] Detecting regions of interest in the cabin based on time-of-flight data, and

[0068] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0069] Some embodiments relate to a vehicle, wherein the vehicle includes an in-cabin monitoring system, the in-cabin monitoring system including a time-of-flight device configured to perform time-of-flight measurements to obtain time-of-flight data, a radar device configured to perform radar measurements to obtain radar data, and a controller, wherein the controller includes circuitry configured to:

[0070] Get time-of-flight data and radar data,

[0071] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data,

[0072] Detecting regions of interest in the cabin based on time-of-flight data, and

[0073] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0074] The vehicle may be a car, train, airplane, etc. having a cabin with space for objects / passengers.

[0075] The time-of-flight device may be a direct time-of-flight device, which may also be referred to as LiDAR (“Light Detection And Ranging”), or a generally known indirect time-of-flight device.

[0076] Thus, in some embodiments, the time-of-flight device is an indirect time-of-flight device. Typically, in some embodiments, the indirect time-of-flight device includes one or more image sensors, such as a pixel array of multiple CAPD ("current-assisted photon demodulator") pixels (two-tapped in some embodiments), one or more light sources, such as a laser diode array (e.g., emitting visible light or infrared light), and a controller for controlling the overall operation.

[0077] As is generally known, in some embodiments, an indirect time-of-flight device performs a time-of-flight measurement based on multiple correlated measurements (typically four correlated measurements), wherein a modulated signal is applied to an image sensor and a light source with a phase shift (e.g., 0°, 90°, 180°, and 270°) (the signal is phase shifted between the image sensor and the light source).

[0078] In some embodiments, time-of-flight data is acquired based on these correlation measurements by analog-to-digital conversion in order to acquire an amplitude image or a phase image as is commonly known, wherein the amplitude image represents 2D image formation and the phase image represents 3D image information based on which a 3D point cloud can be generated.

[0079] Typically, in some embodiments, the radar device transmits high frequency electromagnetic waves (eg, in the GHz (gigahertz) range, such that the wavelength is in the mm (millimeter) range) and detects the portion of the electromagnetic waves that has been reflected by an object / passenger.

[0080] In some embodiments, the radar device measures (acquires), for example, the relative (radial) velocity of an object and the distance to the object, or the relative (radial) velocity of an object, the distance to the object, and at least one detection angle (azimuth and / or elevation) to the object.

[0081] Furthermore, in some embodiments, the radar device includes multiple transmit / receive antennas, and the angular / range / velocity resolution and / or field of view are adjusted by, for example, applying phase-shifted modulation signals to the transmit / receive antennas, as is known.

[0082] In some embodiments, the radar device uses an antenna with a fixed pattern of transmit and receive elements and the shape of the elements (patches on the PCB connected to the TX and RX channels of the radar), and the field of view is fixed as the area in space (solid angle) to which most of the transmitted signals will be transmitted and from which the reflected signals will be received. In some embodiments, for example, it switches between two independent antennas on the PCB (control command switching) to adjust the field of view. In some embodiments, it switches between different antenna arrays of the radar device to adjust the field of view of the radar device.

[0083] In some embodiments, the angle / range / velocity resolution and power mode are adjusted by adjusting radar signal parameters and / or adjusting the number of activated RX / TX channels.

[0084] The radar data may include at least one of the relative (radial) velocity of the object, the distance to the object, and the detection angle (azimuth and elevation) to the object, which are generally known, and thus represent 3D information about the object.

[0085] The circuit may be based on or may include or may be implemented as integrated circuit logic, or may be implemented by a CPU (central processing unit), an application processor, a graphics processing unit (GPU), a microcontroller, an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), etc. or a combination thereof.

[0086] The functions may be implemented by software executed by a processor such as a microprocessor. The circuit may be based on or may include or may be implemented by typical electronic components configured to implement the functions described herein. The circuit may be based on or may include or may be implemented partly by typical electronic components and integrated circuit logic and partly by software.

[0087] The circuitry may include data storage capabilities for storing data, such as memory that may be based on semiconductor memory technology (eg, RAM, EPROM, etc.) or magnetic storage technology (eg, hard drive), or the like.

[0088] The circuit may include a data bus for receiving and sending data via the data bus. The circuit may implement a communication protocol for receiving and sending data via the data bus. The data bus may be or may be based on a controller area network (CAN) bus, an I2C (inter-integrated circuit) interface, or the like.

[0089] The controller may acquire the time-of-flight data and the radar data via a data bus (eg, a CAN bus) connecting the time-of-flight device and the radar device.

[0090] The controller may convert the radar data and the time-of-flight data into a single coordinate system to overlay / combine the different data sets (eg, based on a known mounting location / orientation within the cockpit).

[0091] Typically, control performs object / passenger monitoring in the cabin based on time-of-flight data and radar data.

[0092] In some embodiments, performing object / passenger monitoring includes detecting presence of an object / passenger in the cabin, registering the detected object / passenger, tracking the detected object / passenger in the cabin, and identifying behavior of the detected object / passenger.

[0093] In some embodiments, performing object / passenger monitoring includes detecting occlusion of an object / passenger. In some embodiments, control supplements information extracted based on time-of-flight data with information extracted based on radar data, for example, for detecting occlusion of an object / passenger.

[0094] The controller detects areas of interest in the cockpit based on the time-of-flight data.

[0095] The region of interest may be associated with an object (e.g. a child seat, some luggage, etc.) or a passenger (e.g. a driver, a child, an animal, etc.) such that, for example, 2D / 3D coordinates representing the (entire) region / space around the object / passenger represent the region of interest.

[0096] In some embodiments, the region of interest is a three-dimensional region of interest. In some embodiments, the region of interest corresponds to a three-dimensional box.

[0097] In some embodiments, control determines the three-dimensional location, orientation, and / or size and / or shape of the region of interest based on the time-of-flight data. In some embodiments, control determines the three-dimensional location, orientation, and / or size and / or shape of the region of interest based on depth information (e.g., phase image) and / or reflectance information (e.g., amplitude image).

[0098] In some embodiments, the region of interest represents a first rough estimate of important components in the cabin for further monitoring. In some embodiments, the region of interest can be detected based on the amplitude image by simple edge detection, etc., without the need to construct a 3D point cloud (at multiple time points) based on the phase image to save computing resources.

[0099] Typically, in some embodiments, bidirectional adjustment of two sensing pipelines for in-cabin monitoring systems is employed. For sensor fusion, time-of-flight processing data (acquired by the time-of-flight sensor) is used for coarse 3D positioning and identification of 3D regions of interest, related to the volume of occupants, child seats, children, animals, and other objects of interest. Once the coarse position is detected and an initial estimate of the occupant position is known, in some embodiments, radar sensing is used for more detailed occupant inspection and monitoring, for example in terms of small / fine movements and breathing. This may be important for situations when the passenger is not moving at all (e.g., sleeping), in which case it is difficult for the radar system alone to distinguish between the background (seat) and the passenger. The use of time-of-flight data can help avoid passenger detection and positioning losses (i.e., false negatives). In addition to point cloud data, in some embodiments, (time-of-flight) confidence data can also be used to detect regions of interest such as the head when they are not visually obstructed.

[0100] In other words, in some embodiments, it is envisaged to iteratively adjust data processing and data acquisition to optimize the in-cabin monitoring system for a given situation, wherein the in-cabin monitoring system can be optimized, for example, with respect to detection accuracy of objects / passengers and their behavior or with respect to detection speed or with respect to power consumption or a combination thereof.

[0101] Therefore, the controller adjusts the operating mode of the in-cabin monitoring system based on the detected region of interest.

[0102] In some embodiments, the operating modes are predetermined. For example, the in-cabin monitoring system can operate in a reduced power mode, an object / passenger presence detection / recording mode for detecting / recording an area of ​​interest, an object / passenger tracking mode, and an object / passenger behavior recognition mode.

[0103] In some embodiments, adjusting the operating mode includes adjusting a data processing mode of the controller and / or adjusting a data acquisition mode of the time of flight device and / or the radar device.

[0104] In some embodiments, adjusting the data acquisition mode of the radar device includes adjusting device parameters including at least one of field of view, frame rate, angular / range / velocity resolution, and power mode.

[0105] In some embodiments, adjusting the data acquisition mode of the time-of-flight device includes adjusting device parameters including at least one of integration time, frame rate, modulation frequency, illumination field, multi-wavelength light emission, and power mode.

[0106] In some embodiments, the circuitry is configured to extract information about the presence of a passenger in the cabin and / or the behavior of the passenger based on the radar data using the detected region of interest as prior knowledge.

[0107] In some embodiments, the circuit is configured to detect at least one sub-region of interest in the region of interest based on the time-of-flight data using the extracted information as prior knowledge.

[0108] In some embodiments, the circuitry is configured to extract information about the presence of passengers in the cabin and / or the behavior of passengers in the sub-region of interest based on the time-of-flight data and the radar data.

[0109] As mentioned above, it is contemplated to iteratively adjust the operating mode of the in-cabin monitoring system with respect to data processing and / or data acquisition.

[0110] illustrate:

[0111] The controller may detect regions of interest based on time-of-flight data in an object / passenger presence detection / recording mode, wherein time-of-flight device parameters are adapted to detect static features over a large spatial range.

[0112] For example, in some embodiments, the integration time, the light emission intensity according to the integration time, and the modulation frequency of the time-of-flight device are adjusted (eg, large integration time, low light emission intensity, and small modulation frequency).

[0113] Furthermore, in some embodiments, the controller detects the region of interest based on the amplitude image by simple edge detection, etc., without constructing a 3D point cloud based on the phase image (at multiple time points), so as to save computing resources.

[0114] Once a region of interest is detected, the controller may adjust radar device parameters such that the angular / range resolution (and / or frame rate and / or field of view) is optimized for the detected region of interest.

[0115] It has been recognised that, for example, in object / passenger presence detection / registration modes, when the detected region of interest is used as prior knowledge in the data processing of the radar data, the detection speed / accuracy can be improved and the power consumption can be reduced since the confidence level may already be sufficient when processing fewer time / space points.

[0116] Thus, as described above, in some embodiments the circuitry is configured to extract information about the presence of a passenger in the cabin and / or the passenger's behavior based on the radar data using the detected region of interest as prior knowledge.

[0117] The object / passenger tracking mode may be similar to the object / passenger presence detection / registration mode, however, the integration time, frame rate and modulation frequency of the time-of-flight device may be adjusted to account for rapid scene changes in, for example, the cabin.

[0118] Furthermore, data processing of the time-of-flight data may include, for example, generating a 3D point cloud based on the phase image in order to, for example, take into account 3D movements of objects / passengers.

[0119] In some embodiments, the controller switches between the object / passenger presence detection / registration mode and the object / passenger tracking mode at predetermined intervals or based on a random number, etc.

[0120] It is further appreciated that in addition to object / passenger presence detection and tracking, analysis of the object / passenger's behavior is useful in some embodiments, and the operating mode should be adjusted accordingly.

[0121] In particular, it has been recognized that certain parts of passengers (eg, head, chest, hands, fingers, etc.) should be detected and analyzed to derive meaning from their behavior.

[0122] Therefore, in some embodiments, in the object / passenger behavior recognition mode, data processing and / or data acquisition is adjusted to optimize behavior recognition.

[0123] For example, the controller can adjust the time-of-flight device parameters more frequently and reliably to detect smaller areas in the cockpit by, for example, adjusting the frame rate, modulation frequency, and illumination field (e.g., increasing the frame rate, increasing the modulation frequency, and decreasing the illumination field). Similarly, for example, the controller adjusts the field of view and range / angle resolution of the radar device toward smaller areas and higher resolution.

[0124] Furthermore, in terms of data processing, it has been recognized that information extracted based on radar data should be used to detect sub-regions of interest within the detected region of interest using time-of-flight data, since, as mentioned above, radar data provides additional information about visual occlusions and small movements, which can improve the analysis of amplitude and phase images.

[0125] Thus, in some embodiments, the circuit is configured to detect at least one sub-region of interest in the region of interest based on the time-of-flight data using information extracted based on the radar data as prior knowledge.

[0126] The sub-region of interest may be a smaller region within the region of interest, for example, the sub-region of interest may be a passenger and the sub-region of interest may be the head, chest, hands, fingers of the passenger.

[0127] As a next iterative step, in some embodiments, to further improve detection accuracy and thus enable behavior recognition, these two data are used in the sub-region of interest to increase the confidence level.

[0128] Thus, in some embodiments, the circuitry is configured to extract information about the presence of passengers in the cabin and / or the behavior of passengers in the sub-region of interest based on the time-of-flight data and the radar data.

[0129] In some embodiments, for example, in the reduced power mode, data processing is adjusted to be performed less frequently than in the normal power mode and / or based on only a subset of data (e.g., only a spatial subset of data and / or a temporal subset of data). In addition, in such embodiments, for example, data acquisition is adjusted to acquire data less frequently than in the normal power mode and / or at a lower light / radar wave emission intensity.

[0130] The controller may determine to adjust the operating mode to a reduced power mode based on vehicle status (eg, when the vehicle is parked) and / or based on the region of interest (eg, when no region of interest is detected or only luggage presence is detected).

[0131] Thus, in some embodiments, the circuitry is configured to place the time-of-flight device and the radar device in a reduced power mode when a predetermined state of the vehicle is detected, and to detect passenger presence in the cabin based on radar data acquired in the reduced power mode.

[0132] In this embodiment, the time-of-flight device is also placed in a reduced power mode or in a sleep mode until a control command wakes up the time-of-flight device.

[0133] In some embodiments, the circuitry is configured to place the time of flight device in a normal power mode for detecting the region of interest in response to detecting the presence of a passenger in the cabin.

[0134] In some embodiments, the circuit is configured to place the radar device in a normal power mode in response to detecting the region of interest.

[0135] Thus, in some embodiments, the time of flight device is awakened only when the radar data indicates the presence of a passenger, since radar devices typically consume less power than time of flight devices.

[0136] Additionally, the control instructs the radar device to operate in the normal power mode only when the time-of-flight data indicates an area of ​​interest for further inspection by the radar device.

[0137] Some embodiments relate to a (corresponding) control method for an in-cabin monitoring system for a vehicle, wherein the in-cabin monitoring system comprises a controller, a time-of-flight device configured to perform time-of-flight measurements to obtain time-of-flight data, and a radar device configured to perform radar measurements to obtain radar data, wherein the control method comprises:

[0138] Obtain flight time data and radar data;

[0139] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data;

[0140] detecting an area of ​​interest in the cabin based on the time-of-flight data; and

[0141] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0142] The control method may be performed by the control as described herein.

[0143] In some embodiments, when executed on a computer and / or processor, the method described herein is also implemented as a computer program that causes the computer and / or processor to perform the method. In some embodiments, a non-transitory computer-readable recording medium is also provided in which a computer program product is stored, which, when executed by a processor (such as the processor described above), causes the method described herein to be performed.

[0144] return Figure 1 , schematically shows a side view of an embodiment of a vehicle 1 including an in-cabin monitoring system 3, which is described below with reference to Figure 1 , Figure 2 and Figure 3 Discussion, among which Figure 2 An embodiment of a vehicle 1 including an in-cabin monitoring system 3 is schematically shown in a bird's eye view, and wherein Figure 3 An embodiment of the in-cabin monitoring system 3 is schematically shown in a block diagram.

[0145] The vehicle 1 has a cabin 2 , wherein the vehicle 1 is a motor vehicle in which an in-cabin monitoring system 3 is installed for monitoring objects / passengers in the cabin 2 .

[0146] The in-cabin monitoring system 3 includes an indirect time-of-flight device 4 (hereinafter iToF device), a radar device 5 and a controller 6 .

[0147] The iToF device 4, the radar device 5, and the controller 6 are connected to a CAN bus (not shown) to exchange data and commands via the CAN bus. The controller 6 may further exchange data 7 with other sensors or actuators or control units of the vehicle 1. For example, the controller 6 may receive vehicle state data indicating the state of the vehicle 1 (e.g., whether the vehicle 1 is moving, whether the engine is on, whether the door is closed, etc.) to detect whether the vehicle 1 is in a predetermined state.

[0148] Figure 1 and Figure 2, the use positioning of the iTOF device 4 and the radar device 5 in the cockpit 2 is shown, wherein the iToF device 4 is mounted in a middle top-roof position (with minimal visual obstruction to the seats and footwells) and the radar device 5 is mounted in the area around the rearview mirror of the vehicle 1 and oriented towards the passengers for optimal discrimination in the radial distance direction.

[0149] In other embodiments, the positioning of both the radar device 5 and the iToF device 4 may also take other absolute and relative positions, depending on the type of vehicle 1 and the application scenario. However, the main constraint is the sufficient overlapping area of ​​the field of view, such as Figure 1 and 2 The dashed and dotted lines in FIG. 4 are used for accurate calibration in offline and online modes and are the maximum possible non-overlapping 3D areas that can be used to supplement one sensing system by another sensing system. In addition, in other embodiments, it is proposed to independently define the position of the sensor by utilizing the best position of two sensors in a specific vehicle.

[0150] In some embodiments, the iToF device 4 and the radar device 5 are coupled to actuators in order to adjust the emission / detection direction, for example based on the detected region of interest.

[0151] The iToF device 4 may include an image sensor and a light source to generate a field of view ( Figure 1 and Figure 2 ), or the iToF device 4 may include, for example, two image sensors and two light sources to monitor opposite directions. Thus, the iToF device 4 may include two independent sub-devices for improving the detection range and accuracy, wherein the corresponding data are fused to generate a combined image / depth information.

[0152] The radar device 5 may include one or more antenna arrays, each antenna array including a plurality of antennas for transmitting and receiving radar signals.

[0153] In some embodiments, some objectives of the in-cabin monitoring system 3 using radar-iToF fusion include:

[0154] Specific locations are monitored: front seats, rear seats and footwell.

[0155] The number of passengers in the vehicle 1 is detected when the vehicle is stationary or moving.

[0156] Detects whether a child is left alone in a car when the car is stationary, the child is asleep or awake, and the child is covered or uncovered (visibly obscured).

[0157] Detecting behavior of passengers (e.g., the car is being driven and a child is doing something that may be unsafe), where detecting behavior includes: using ToF data to detect parts of the body such as the head / face / chest (and e.g., whether the eyes are open), using radar data to detect whether a passenger (e.g., a child) is sleeping (e.g., the passenger is not moving and breathing indicates that the passenger is sleeping), and refining smaller sub-regions of interest within a larger region of interest for radar checks for breathing and for using 2D confidence images (amplitude images) for visible facial features.

[0158] Adjusting movable seats and seat postures in future cabin autonomous systems.

[0159] Reference Figure 3 The controller 6 includes a processor that executes a control application 61 for controlling the overall operation of the in-cabin monitoring system 3 , a ToF processing block 62 for processing ToF data acquired from the iToF device 4 , and a radar processing block 63 for processing radar data acquired from the radar device 5 .

[0160] Typically, the controller 6 acquires time-of-flight data (hereinafter ToF data) from the iToF device 4 and radar data from the radar device 5 via a CAN bus for sensor fusion, thereby performing object / passenger monitoring in the cabin 2 based on the ToF data and the radar data.

[0161] Performing object / passenger monitoring includes detecting the presence of an object / passenger in the cabin, registering the detected object / passenger, tracking the detected object / passenger in the cabin, and identifying the behavior of the detected object / passenger. In some embodiments, performing object / passenger monitoring includes detecting occlusion of an object / passenger.

[0162] Typically, the ToF data is processed in the ToF processing block 62, and the radar data is processed in the radar processing block 63, wherein the control application 61 provides information extracted based on the ToF data (processed ToF data) as prior knowledge to the radar processing block 63, and provides information extracted based on the radar data (processed radar data) as prior knowledge to the ToF processing block 62. However, the control application 61 may also perform processing using the ToF data and the radar data based on the two data sets.

[0163] In general, the controller 6 detects a region of interest in the cabin 2 based on the ToF data, and adjusts the operating mode of the in-cabin monitoring system 3 based on the region of interest.

[0164] Adjusting the operation mode includes adjusting a data processing mode of the controller 6 and / or adjusting a data acquisition mode of the iToF device 4 and / or the radar device 5 .

[0165] Adjusting the data acquisition mode of the radar device 5 includes adjusting device parameters including at least one of a frame rate, a field of view, an angular / range / velocity resolution, and a power mode.

[0166] Adjusting the data acquisition mode of the iToF device 4 includes adjusting device parameters including at least one of accumulation time, frame rate, modulation frequency, illumination field, multi-wavelength light emission, and power mode.

[0167] In addition, in some embodiments, the controller 6 adjusts the data processing mode in the radar processing block 63 to process only the radar data corresponding to the field of interest in the field of view of the radar device. For example, the number of angles in the radar data being processed is limited. A certain range of angles (e.g., from -45° to +45°, with each 15° step) can be maintained and processed while discarding other information (<-45° and >+45°). Therefore, in some embodiments, the field of view of interest is adjusted during the processing stage.

[0168] exist Figure 3 In, a general overview of the interaction framework is described, including communication at the sensor system level (4, 5) and the processing block level (62, 63). In addition, as described above, a control application 61 is shown, which collects processed data and uses the extracted information to adjust at the sensor system level and the processing block level.

[0169] In this manner, the controller 6 influences the manner in which the sensed data is sensed and processed based on a priori knowledge of the extracted same and other sensor processing data, in what may be referred to as automatic bidirectional inter- and intra-frame calibration.

[0170] Typically, as described above, bidirectional alignment of two sensing pipelines for in-cabin monitoring systems is employed. For sensor fusion, ToF data is used for coarse 3D localization and identification of 3D regions of interest related to the volume of occupants, child seats, children, animals, and other objects of interest. Once the coarse position is detected and an initial estimate of the passenger position is known, radar sensing is used for more detailed passenger inspection and monitoring, for example in terms of minor / fine movements and breathing.

[0171] Therefore, the controller 6 uses the detected region of interest as a priori knowledge to extract information about the presence of a passenger in the cabin and / or the behavior of the passenger based on the radar data.

[0172] This can be important for situations when the passenger is not moving at all (e.g. sleeping), in which case it is difficult for a radar system alone to distinguish between the background (seat) and the passenger. Using ToF data can help avoid missed passenger detection and localization, i.e. false negatives. In addition to the point cloud data (based on the phase image), the ToF confidence data (amplitude image) can also be used to detect, for example, the head region of interest in case it is not visually occluded.

[0173] In terms of radar device parameters adjusted at the sensor system level, as described above, the controller 6 adjusts, for example, the power mode and field of view to focus more on the selected 3D region of interest, and thus increases the range and angular resolution of the acquired radar data, depending on input from the TOF processing block 62. In other cases, when there is not much activity, the controller 6, for example, reduces the power of the radar device 5 and instructs to zoom in on the field of view to track more global changes.

[0174] Generally, the use of radar signals may avoid issues regarding occlusion of objects / passengers, as detection issues due to different materials of objects / passengers in the car are reduced, as is the case with the iToF device 4 .

[0175] Thus, there may be additional benefits to fusing radar data with ToF data over the same region of interest, for example in situations where an object / passenger is partially visually obscured or invisible due to the material of the object by the IR technology of the iToF device 4. In such situations, combining radar and ToF information may facilitate passenger identification and tracking of their movements compared to situations where only ToF technology is used.

[0176] Furthermore, it has been recognised that using ToF information to supplement radar motion sensing of a specific passenger or passengers and their interactive behaviour may improve identification of behaviour of passengers in the vehicle 1 , as the radar device 5 typically only provides approximate information about the position of the passenger in question.

[0177] Thus, ToF data is used in combination with radar data to detect, for example, the body / arms / head of an occupant to understand the movement of these parts and therefore infer their behavior. This can allow for full and extended in-cabin safety monitoring with multiple objects and tasks, such as detecting and recording the presence of passengers in the vehicle 1 as well as their activities, precise location and interactions with other passengers and objects in the car.

[0178] Therefore, the controller 6 detects at least one sub-region of interest in the region of interest based on the ToF data using information extracted based on the radar data as a priori knowledge.

[0179] Furthermore, the controller 6 extracts information about the presence of passengers in the cabin and / or the behavior of passengers in the sub-area of ​​interest based on the time-of-flight data and the radar data.

[0180] As described above, with respect to the iToF device parameters adjusted at the sensor system level, the radar processing block 63 provides information to adjust ToF data acquisition at the system level using the device parameters: integration time, frame rate, modulation frequency, illumination field, multi-wavelength light transmitter adjustment, and power mode. The controller 6 uses the output from the iToF device 4 to confirm the presence of a human, animal, or child seat and confirm the results obtained based on the radar data.

[0181] In some embodiments, some envisioned use case scenarios include:

[0182] In static conditions (e.g., not much activity in the cabin), the radar device 5 operates in a reduced power mode, monitoring any changes that may occur (because, typically, the radar device 5 consumes less power than the iToF device). In such a scenario, the iToF device can operate in a reduced power mode at a reduced frame rate (e.g., 2-3fps) and reduced transmitter light intensity. When the radar device 5 begins to notice some changes, the iToF device 4 begins to detect the 3D area of ​​interest in normal power mode at a higher fps and using a higher level of transmitter light intensity. The controller 6 then optimizes the radar device 5 parameters (e.g., reducing the field of view and increasing the angular resolution) to detect smaller movements and other important details.

[0183] Once the fused radar and ToF processing blocks 63 and 62 more accurately indicate an area of ​​interest for inspection, perhaps even portions of an occupant's body, the controller 6 adjusts the ToF device 4 parameters, such as the modulation frequency, to reduce the distance range as the accuracy of the depth information increases, thereby getting into finer details about the object's structure and motion, thus enabling more accurate activity identification.

[0184] The illumination field of the iToF device 4 is adjusted based on feedback from the radar device 5 to manage power.

[0185] For example, the accumulation time of the iToF device 4 is adjusted according to the scene distance and / or reflectivity.

[0186] When information about a sub-region of interest is known the frame rate of the iToF device 4 is increased so that, for example, only this part of the data is extracted and further processed to eventually be used together with the radar data in the ring.

[0187] Figure 4 A flow chart schematically illustrates an embodiment of a control method 100 for an in-cabin monitoring system of a vehicle, which is discussed below.

[0188] As reference Figure 1 , Figure 2 and Figure 3 As discussed, the control method 100 is executed by the controller 6 of the in-cabin monitoring system 3 of the vehicle 1 .

[0189] At 101 , time-of-flight data and radar data are acquired as discussed herein.

[0190] At 102 , object / passenger monitoring in the cabin is performed based on time-of-flight data and radar data, as discussed herein.

[0191] At 103 , a region of interest in the cockpit is detected based on the time-of-flight data, as discussed herein.

[0192] At 104 , an operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest, as discussed herein.

[0193] At 105 , a data processing mode of a control and a data acquisition mode of a time of flight device and a radar device are adjusted as discussed herein.

[0194] At 106, as discussed herein, adjusting the data acquisition mode of the radar device includes adjusting device parameters including at least one of frame rate, field of view, angular / range / velocity resolution, and power mode, and adjusting the data acquisition mode of the time of flight device includes adjusting device parameters including at least one of accumulation time, frame rate, modulation frequency, illumination field, multi-wavelength light emission, and power mode.

[0195] At 107 , information regarding the presence of passengers in the cabin and / or the behavior of passengers is extracted based on the radar data using the detected region of interest as prior knowledge, as discussed herein.

[0196] At 108 , at least one sub-region of interest in the region of interest is detected based on the time-of-flight data using information extracted based on the radar data as prior knowledge, as discussed herein.

[0197] At 109 , information regarding the presence of passengers and / or behavior of passengers in the cabin in the sub-region of interest is extracted based on the time-of-flight data and the radar data, as discussed herein.

[0198] Returning to the general explanation, some aspects of some embodiments of the in-cabin monitoring system as described herein are summarized:

[0199] It is envisioned to monitor the entire cabin, including rear seat activities and their correlation with front seat occupant presence and activities. Consider using the 3D point cloud data acquired from the iToF device in conjunction with a radar device, for example, the radar device is attached near the iToF device, or the radar device and the iToF device are split apart at the location of the car. Either mounted close or further away, in some embodiments, the two devices are precisely calibrated and complement each other to utilize both detection technology and redundancy for ultimate functional safety compliance.

[0200] Based on the position of the sensors and the acquired and fused data, in some embodiments, the occupants are registered and their postures are estimated, thereby describing the activities (e.g. driver body turned back, head turned back, driver out of seat position, child lying down - not moving (breathing), child moving, child sitting, etc.). By fusing the information of the iToF device and the radar device, in some embodiments, the entire cabin is covered in a multi-scale sense: from the coarse scale of counting the number of occupants (including children and animals), their 3D position and activities to more precise recognition tasks about each occupant, such as body shape, size, age, gender and other relevant information for safety issues in transportation.

[0201] In some embodiments, the iToF device is located in the middle roof area (see, e.g., Figure 1 and Figure 2 ) to have a minimum amount of visual obstruction, while the radar equipment should be optimally placed in the area around the reflector window to facilitate maximum discrimination in the radial range direction.

[0202] In some embodiments, ToF data is used to perform 3D monitoring and identification of a 3D volume around a specific object of interest in the cabin of a vehicle, such as a child seat or any other important object in the car that is required to be more accurately moved within the 3D box by the radar device.

[0203] For example, a particular case of interest is detecting a child seat using an iTOF device to detect a 3D bounding box of the child seat (with or without a child) and then using a radar device to focus only on this 3D area to be more sensitive and less prone to noise.

[0204] Within the bounding box, in some embodiments, the controller detects motion and breathing under occlusion through radar data fusion for improved 3D registration of occupant body pose, shape, and motion tracking.

[0205] Application scenarios include, for example: pre-crash protection with intervention and safety warnings and / or post-crash detection of posture / severity of injuries and health monitoring.

[0206] In some embodiments, iToF devices are used to reduce noise and accuracy in radar imaging by using prior knowledge acquired based on ToF data.

[0207] In some embodiments, a fused iToF device and a radar device are used to monitor the footwell area in the vehicle 1 and detect and identify objects and their movements.

[0208] In some embodiments, the fused iToF device and radar device are used to monitor driver behavior regarding driver distraction (e.g., leaning back into the back seat, leaning into the footwell area) and sudden illness (small movements of the head and the driver's breathing, and other medical issues that may affect driving safety).

[0209] In some embodiments, an iToF device is used in conjunction with a radar device to detect the head and chest of the occupant, and facial features (from the iToF confidence data) are used to infer sleep by narrowing the region of interest for breathing detection via the radar device.

[0210] In some embodiments, radar-based sensing is complemented with iToF-based sensing (regarding detection of the occupant's body / arms / head) to understand the movement of these parts and therefore draw conclusions about the occupant's behavior / activity.

[0211] In some embodiments, iToF based sensing is compensated in the case of visual occlusions and material issues due to the use of IR technology by using radar equipment to see through objects and also using semi-reflectivity to extract missing reflections.

[0212] In some embodiments, the complexity of radar detection is reduced, such as through predefined schemes, the radar device can be adjusted according to the detection context. For example: face / chest detection via iToF, and then detailed scan for human vital signs monitoring via radar device, and / or child seat detection via iToF, and additional detailed scan for breathing children only in specific areas instead of the entire cabin.

[0213] In some embodiments, an iTOF device is used to more accurately and robustly identify 3D objects and corresponding 3D regions of interest, such as a bounding box in a vehicle cabin, such as a child seat or any other important object / occupant in a car required to detect more accurate movement within this 3D box. Then, in some embodiments, a radar device is used for more accurate detection of living objects under occlusions / coverings, such as blankets.

[0214] In some embodiments, only this 3D area is focused on to more sensitively detect heartbeat or respiration.

[0215] In some embodiments, an iToF device is used in combination with a radar device to:

[0216] Detect the occupant's head and chest and use facial features to infer sleep (narrowing the region of interest for breathing detection).

[0217] The fact that radar devices are usually transparent to a certain extent while also leaving some reflectivity that is more or less scattered, depending on the material, is exploited. Thus, in some cases, the material-dependent properties of iToF devices and radar devices are exploited to compensate for weaknesses.

[0218] In some cases, the iToF device 4 may have limited performance for certain materials: hair, special clothes, etc., and there are some gaps in the depth and reflection data. Due to this, the 3D recognition task may reduce its performance and may fail. Fusion of ToF data with radar data can also facilitate the reconstruction of missing parts due to reduced iToF signal levels and / or object occlusions. ToF data can be used to obtain a rough estimate of the surface and volume of a 3D object, while radar data can allow the missing parts of the 3D object to be added through the fusion process to improve recognition performance.

[0219] In some embodiments, accurate 3D body and body features / head registration / pose estimation / tracking of adult occupants and children are extracted in real time.

[0220] In some embodiments, occupant activity recognition includes: monitoring of children's movements and interactions with other passengers in the vehicle, and driver behavior monitoring related to driving or non-driving tasks that may affect the safety of children and other occupants. For example, phone use while driving can be monitored or turning to see what other passengers are doing behind in the vehicle.

[0221] In some embodiments, monitoring of the car's footwells at the front and rear may be a scenario where iTOF-radar fusion may be beneficial in detecting and identifying objects, their size, and their movement.

[0222] In some embodiments, inter-sensor system adjustments based on outputs of other sensor processing blocks:

[0223] Use the ToF processing block output to adjust radar sensor settings in real time: frame rate, field of view, angular / range / velocity resolution, and power mode.

[0224] Use the radar processing block output to adjust the TOF sensor settings in real time: power level, fps, modulation frequency, integration time, illumination field, wavelength, and bandwidth of the wavelength of the transmitting active light source.

[0225] This setup reduces power consumption and processing power for in-cabin monitoring systems within cars.

[0226] It should be appreciated that the embodiments describe methods with an exemplary order of method steps. However, the specific order of method steps is given only for illustrative purposes and should not be interpreted as being binding.

[0227] If not stated otherwise, all units and entities described in this specification and claimed in the appended claims may be implemented as integrated circuit logic, for example on a chip, and if not stated otherwise, the functions provided by these units and entities may be implemented by software.

[0228] To the extent that the embodiments of the present disclosure described above are at least partially implemented using software-controlled data processing equipment, it should be understood that providing such software-controlled computer programs and the transmission, storage or other media through which such computer programs are provided are contemplated as aspects of the present disclosure.

[0229] It should be noted that the present technology can also be configured as described below.

[0230] (1) A controller for an in-cabin monitoring system for a vehicle, the in-cabin monitoring system comprising a controller, a time-of-flight device configured to perform time-of-flight measurements to obtain time-of-flight data, and a radar device configured to perform radar measurements to obtain radar data, wherein the controller comprises a circuit configured to:

[0231] Obtain flight time data and radar data;

[0232] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data;

[0233] detecting an area of ​​interest in the cabin based on the time-of-flight data; and

[0234] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0235] (2) The controller of (1), wherein adjusting the operating mode includes adjusting a data processing mode of the controller and / or adjusting a data acquisition mode of the time-of-flight device and / or the radar device.

[0236] (3) A controller according to (2), wherein the circuit is configured to extract information about the presence of a passenger in the cabin and / or the passenger's behavior based on the radar data using the detected region of interest as prior knowledge.

[0237] (4) A controller according to (3), wherein the circuit is configured to detect at least one sub-region of interest in the region of interest based on the time-of-flight data using information extracted based on the radar data as prior knowledge.

[0238] (5) A controller according to (4), wherein the circuit is configured to extract information about the presence and / or behavior of passengers in the cabin in the sub-region of interest based on the time-of-flight data and the radar data.

[0239] (6) A controller according to any one of (2) to (5), wherein adjusting the data acquisition mode of the radar device includes: adjusting device parameters including at least one of frame rate, field of view, angular / range / velocity resolution, and power mode.

[0240] (7) A controller according to any one of (2) to (6), wherein adjusting the data acquisition mode of the time-of-flight device includes adjusting device parameters including at least one of accumulation time, frame rate, modulation frequency, illumination field, multi-wavelength light emission, and power mode.

[0241] (8) The controller according to any one of (1) to (7), wherein the circuit is configured as follows:

[0242] placing the radar device in a reduced power mode when a predetermined state of a vehicle is detected; and

[0243] Based on radar data acquired in a reduced power mode, the presence of an occupant in the cabin is detected.

[0244] (9) The controller of (8), wherein the circuit is configured to place the time-of-flight device in a normal power mode for detecting the region of interest in response to detecting the presence of a passenger in the cabin.

[0245] (10) The controller of (9), wherein the circuit is configured to place the radar device in a normal power mode in response to detecting the region of interest.

[0246] (11) A controller according to any one of (1) to (10), wherein performing object / passenger monitoring includes: detecting the presence of an object / passenger in the cabin, recording the detected object / passenger, tracking the detected object / passenger in the cabin, and identifying the behavior of the detected object / passenger.

[0247] (12) A control method for an in-cabin monitoring system for a vehicle, the in-cabin monitoring system comprising a controller, a time-of-flight device configured to perform time-of-flight measurements to obtain time-of-flight data, and a radar device configured to perform radar measurements to obtain radar data, wherein the control method comprises:

[0248] Obtain flight time data and radar data;

[0249] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data;

[0250] detecting an area of ​​interest in the cabin based on the time-of-flight data; and

[0251] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0252] (13) The control method according to (12), wherein adjusting the operation mode includes: adjusting a data processing mode of the controller and / or adjusting a data acquisition mode of the time of flight device and / or the radar device.

[0253] (14) The control method according to (13), comprising: extracting information about the presence of passengers in the cabin and / or the behavior of passengers using the detected region of interest as prior knowledge based on radar data.

[0254] (15) The control method according to (14), comprising: based on the flight time data, detecting at least one sub-region of interest in the region of interest using information extracted based on the radar data as prior knowledge.

[0255] (16) The control method according to (15), comprising: extracting information about the presence of passengers in the cabin and / or the behavior of passengers in the sub-area of ​​interest based on the flight time data and the radar data.

[0256] (17) A control method according to any one of (13) to (16), wherein adjusting the data acquisition mode of the radar device includes: adjusting device parameters including at least one of frame rate, field of view, angular / range / velocity resolution and power mode, and / or wherein adjusting the data acquisition mode of the time of flight device includes: adjusting device parameters including at least one of accumulation time, frame rate, modulation frequency, illumination field, multi-wavelength light emission and power mode.

[0257] (18) An in-cabin monitoring system for a vehicle, wherein the in-cabin monitoring system comprises:

[0258] a time-of-flight device configured to perform a time-of-flight measurement to obtain time-of-flight data;

[0259] a radar device configured to perform radar measurements to acquire radar data; and

[0260] A controller includes a circuit configured to:

[0261] Get time-of-flight data and radar data,

[0262] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data,

[0263] Detecting regions of interest in the cabin based on time-of-flight data, and

[0264] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0265] (19) A vehicle, wherein the vehicle includes an in-cabin monitoring system, the in-cabin monitoring system comprising:

[0266] a time-of-flight device configured to perform a time-of-flight measurement to obtain time-of-flight data;

[0267] a radar device configured to perform radar measurements to acquire radar data; and

[0268] A controller includes a circuit configured to:

[0269] Get time-of-flight data and radar data,

[0270] Perform object / passenger monitoring in the cabin based on time-of-flight data and radar data,

[0271] Detecting regions of interest in the cabin based on time-of-flight data, and

[0272] An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

[0273] (20) A vehicle according to (19), wherein the vehicle is an automobile, and wherein the time of flight device is mounted at a mid-top-top position and the radar device is mounted at a rearview mirror position.

[0274] (21) A computer program comprising a program code which, when executed on a computer, causes the computer to execute the method according to any one of (12) to (17).

[0275] (22) A non-transitory computer-readable recording medium having a computer program product stored therein, which, when executed by a processor, causes execution of the method according to any one of (12) to (17).

Claims

1. A controller for an in-cabin monitoring system of a vehicle, the in-cabin monitoring system comprising the controller, a time-of-flight device configured to perform time-of-flight measurements to obtain time-of-flight data, and a radar device configured to perform radar measurements to obtain radar data, the controller comprising a circuit configured to: Obtain flight time data and radar data; performing object / passenger monitoring in the cabin based on the time-of-flight data and the radar data; detecting an area of ​​interest in the cockpit based on the time-of-flight data; as well as An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

2. The controller according to claim 1, wherein: Adjusting the operating mode includes adjusting a data processing mode of the controller and / or adjusting a data acquisition mode of the time-of-flight device and / or the radar device.

3. The controller according to claim 2, wherein: The circuit is configured to extract information about the presence of a passenger in the cabin and / or a passenger's behavior based on the radar data using the detected region of interest as a priori knowledge.

4. The controller according to claim 3, wherein: The circuit is configured to detect at least one sub-region of interest in the region of interest based on the time-of-flight data using information extracted based on the radar data as prior knowledge.

5. The controller according to claim 4, wherein: The circuit is configured to extract information about the presence of passengers and / or behavior of passengers in the cabin in a sub-region of interest based on the time-of-flight data and the radar data.

6. The controller according to claim 2, wherein: Adjusting the data acquisition mode of the radar device includes adjusting device parameters including at least one of a frame rate, a field of view, an angular / range / velocity resolution, and a power mode.

7. The controller according to claim 2, wherein: Adjusting the data acquisition mode of the time-of-flight device includes adjusting device parameters including at least one of accumulation time, frame rate, modulation frequency, illumination field, multi-wavelength light emission, and power mode.

8. The controller according to claim 1, wherein: The circuit is configured as: placing the radar device in a reduced power mode when a predetermined state of the vehicle is detected; as well as Based on radar data acquired in the reduced power mode, the presence of an occupant in the cabin is detected.

9. The controller according to claim 8, wherein: The circuitry is configured to place the time-of-flight device in a normal power mode to detect a region of interest in response to detecting the presence of a passenger in the cabin.

10. The controller according to claim 9, wherein: The circuit is configured to place the radar device in a normal power mode in response to detecting the region of interest.

11. The controller according to claim 1, wherein: Performing object / passenger monitoring includes detecting the presence of an object / passenger in the cabin, registering the detected object / passenger, tracking the detected object / passenger in the cabin, and identifying the behavior of the detected object / passenger.

12. A control method for an in-cabin monitoring system of a vehicle, the in-cabin monitoring system comprising a controller, a time-of-flight device configured to perform time-of-flight measurements to obtain time-of-flight data, and a radar device configured to perform radar measurements to obtain radar data, the control method comprising: Obtain flight time data and radar data; performing object / passenger monitoring in the cabin based on the time-of-flight data and the radar data; detecting an area of ​​interest in the cockpit based on the time-of-flight data; as well as An operating mode of the in-cabin monitoring system is adjusted based on the detected region of interest.

13. The control method according to claim 12, wherein: Adjusting the operating mode includes adjusting a data processing mode of the controller and / or adjusting a data acquisition mode of the time-of-flight device and / or the radar device.

14. The control method according to claim 13, comprising: Based on the radar data, information about the presence of passengers in the cabin and / or the behavior of passengers is extracted using the detected regions of interest as prior knowledge.

15. The control method according to claim 14, comprising: Based on the time-of-flight data, at least one sub-region of interest in the region of interest is detected using information extracted based on the radar data as prior knowledge.

16. The control method according to claim 15, comprising: Based on the time-of-flight data and the radar data, information about the presence of passengers in the cabin and / or the behavior of passengers in a sub-region of interest is extracted.

17. The control method according to claim 13, wherein: Adjusting the data acquisition mode of the radar device includes: adjusting device parameters including at least one of frame rate, field of view, angular / range / velocity resolution and power mode, and / or wherein adjusting the data acquisition mode of the time of flight device includes: adjusting device parameters including at least one of accumulation time, frame rate, modulation frequency, illumination field, multi-wavelength light emission and power mode.

18. A cabin monitoring system for a vehicle, comprising: a time-of-flight device configured to perform a time-of-flight measurement to obtain time-of-flight data; a radar device configured to perform radar measurements to acquire radar data; as well as A controller comprising a circuit configured to: Get time-of-flight data and radar data, performing object / passenger monitoring in the cabin based on the time-of-flight data and the radar data, An area of ​​interest in the cabin is detected based on the time-of-flight data, and an operating mode of the in-cabin monitoring system is adjusted based on the detected area of ​​interest.

19. A vehicle, comprising an in-cabin monitoring system, the in-cabin monitoring system comprising: a time-of-flight device configured to perform a time-of-flight measurement to obtain time-of-flight data; a radar device configured to perform radar measurements to acquire radar data; as well as A controller comprising a circuit configured to: Get time-of-flight data and radar data, performing object / passenger monitoring in the cabin based on the time-of-flight data and the radar data, An area of ​​interest in the cabin is detected based on the time-of-flight data, and an operating mode of the in-cabin monitoring system is adjusted based on the detected area of ​​interest.

20. The vehicle according to claim 19, wherein: The vehicle is a car and wherein the time of flight device is mounted at an intermediate roof position and the radar device is mounted at a rear view mirror position.

Images