Method and apparatus for processing data associated with the interior of a vehicle

Abstract

Method, for example a computer-implemented method, for processing data associated with the interior of a vehicle, for example a motor vehicle, for example for controlling at least one function associated with the vehicle, comprising: providing data associated with the interior of the vehicle from at least one video sensor, for example an interior camera, providing data associated with the interior of the vehicle from at least one radar sensor, evaluating, for example jointly evaluating, the data of the at least one video sensor and the data of the at least one radar sensor.

Description

State of the art

[0001] The disclosure relates to a method for processing data associated with the interior of a vehicle.

[0002] The disclosure also relates to a device for processing data associated with the interior of a vehicle. Disclosure of the invention

[0003] Exemplary embodiments relate to a method, for example a computer-implemented method, for processing data associated with the interior of a vehicle, for example a motor vehicle, for example for controlling at least one function associated with the vehicle, comprising: providing data associated with the vehicle interior from at least one video sensor, for example an interior camera; providing data associated with the vehicle interior from at least one radar sensor; and evaluating, for example jointly evaluating, the data from the at least one video sensor and the data from the at least one radar sensor. In further exemplary embodiments, this allows for the efficient determination of information regarding the vehicle interior, e.g., about seat occupancy and / or the presence of, for example, persons in the vehicle interior.

[0004] In further exemplary embodiments, one or more functions associated with the vehicle can be controlled based on the evaluation, e.g. the joint evaluation, of the data from the at least one video sensor and the data from the at least one radar sensor.

[0005] In further exemplary embodiments, the functions associated with the vehicle have at least one of the following elements: a) safety-relevant function, b) comfort function.

[0006] In further exemplary embodiments, safety-relevant functions are associated with at least one of the following aspects: a) detection of whether a natural person is sitting in the driver's seat (or, for example, "only" a dummy, e.g., a mannequin), and, optionally, locking at least one vehicle function (e.g., drive system) or controlling the vehicle to achieve a safe state (e.g., stopping), based on this detection; b) detection of a possible medical emergency, e.g., concerning the driver (e.g., by evaluating a respiratory rate and / or a head position or pose and / or a body pose, etc.); c) making an emergency call, e.g., in the event of a possible medical emergency, e.g., concerning the driver; d) detection of a person and / or another living being inside the vehicle, e.g., left behind in a locked vehicle, and optionally, signaling, e.g., acoustic signaling (e.g.,Activating a horn) and / or visual signaling (e.g., activating the hazard warning lights) (and / or sending a message to the driver's mobile device) and / or partially opening at least one window of the vehicle and / or activating an air conditioning system, e) detecting a break-in or attempted break-in and controlling at least one vehicle system to signal the break-in or attempted break-in.

[0007] In further exemplary embodiments, comfort functions are associated with at least one of the following aspects: a) performing functions for a video conference, b) seat occupancy detection and control of multimedia or audio or video systems and / or an air conditioning system based on seat occupancy detection.

[0008] In other exemplary embodiments, the at least one video sensor is, for example, part of a camera system, e.g. for driver monitoring, e.g. for detecting drowsiness and / or distraction.

[0009] In further exemplary embodiments, the at least one video sensor can also be used for convenience functions such as video conferencing. In further exemplary embodiments, the at least one video sensor is designed, for example, as a Driver Monitoring Camera (DMC) or as an Occupant Monitoring Camera (OMC), or as a combination thereof, for example as an xMC, which, in further exemplary embodiments, covers, for example, an entire vehicle interior.

[0010] In other exemplary embodiments, the at least one radar sensor is, for example, an interior radar sensor (e.g., cabin-sensing radar) that operates, for example, in a 60 GHz frequency band.

[0011] In other exemplary embodiments, the at least one radar sensor is, for example, a UWB (Ultra Wide Band) type radar sensor.

[0012] In further exemplary embodiments, the at least one radar sensor can be used, for example, for "Child Left Behind" detection, i.e., to detect whether a child has been left in the vehicle interior. One possible measure of such a "Child Left Behind" function is that, after the driver has exited and locked the vehicle, the presence of a child left inside is detected, in order to then transmit a warning to the driver, for example, via a smartphone or, if necessary, vehicle warning signals such as the horn or hazard warning lights. Furthermore, in other exemplary embodiments, the air conditioning can be controlled and / or windows can be partially opened.

[0013] Some conventional approaches, such as systems with purely image-based detection and / or recognition, are subject to physical limitations, e.g., camera obstruction, object obstruction (e.g., newspaper, hand, hat, blanket on child seat), or so-called faking as an attempt to deceive, e.g., for driver presence, e.g., to be able to activate a semi-automated driving function, e.g., "Highway Pilot".

[0014] The at least one radar sensor according to exemplary embodiments emits, for example, at least intermittently, an electromagnetic signal, e.g., in the form of a wave. This enables object localization and / or speed determination, e.g., based on signal components reflected from objects, e.g., by time-of-flight measurement and / or evaluation of the Doppler effect. The occlusions or faking attacks described above with regard to purely image-based systems have no or no significant influence on the evaluation of the radar signals. Similarly, in further exemplary embodiments, a camera represents, for example, eyes and a radar sensor, for example, ears, e.g., of a person, e.g., to perceive the vehicle interior. Through sensor data fusion of data from sensors of both types (video and radar) according to exemplary embodiments, further exemplary embodiments can achieve, for example, improved performance and / or availability.for safety-relevant functions and / or for comfort functions for the occupants, for example, significantly increased.

[0015] In further exemplary embodiments, the method includes at least one of the following elements: a) combining the data of at least one video sensor with the data of at least one radar sensor, b) combining data derived from the data of at least one video sensor with data derived from the data of at least one radar sensor.

[0016] In further exemplary embodiments, it is also possible to combine data from at least one video sensor with data derived from the data of at least one radar sensor.

[0017] In further exemplary embodiments, it is also possible to combine data from at least one radar sensor with data derived from the data of at least one video sensor.

[0018] In further exemplary embodiments, the method includes at least one of the following elements: a) fusing the data of the at least one video sensor or data derived therefrom with the data of the at least one radar sensor or data derived therefrom, b) performing an early fusion (e.g., related to the acquisition of the data by means of the respective sensors), for example, early fusion, of the data of the at least one video sensor or data derived therefrom with the data of the at least one radar sensor or data derived therefrom, c) performing a late fusion (e.g., related to the acquisition of the data by means of the respective sensors), for example, late fusion, of the data of the at least one video sensor or data derived therefrom with the data of the at least one radar sensor or data derived therefrom.

[0019] In further exemplary embodiments, the method includes at least one of the following elements: a) capturing at least a part of the vehicle interior with the at least one video sensor, wherein, for example, the at least one video sensor is arranged in the area of ​​an interior mirror, b) providing one or more video data streams, wherein, for example, the multiple video data streams are each associated with b1) a different context and / or with b2) a different region of the vehicle interior and / or with b3) a different color channel.

[0020] In further exemplary embodiments, the method includes at least one of the following elements: a) using multiple radar sensors, wherein, for example, at least one of the multiple radar sensors is arranged in a roof module of the vehicle or in the area of ​​an interior mirror of the vehicle, b) cascading the multiple radar sensors, for example by means of a daisy-chain mechanism.

[0021] In further exemplary embodiments, the method includes: providing a temporal reference between the data of the at least one video sensor and the data of the at least one radar sensor, for example using timestamps, for example according to a Precision Time Protocol, PTP, and, optionally, evaluating at least a part of the data of the at least one video sensor and the data of the at least one radar sensor based on the temporal reference.

[0022] In further exemplary embodiments, the method includes: performing signal processing and / or at least one detection algorithm using at least one radar sensor, and, optionally, providing an evaluation result.

[0023] Further exemplary embodiments relate to a device for carrying out the method according to the embodiments. For example, in further exemplary embodiments, the device can be integrated into a control unit, e.g., for a vehicle.

[0024] Further exemplary embodiments relate to a vehicle, for example a motor vehicle, with at least one device according to the embodiments.

[0025] Other exemplary embodiments relate to a computer-readable storage medium comprising instructions which, when executed by a computer, cause it to execute the method according to the embodiments.

[0026] Other exemplary embodiments relate to a computer program comprising instructions which, when the program is executed by a computer, cause it to execute the method according to the embodiments.

[0027] Other exemplary embodiments relate to a data carrier signal that transmits and / or characterizes the computer program according to the embodiments.

[0028] Further exemplary embodiments relate to the use of the method according to the embodiments and / or the device according to the embodiments and / or the vehicle according to the embodiments and / or the computer-readable storage medium according to the embodiments and / or the computer program according to the embodiments and / or the data carrier signal according to the embodiments for at least one of the following elements: a) monitoring the vehicle interior, b) sensor data fusion, c) increasing performance and / or availability, for example for safety-related functions and / or for comfort functions, for example for vehicle occupants, d) improving detection performance and / or system availability, e.g.in the event of a temporary failure of the at least one video sensor and / or the at least one radar sensor, e) weighting the data of the at least one video sensor and the data of the at least one radar sensor.

[0029] Further features, applications, and advantages of the invention will become apparent from the following description of exemplary embodiments of the invention, which are illustrated in the figures of the drawing. All described or illustrated features, individually or in any combination, constitute the subject matter of the invention, irrespective of their compilation in the claims or their cross-reference, and irrespective of their formulation or representation in the description or in the drawing.

[0030] The drawing shows: Fig. 1. A simplified flowchart schematically shown according to exemplary embodiments. Fig. 2. A simplified block diagram schematically shown according to exemplary embodiments. Fig. 3. Schematically, a simplified flowchart according to exemplary embodiments, Fig. 4. Schematically, a simplified flowchart according to exemplary embodiments, Fig. 5. A simplified flowchart schematically illustrates exemplary embodiments. Fig. 6. Schematically, a simplified flowchart according to exemplary embodiments, Fig. 7. Schematically, a simplified flowchart according to exemplary embodiments, Fig. 8. Schematically, a simplified flowchart according to exemplary embodiments, Fig. 9. A simplified block diagram schematically shows exemplary embodiments. Fig. 10. A simplified block diagram according to exemplary embodiments, Fig. 11. A simplified block diagram schematically according to exemplary embodiments, Fig. 12 schematic aspects of uses according to exemplary embodiments.

[0031] Exemplary embodiments, Fig. 1, Fig. 2, refer to a method, for example a computer-implemented method, for processing data DAT-IR associated with a vehicle interior 10-IR of a vehicle, for example motor vehicle, comprising: providing 100 ( Fig. 1) Providing data DAT-IR-VID associated with the vehicle interior 10-IR from at least one video sensor 1, 1a, 1b, ..., for example, an interior camera; providing data DAT-IR-RAD associated with the vehicle interior 10-IR from at least one radar sensor 2, 2a, 2b, ...; evaluating data DAT-IR-RAD associated with the vehicle interior 10-IR from at least one radar sensor 2, 2a, 2b, ..., for example, jointly evaluating the data DAT-IR-VID of the at least one video sensor 1, 1a, 1b, ... and the data DAT-IR-RAD of the at least one radar sensor 2, 2a, 2b, .... In further exemplary embodiments, this allows for the efficient determination of information regarding the vehicle interior 10-IR, e.g., about seat occupancy and / or the presence of persons in the vehicle interior 10-IR.

[0032] In further exemplary embodiments, based on the evaluation 104, e.g. the joint evaluation 104a, of the data of the at least one video sensor and the data of the at least one radar sensor, one or more functions associated with the vehicle 10 can be controlled, see the optional block 106 according to Fig. 1.

[0033] In further exemplary embodiments, the functions associated with the vehicle 10 have at least one of the following elements: a) safety-relevant function, b) comfort function.

[0034] In further exemplary embodiments, safety-relevant functions are associated with at least one of the following aspects: a) detection of whether a natural person is sitting in the driver's seat (or, for example, "only" a dummy, e.g., a mannequin), and, optionally, locking at least one vehicle function (e.g., drive system) or controlling the vehicle 10 to achieve a safe state (e.g., stopping), based on the detection; b) detection of a possible medical emergency, e.g., concerning the driver (e.g., by evaluating a respiratory rate and / or a head position or pose and / or a body pose, etc.); c) making an emergency call, e.g., in the event of a possible medical emergency, e.g., concerning the driver; d) detection of a person and / or another living being located in the vehicle 10, e.g., left behind in a locked vehicle, and optionally, signaling, e.g., acoustic signaling (e.g.,Activating a horn) and / or visual signaling (e.g., activating the hazard warning lights) (and / or sending a message to the driver's mobile device) and / or partially opening at least one window of the vehicle and / or activating an air conditioning system, e) detecting a break-in or attempted break-in and controlling at least one vehicle system to signal the break-in or attempted break-in.

[0035] In further exemplary embodiments, comfort functions are associated with at least one of the following aspects: a) performing functions for a video conference, b) seat occupancy detection and control of multimedia or audio or video systems and / or an air conditioning system based on seat occupancy detection.

[0036] In further exemplary embodiments, Fig. 2, is at least one video sensor 1, 1a, 1b, ... e.g. part of a camera system, e.g. for driver monitoring, e.g. for detecting drowsiness and / or distraction.

[0037] In further exemplary embodiments, Fig. 2, the at least one video sensor 1, 1a, 1b, ... can also be used for convenience functions such as video conferencing. In further exemplary embodiments, the at least one video sensor 1, 1a, 1b, ... is configured, for example, as a Driver Monitoring Camera (DMC) or as an Occupant Monitoring Camera (OMC), or as a combination thereof, for example as an xMC, which, in further exemplary embodiments, covers, for example, the entire vehicle interior 10-IR.

[0038] In further exemplary embodiments, Fig. 2, is at least one radar sensor 2, 2a, 2b, ... e.g. an interior radar sensor (e.g. Cabin-Sensing Radar) that operates in a 60 GHz frequency band.

[0039] In further exemplary embodiments, the at least one radar sensor 2, 2a, 2b, ... is, for example, a radar sensor of the UWB (Ultra Wide Band) type.

[0040] In further exemplary embodiments, Fig. 2, at least one radar sensor 2, 2a, 2b, ... is applicable, for example, for "Child Left Behind" detection, i.e., detecting whether, for example, a child has been left in the vehicle interior 10-IR. An example of a possible measure for such a "Child Left Behind" function is that, after the driver has left and locked the vehicle 10, the presence of a child left in the vehicle interior 10-IR is detected, in order to then transmit a warning to the driver, for example, via a smartphone or, if necessary, vehicle warning signals such as the horn or hazard warning lights. Furthermore, in other exemplary embodiments, the air conditioning can be controlled and / or windows can be partially opened.

[0041] Some conventional approaches, such as systems with purely image-based detection and / or recognition, are subject to physical limitations, e.g., camera obstruction, object obstruction (e.g., newspaper, hand, hat, blanket on child seat), or so-called faking as an attempt to deceive, e.g., for driver presence, e.g., to be able to activate a semi-automated driving function, e.g., "Highway Pilot".

[0042] The at least one radar sensor 2, 2a, 2b, ... according to exemplary embodiments emits, for example, at least intermittently, an electromagnetic signal, e.g., in the form of a wave, which, based on signal components reflected from objects, e.g., by time-of-flight measurement and / or evaluation of the Doppler effect, enables object localization and / or determination of a speed, wherein the occlusions or faking attacks described above with regard to conventional purely image-based systems have no or no significant influence on the evaluation of the radar signals. Similarly, in further exemplary embodiments, a camera represents, for example, eyes and a radar sensor represents, for example, ears, e.g., of a person, e.g., to perceive the vehicle interior. Through sensor data fusion of data from sensors of both types (video and radar) according to exemplary embodiments, in further exemplary embodiments, e.g.,Performance and / or availability, e.g. for safety-relevant functions and / or for comfort functions for the occupants, can be significantly increased.

[0043] In Fig. Reference numeral EB-1 symbolizes, by way of example, a detection range of a video sensor 1, and reference numeral EB-2 symbolizes, by way of example, a detection range of a radar sensor 2. Further optional video sensors 1a, 1b, ... according to further exemplary embodiments are described in Fig. 2 is symbolized by a respective dashed rectangle. Further optional radar sensors 2a, 2b, ... according to further exemplary embodiments are shown in Fig. 2 is symbolized by a respective dashed rectangle.

[0044] In further exemplary embodiments, Fig. 3, it is provided that the method comprises at least one of the following elements: a) Combining 110 the DAT-IR-VID data of at least one video sensor 1, 1a, 1b, ... with the DAT-IR-RAD data of at least one radar sensor 2, 2a, 2b, ..., b) Combining 112 DAT-IR-VID' data derived from the DAT-IR-VID data of at least one video sensor 1, 1a, 1b, ... with DAT-IR-RAD' data derived from the DAT-IR-RAD data of at least one radar sensor 2, 2a, 2b, ....

[0045] In further exemplary embodiments, it is also possible to combine data DAT-IR-VID of at least one video sensor 1, 1a, 1b, ... with data DAT-IR-RAD' derived from the data DAT-IR-RAD of at least one radar sensor 2, 2a, 2b, ... .

[0046] In further exemplary embodiments, it is also possible to combine data DAT-IR-RAD of at least one radar sensor 2, 2a, 2b, ... with data DAT-IR-VID' derived from the data DAT-IR-VID of at least one video sensor 1, 1a, 1b, ... .

[0047] In further exemplary embodiments, Fig. 4, it is provided that the method comprises at least one of the following elements: a) fusing 120 the data DAT-IR-VID of the at least one video sensor 1, 1a, 1b, ... or derived data DAT-IR-VID' with the data DAT-IR-RAD of the at least one radar sensor 2, 2a, 2b, ... or derived data DAT-IR-RAD', b) performing 122 an early fusion (e.g., related to obtaining the data by means of the respective sensors), for example, early fusion, of the data DAT-IR-VID of the at least one video sensor 1, 1a, 1b, ... or derived data DAT-IR-VID' with the data DAT-IR-RAD of the at least one radar sensor 2, 2a, 2b, ... or derived data DAT-IR-RAD', c) performing 124 a late fusion (e.g., related to obtaining the data by means of of the respective sensors) fusion, for example late fusion, of the data DAT-IR-VID of at least one video sensor 1, 1a, 1b, ... or ...derived data DAT-IR-VID' with the data DAT-IR-RAD of at least one radar sensor 2, 2a, 2b, ... or derived data DAT-IR-RAD'.

[0048] In further exemplary embodiments, Fig. 5, it is provided that the method includes at least one of the following elements: a) Capturing 130 at least a part 10-IR' of the vehicle interior 10-IR ( Fig. 2) with the at least one video sensor 1, 1a, 1b, ..., wherein, for example, the at least one video sensor 1 ( Fig. 2) is located in the area of ​​an interior mirror, b) provide 132 ( Fig. 5) one or more video data streams VDS, wherein, for example, the multiple video data streams VDS are each associated with b1) a different context and / or with b2) a different region of the vehicle interior 10-IR and / or with b3) a different color channel.

[0049] In further exemplary embodiments, Fig. 6, it is provided that the method includes at least one of the following elements: a) Using 140 multiple radar sensors 2, 2a, 2b, ..., wherein, for example, at least one radar sensor 2 ( Fig. 2) which is arranged in a roof module of the vehicle 10 or in the area of ​​an interior mirror of the vehicle 10, b) Cascading 142 ( Fig. 6) of the multiple radar sensors 2, 2a, 2b, ..., for example by means of a daisy-chain mechanism.

[0050] In further exemplary embodiments, Fig. 7, the method is provided to: provide 150 a temporal reference ZB between the data DAT-IR-VID of the at least one video sensor 1, 1a, 1b, ... and the data DAT-IR-RAD of the at least one radar sensor 2, 2a, 2b, ..., for example using timestamps ZS, for example according to a Precision Time Protocol, PTP, and, optionally, evaluate 152 at least a part of the data DAT-IR-VID of the at least one video sensor 1, 1a, 1b, ... and the data DAT-IR-RAD of the at least one radar sensor 2, 2a, 2b, ... based on the temporal reference ZB.

[0051] In further exemplary embodiments, Fig. 8, is provided that the procedure includes: executing 160 a signal processing SV and / or at least one detection algorithm ALG-DET using the at least one radar sensor 2, 2a, 2b, ..., and, optionally, providing 162 an evaluation result ERG-AUSWERT.

[0052] Further exemplary embodiments, Fig. References 9 to a device 200 for carrying out the method according to the embodiments. For example, in further exemplary embodiments, the device 200 can be integrated into a control unit, e.g. for a vehicle 10, or forms such a control unit.

[0053] In further exemplary embodiments, Fig. 9, it is provided that the device 200 comprises: a computing device (“computer”) 202 having at least one computing core, a storage device 204 associated with the computing device 202 for at least temporary storage of at least one of the following elements: a) data DAT (e.g. the data DAT-IR-VID and / or DAT-IR-RAD and / or derivable or derived data DAT-IR-VID’ and / or DAT-IR-RAD’, and / or data obtainable during, e.g., joint, evaluation 104, 104a, for example fusion), b) computer program PRG, for example for carrying out the method according to the embodiments.

[0054] In further exemplary embodiments, the storage device 204 includes volatile memory (e.g., main memory (RAM)) 204a, and / or non-volatile (NVM) memory (e.g., flash EEPROM) 204b, or a combination thereof or with other memory types not explicitly mentioned.

[0055] Other exemplary embodiments relate to a computer-readable storage medium SM, comprising instructions PRG' which, when executed by a computer 202, cause it to execute the method according to the embodiments.

[0056] Further exemplary embodiments relate to a computer program PRG, PRG', comprising instructions which, when the program PRG, PRG' is executed by a computer 202, cause it to execute the method according to the embodiments.

[0057] Further exemplary embodiments relate to a data carrier signal DCS, which characterizes and / or transmits the computer program PRG, PRG' according to the embodiments. The data carrier signal DCS is, for example, exchangeable (transmittable and / or receiveable) via an optional data interface 206 of the device 200.

[0058] Further exemplary embodiments, Fig. 2, refer to a vehicle 10, for example a motor vehicle, with at least one device 200 according to the embodiments.

[0059] Fig. Figure 10 schematically shows a simplified block diagram according to exemplary embodiments. Element E1 symbolizes aspects of sensors, in this case, for example, comprising a video sensor, e.g., in the form of an interior camera E2, to which an optional lighting device E2a (e.g., operating in the infrared range) is assigned, and a radar sensor device E3 comprising two radar sensors E3a, E3b, which are connected, e.g., by means of a daisy-chain mechanism, for example, cascaded.

[0060] Element E10 symbolizes aspects of an evaluation of data from sensors E1 according to exemplary embodiments, for example, by a control unit. In other exemplary embodiments, element E10 can also symbolize a control unit. Element E11 symbolizes control logic. Element E12 symbolizes a video interface for receiving the DAT-IR-VID data from the interior camera E2; see also arrow a1, which symbolizes, for example, data transmission via an interface, e.g., of the LVDS type. Arrow a2 symbolizes optional control of the optional lighting device E2a, for example, for activating or deactivating the lighting device E2a or for setting an intensity. Arrow a3 symbolizes a data interface from the radar sensors E3a and E3b to block E10, for example, an interface E13 for receiving the DAT-IR-RAD data from the radar sensors E3a and E3b. The data interface a3 is, for example, of the CAN-FD or Ethernet type.

[0061] Element E14 symbolizes a control for the interior camera E2, element E15 symbolizes a video preprocessing of the data DAT-IR-VID, and element E16 symbolizes a further processing of the data DAT-IR-VID, for example for object recognition.

[0062] Element E17 symbolizes radar signal preprocessing for the DAT-IR-RAD data from block E13, and optionally, the timestamping of the processed data. Element E18 symbolizes data fusion for a fusion, for example, combined evaluation, of the video data DAT-IR-VID and the radar data DAT-IR-RAD, e.g., in the sense of late fusion, which, for example, characterizes data fusion at the object or symbol level with respect to the data DAT-IR-VID and DAT-IR-RAD. Optionally, in further exemplary embodiments, a time reference can be established between the fused data and, for example, other data or signals E19 of vehicle 10, for example, using the Precision Time Protocol (PTP).

[0063] Element E20 symbolizes a data record associated with a video data stream, e.g., video stream record. Element E21 symbolizes an output device, for example, output logic, for outputting data from at least one of the elements E16, E18, E20, for example, via at least one of the optional interfaces E22 (e.g., of type Ethernet and / or CAN and / or EDR (event data recorder)).

[0064] Fig. Figure 11 schematically shows a simplified block diagram according to exemplary embodiments. The blocks E1 to E16, E19, E20, E21 from Fig. 11 correspond at least approximately to blocks E1 to E16, E19, E20, E21 from Fig. 10.

[0065] Element E17a according to Fig. 11 symbolizes radar signal preprocessing with optional timestamps, and optionally the execution of at least one transformation, for example FFT (Fast Fourier Transform), and optionally clustering.

[0066] Element E17b according to Fig. 11 symbolizes aspects of radar signal object recognition (e.g., perception) or detection.

[0067] In further exemplary embodiments, at least parts of the functionality of blocks 17a and / or 17b can be implemented in the radar sensor device E3.

[0068] Further exemplary aspects and embodiments are described below, which in further exemplary embodiments can each be combined individually or in any combination with at least one of the aspects described above as examples.

[0069] In further exemplary embodiments, Fig. Block E10 includes a video sensor, e.g., in the form of an interior camera, E2, and several radar sensors E3a, E3b, wherein the radar sensors E3a, E3b can be operated, e.g., switched (e.g., alternating with each other) or cascaded. Block E10 can be implemented by means of a control unit, which, for example, is connected to the device 200 according to Fig. 9 similar or identical configurations.

[0070] In other exemplary embodiments, the interior camera E2 is located, for example, in the area of ​​an interior mirror of the vehicle 10 ( Fig. 2) It mounts and captures, for example, the entire interior 10-IR and delivers, for example, video data streams in different contexts, such as different image regions, such as so-called ROI, "Region of Interest", and / or one or more color channels - for example, in the case of an OMC - for example, in temporal sequence, for example, for optimal function-oriented image processing in suitable resolution and / or frame rate.

[0071] In other exemplary embodiments, the radar sensors E3a, E3b can be integrated, for example, in a vehicle roof module (not shown) or in the area of ​​the interior mirror (not shown).

[0072] In further exemplary embodiments, at least one further radar sensor E3b can be cascaded, e.g. by means of a daisy chain, e.g. to minimize the number of radar interfaces a3 to the control unit E10.

[0073] In further exemplary embodiments, the radar sensor E3a, which forms an interface to the control unit E10, has, depending on the design of the architecture, for example a CAN or Ethernet interface a3, the latter supporting, for example, a higher data bandwidth, which allows, for example, raw signals, for example for an Early Fusion design, to be transmitted to the control unit E10.

[0074] The following are exemplary aspects of further exemplary embodiments, for example with reference to a Late Fusion architecture.

[0075] As described above, Fig. 10 a video sensor E2, e.g. of the xMC type, and two cascaded radar sensors E3a, E3b, which are connected to the control unit E10 via corresponding video or radar interfaces a1, a3, for example CAN or Ethernet.

[0076] In further exemplary embodiments, a functional path for a video pipeline is represented by a Camera Drive Control E14, a Video Preprocessing E15, a Video Perception E16 and a Video Stream Record E20.

[0077] In further exemplary embodiments, a functional path for a radar pipeline is essentially formed by the radar preprocessing unit E17, which, for example, processes the received radar signals and, similarly to the video preprocessing unit E15, imprints a timestamp, for example, to enable a temporally correct assignment of the sensor data of both types (video, radar) in the downstream video / radar late fusion block E18. In further exemplary embodiments, the timestamp itself can, for example, be generated in the control unit E10 or, for example, be generated in another control unit (not shown) and provided, for example, via PTP.

[0078] Since, in further exemplary embodiments, both sensor types E2 and E3 do not necessarily transmit at the same data rate and are not necessarily synchronized with each other, a timestamp of the respective sensor data points, e.g., the radar signal at time t_r (~500 ms), is related to the video signal t_v (~33 ms) in the fusion block E18, e.g., by suitable temporal filtering of both signals, e.g., using a Kalman filter or by, e.g., feedback-controlled neural networks, e.g., to generate a fused value at time t_f (~100 ms). Subsequently, in further exemplary embodiments, a result can be output by the function block E21, e.g., in a predefined, e.g., customer-specific, data format to the desired output interface(s) E22, e.g., with a predefined timing t_out, e.g., via CAN-out or Event Data Recorder DER, e.g., in the event of a crash.

[0079] In other exemplary embodiments, the control logic E11 is superior and represents, for example, a control instance, e.g., to direct several, e.g., all, function blocks in a correct temporal and / or functional context and / or timing in input and / or output.

[0080] In other exemplary embodiments, e.g. in a late fusion, Fig. 11, signal processing E17a and / or detection algorithms E17b can run independently on the radar sensors E3a, E3b, and the measurement signals or characteristics such as movement and / or respiration and / or heartbeat on the respective assigned seats are then transmitted to the control unit E10 via the sensor interface a3, e.g. CAN or Ethernet.

[0081] In further exemplary embodiments, Fig. 10, a fusion E18 is performed at a so-called object level, i.e. e.g. based on localized features of video perception, e.g. through classical computer vision or neural networks or deep learning and the provided radar features.

[0082] Further exemplary embodiments include the following exemplary configurations and alternatives, e.g., for an Early Fusion architecture:

[0083] In further exemplary embodiments that use aspects of processing in the sense of early fusion, for example radar signal processing (e.g. radar preprocessing and / or radar perception) takes place completely or at least partially on the E10 control unit ( Fig. 10, Fig. 11) instead, for example by existing building blocks of type DSP (digital signal processor) or SoC (system on chip) or components 202, 204 of the device 200 according to Fig. 9, and the radar sensors E3a, E3b are, for example, designed to transmit raw signals, which characterize received radar signals, to the control unit E10 (see Fig. 11) This allows, in further exemplary embodiments, for example, fusion at the signal level using early fusion, enabling, for example, improved performance compared to some late fusion approaches. In further exemplary embodiments, machine learning (ML) methods can also be applied for early fusion signal processing, thereby generating, for example, a performance gain and allowing the use of cost-effective radar heads or radar sensors E3a, E3b, for example, with a smaller installation space. In further exemplary embodiments, which relate, for example, to early fusion at the signal level, it is possible, for example, by applying a 2D FFT of received radar signals, e.g. radar ramps (e.g. different frequency ramps associated with received radar signals), instead of one-dimensional output values ​​for a location / detected object position and an object velocity, to determine, for example, a two-dimensional representation for a distance-velocity profile of possible objects, i.e., a two-dimensional representation that characterizes a profile of distance and / or velocity of possible objects.

[0084] In further exemplary embodiments, a two-dimensional representation, which characterizes, for example, the course of distance and / or speed of possible objects, can be efficiently linked or combined with video sensor signals, for example by feature combination, for example using at least one classical approach and / or by means of at least one machine learning (ML) or deep learning approach.

[0085] In other exemplary embodiments, the radar performs preprocessing, e.g., block E17. Fig. 10 or block E17a from Fig. 11 e.g. a two-dimensional (2D) FFT of a received, e.g. “raw”, radar signal from one or more radar sensors E3a, E3b and, optionally, e.g. an evaluation in the sense of a constant false alarm rate (CFAR), e.g. to determine relevant reflection points or echoes from corresponding objects or occupants.

[0086] In further exemplary embodiments, radar preprocessing is followed, for example, by so-called radar perception, see block E17b according to Fig. 11.

[0087] In further exemplary embodiments, radar perception consists in a first step of estimating the distance and / or speed and / or angle of radar reflection points of objects or occupants, which are then, for example, clustered (i.e., divided into clusters or groups) and classified in a second step and assigned to a respective vehicle seat, for example, to output signals such as respiration and / or heartbeat and / or movement. Since these output signals are not additionally filtered in further exemplary embodiments, for example, within the radar sensor itself—as is the case with some late fusion approaches—based on application parameters that characterize, for example, time windows for averaging the output signals, further exemplary embodiments can, for example,An optimal combination and / or weighting of individual sensor signals of the considered types (video, radar, or signal combinations thereof) can be determined using a trained neural network or a Kalman filter.

[0088] The principle according to the embodiments can be used, for example, to detect or defend against so-called faking attacks, in which, for example, purely video-based facial recognition (e.g., of the "Face ID" type) of the driver is performed, and attempts are made to circumvent the video-based facial recognition using a photograph or (3D) dummy dolls. This is because, according to the embodiments, the principle allows for the fusion of data from at least one video sensor with data from at least one radar sensor, taking into account, for example, respiration and / or heartbeat and / or movement, which are not present in the case of a photograph or dummy doll.

[0089] Furthermore, in other exemplary embodiments, for example, a child in a covered child seat could be reliably detected using the principle according to the embodiments, for example, based on respiration detection, for example, based on a fusion of sensor data, whereby a detection performance depends, for example, on a weighting of the individual and combined sensor signals (video / radar).

[0090] In further exemplary embodiments, a correlation is taken into account between at least some of the following aspects: a) a specific respiration pattern, b) a heartbeat, c) emotions, d) well-being, e) stress, f) physical fitness, for example, fitness, e.g., for a driver, g) an emergency, e.g., a heart attack or the effects of medication or drugs. In further exemplary embodiments, these aspects can be detected, for example, by means of video-based recognition (e.g., to detect an abnormal head or body posture, e.g., sinking into the seat), which is combined with an evaluation of signals from at least one radar sensor.

[0091] Another possible application or use case, according to further exemplary embodiments, is, for example, bridging video sensor obstruction, e.g., in the case of an already activated automated driving mode, e.g., according to SAE Level 2 or higher, whereby the driver is allowed to perform permitted secondary activities such as reading a newspaper. With such video sensor obstruction, at least one radar sensor can still detect the driver's presence in the driver's seat, so that, compared to conventional, purely video-based monitoring, increased precision is achieved, and video sensor obstruction does not directly trigger an error response or a shutdown of the system.

[0092] In further exemplary embodiments, for example, for certain applications, at least one radar sensor can additionally act as a trigger for downstream video functions, e.g., to verify an event by means of video perception.

[0093] Further exemplary embodiments, Fig.12, refer to a use 300 of the method according to the embodiments and / or the device 200 according to the embodiments and / or the vehicle 10 according to the embodiments and / or the computer-readable storage medium SM according to the embodiments and / or the computer program PRG, PRG' according to the embodiments and / or the data carrier signal DCS according to the embodiments for at least one of the following elements: a) monitoring 301 of the vehicle interior, b) sensor data fusion 302, c) increasing 303 performance and / or availability, for example for safety-related functions and / or for comfort functions, for example for vehicle occupants, d) improving 304 detection performance and / or system availability, e.g.in the event of a temporary failure of the at least one video sensor and / or the at least one radar sensor, e) weighting 305 of the data of the at least one video sensor and the data of the at least one radar sensor. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] video stream record. Element E21 symbolizes an output device, for example output logic

[0063]

Claims

[1] Method, for example a computer-implemented method, for processing data (DAT-IR) associated with a vehicle interior (10-IR) of a vehicle, for example a motor vehicle (10), for example for controlling (106) at least one function associated with the vehicle, comprising: providing (100) data (DAT-IR-VID) associated with the vehicle interior (10-IR) of at least one video sensor (1, 1a, 1b, ...), for example an interior camera, providing (102) data (DAT-IR-RAD) associated with the vehicle interior (10-IR) of at least one radar sensor (2, 2a, 2b, ...), evaluating (104), for example jointly evaluating (104a), the data (DAT-IR-VID) of the at least one video sensor (1, 1a, 1b, ...) and the data (DAT-IR-RAD) of the at least one radar sensor (2, 2a, 2b, ...). [2] Method according to claim 1, comprising at least one of the following elements: a) combining (110) the data (DAT-IR-VID) of the at least one video sensor (1, 1a, 1b, ...) with the data (DAT-IR-RAD) of the at least one radar sensor (2, 2a, 2b, ...), b) combining (112) data (DAT-IR-VID') derived based on the data (DAT-IR-VID) of the at least one video sensor (1, 1a, 1b, ...) with data (DAT-IR-RAD') derived based on the data (DAT-IR-RAD) of the at least one radar sensor (2, 2a, 2b, ...). [3] Method according to at least one of the preceding claims, comprising at least one of the following elements: a) fusing (120) the data (DAT-IR-VID) of the at least one video sensor (1, 1a, 1b, ...) or data derived therefrom (DAT-IR-VID') with the data (DAT-IR-RAD) of the at least one radar sensor (2, 2a, 2b, ...) or data derived therefrom (DAT-IR-RAD'), b) carrying out (122) an early fusion, for example early fusion, of the data (DAT-IR-VID) of the at least one video sensor (1, 1a, 1b, ...) or data derived therefrom (DAT-IR-VID) with the data (DAT-IR-RAD) of the at least one radar sensor (2, 2a, 2b, ...) or data derived therefrom (DAT-IR-RAD'), c) carrying out (124) a late fusion, for example Late fusion, the data (DAT-IR-VID) of the at least one video sensor (1, 1a, 1b, ...) or data derived therefrom (DAT-IR-VID') with the data (DAT-IR-RAD) of the at least one radar sensor (2, 2a, 2b, ...) orderived data (DAT-IR-RAD'). [4] Method according to at least one of the preceding claims, comprising at least one of the following elements: a) detecting (130) at least a part (10-IR') of the vehicle interior (10-IR) with the at least one video sensor (1, 1a, 1b, ...), wherein, for example, the at least one video sensor (1, 1a, 1b, ...) is arranged in the region of an interior mirror, b) providing (132) one or more video data streams (VDS), wherein, for example, the plurality of video data streams (VDS) are each associated with b1) a different context and / or with b2) a different region of the vehicle interior (10-IR) and / or with b3) a different color channel. [5] Method according to at least one of the preceding claims, comprising at least one of the following elements: a) using (140) a plurality of radar sensors (2, 2a, 2b, ...), wherein, for example, at least one radar sensor (2) of the plurality of radar sensors (2, 2a, 2b, ...) is arranged in a roof module of the vehicle (10) or in the region of an interior mirror of the vehicle (10), b) cascading (142) the plurality of radar sensors (2, 2a, 2b, ...), for example by means of a daisy-chain mechanism. [6] Method according to at least one of the preceding claims, comprising: providing (150) a temporal reference (ZB) between the data (DAT-IR-VID) of the at least one video sensor (1, 1a, 1b, ...) and the data (DAT-IR-RAD) of the at least one radar sensor (2, 2a, 2b, ...), for example using (150a) time stamps (ZS), for example according to a Precision Time Protocol, PTP, and, optionally, evaluating (152) at least a portion of the data (DAT-IR-VID) of the at least one video sensor (1, 1a, 1b, ...) and the data (DAT-IR-RAD) of the at least one radar sensor (2, 2a, 2b, ...) based on the temporal reference (ZB). [7] Method according to at least one of the preceding claims, comprising: executing (160) a signal processing (SV) and / or at least one detection algorithm (ALG-DET) by means of the at least one radar sensor (2, 2a, 2b, ...), and, optionally, providing (162) an evaluation result (ERG-AUWERT). [8] Method according to at least one of the preceding claims, comprising: controlling (106) the at least one function associated with the vehicle (10), for example based on the evaluation (104; 104a), wherein, for example, the at least one function associated with the vehicle (10) comprises at least one of the following elements: a) safety-relevant function, b) comfort function. [9] Device (200) for carrying out the method according to at least one of the preceding claims. [10] Vehicle, for example motor vehicle, (10) with at least one device (200) according to claim 9. [11] Computer-readable storage medium (SM) comprising instructions (PRG) which, when executed by a computer (202), cause the computer to carry out the method according to at least one of claims 1 to 8. [12] Computer program (PRG) comprising instructions which, when the program (PRG) is executed by a computer (202), cause the computer (202) to carry out the method according to at least one of claims 1 to 8. [13] Data carrier signal (DCS) which transmits and / or characterises the computer program (PRG) according to claim 12. [14] Use (300) of the method according to at least one of claims 1 to 8 and / or the device (200) according to claim 9 and / or the vehicle (10) according to claim 10 and / or the computer-readable storage medium (SM) according to claim 11 and / or the computer program (PRG) according to claim 12 and / or the data carrier signal (DCS) according to claim 13 for at least one of the following elements: a) monitoring (301) of the vehicle interior (10-IR), b) sensor data fusion (302), c) increasing (303) a performance and / or an availability, for example for safety-relevant functions and / or for comfort functions, for example for occupants of the vehicle (10), d) improving (304) a detection performance and / or a system availability, e.g. in the event of a brief failure of the at least one video sensor (1, 1a, 1b, ...) and / or the at least one radar sensor (2, 2a, 2b, ...), e) weighting (305) the data (DAT-IR-VID) of the at least one video sensor (1, 1a, 1b, ...) and the data (DAT-IR-RAD) of at least one radar sensor (2, 2a, 2b, ...).

Images