System and procedure for monitoring a vehicle interior

Abstract

System (10) for monitoring a vehicle interior (12), the system (10) comprising at least one interface for receiving image data (14) acquired by an image acquisition device (xMC), and a first computing unit (ECU-1) for providing at least one first monitoring function (F-1) of the system based on the image data (14), as well as a further computing unit (ECU-2) for providing at least one further monitoring function (F-2) of the system (10) based on the image data (14), wherein the system (10) is configured such that the image data (14) is provided to the first computing unit (ECU-1) via the interface, and that at least a part (14') of the image data (14) is provided by the first computing unit (ECU-1) to the further computing unit (ECU-2), wherein the at least part (14') of the image data (14) comprises image data,which are required for the additional computing unit (ECU-2) to provide the additional monitoring function (F-2).

Description

State of the art

[0001] Driver monitoring systems (DMS) are well-known from the state of the art. These systems are typically single-, dual-, or multi-camera systems consisting of camera modules and active near-infrared illumination (IR modules).

[0002] Current developments also include interior camera systems that capture not only the driver but also the entire vehicle interior in order to be able to display additional vehicle safety-relevant functions, such as seat occupancy detection, as well as comfort-relevant functions. Disclosure of the invention

[0003] One embodiment relates to a system for monitoring a vehicle interior, the system comprising at least one interface for receiving image data captured by an image capture device, and a first computing device for providing at least one first monitoring function of the system based on the image data, as well as a further computing device for providing at least one further monitoring function of the system based on the image data, wherein the system is designed such that the image data are provided to the first computing device via the interface, and that at least a portion of the image data is provided by the first computing device to the further computing device, wherein the at least a portion of the image data comprises image data that is required for the further computing device to provide the further monitoring function.

[0004] The captured image data are, for example, image data streams, for example including video data streams or single image data streams.

[0005] The image data is received via an interface, i.e. provided to the first computing device via the interface.

[0006] The computing devices are, for example, domain or zone control units (ECUs). Domain control units are used in a domain-specific E / E architecture, which is structured according to functions or functional areas. In a zone-oriented E / E architecture, the functions are distributed across domains to zone control units depending on their arrangement in the vehicle. Both domain and zone ECUs must support a variety of sensors and / or actuators in order to provide desired functions, such as safety functions and comfort functions, from the sensor information.

[0007] In particular, the safety-relevant driver monitoring function for monitoring the driver's attention level, for example, drowsiness and / or distraction detection, requires very high computing power. It is therefore proposed that the safety-relevant driver monitoring function be provided by one computing device, for example, the first computing device or the additional computing device, and that further monitoring functions be implemented by the other computing device. It is provided that only the image data required to execute the function are transmitted from one computing device to the other computing device. The forwarding of the image data from the first computing device to the additional computing device is thus function-oriented.By forwarding the image data from the first computing device to the downstream additional computing device, additional functions can be realized that would not be possible due to the limited resources, such as interfaces, bandwidth, performance, of the control units considered alone.

[0008] According to one embodiment, the system may comprise an image capture device for capturing image data, in particular a camera, for capturing at least a part of the vehicle interior, wherein the image capture device is designed to provide the captured image data to the first computing device via the interface.

[0009] According to one embodiment, it is provided that the first monitoring function and / or a further monitoring function that can be provided with the first computing device and / or the or a further monitoring function that can be provided with the further computing device is one of the following functions: a) a driver monitoring function, in particular for monitoring the driver's state of attention, in particular by detecting a posture, a direction of gaze and / or a position of a head or a face orof the driver's eyes, an opening of the eyes, a blinking frequency of the driver's eyes, for example fatigue detection, distraction detection, detection of driver vital signs, gesture detection; b) an occupant monitoring function, for example detection of occupant vital signs, gesture detection, detection of activities in the vehicle interior, detection of situations in the vehicle interior, detection of the presence / absence of occupants in the vehicle interior, detection of the presence / absence of objects in the vehicle interior, c) video telephony function; d) facial recognition function; e) intruder detection in the vehicle interior.

[0010] The driver monitoring function(s) mentioned under a) are also referred to as DM(S) (Driver Monitoring (System) function(s). The occupant monitoring function(s) mentioned under b) are also referred to as OM(S) (Occupant Monitoring (System) function(s).

[0011] The first and / or the further control unit can also each provide several of the aforementioned monitoring functions.

[0012] In a domain-specific E / E architecture, the first computing device is, for example, an ADAS (Advanced Driver Assistance System) control unit. The first computing device provides, for example, driver monitoring functions such as detecting a gaze direction and / or a head position (also known as eye tracking or head tracking), and / or drowsiness detection, and / or distraction detection. Additionally or alternatively, the first computing device can also provide occupant monitoring function(s).

[0013] The additional computing device is, for example, an in-vehicle infotainment (IVI) control unit. The additional computing device provides monitoring functions such as video telephony, facial recognition, and / or intruder detection in the vehicle interior. Additionally or alternatively, the additional computing device can also provide occupant monitoring functions.

[0014] The described ECU cascading can be advantageous because an alternative architecture with an image acquisition device with image split functionality on both ECUs cannot be easily implemented due to limiting factors such as interfaces, bandwidth, performance or customer-specific interface specifications.

[0015] According to one embodiment, the system is configured such that the first computing device provides output data based on processed image data to the additional computing device. In addition to the forwarded image data, processed output data, for example, a detected face of the driver, can also be used as input data for functions of the additional computing device. This advantageously eliminates the need for redundant processing of image data whose processing is relevant to the function of the additional computing device but is already covered by processing of the image data by the first computing device.

[0016] According to one embodiment, the system comprises a control device for controlling the image capture device, and the image capture device is configured such that image data can be provided in at least two different contexts. The system is configured such that a respective context can be specified by means of the control device. The context is specified, for example, by a context-oriented configuration of an image sensor, also called an imager, of the image capture device by means of the control device.

[0017] Further embodiments relate to a method for operating a system for monitoring a vehicle interior according to the described embodiments, wherein the first computing device provides at least one first monitoring function of the system based on the image data, and the further computing device provides at least one further monitoring function of the system based on the image data, the procedure comprises the following steps: Providing image data via the interface to the first computing device, and providing at least a portion of the image data from the first computing device to the further computing device, wherein the at least a portion of the image data comprises image data required by the further computing device to provide the further monitoring function.

[0018] The image data is forwarded from the first computing device to the subsequent computing device, for example, via Gigabit Ethernet (GiE). The first computing device can first perform a format conversion of the image data, for example, to NV12 or YUV420, and an H.264 or H.265 encoding. The conversion and encoding configuration can be tailored to limiting factors such as interfaces, bandwidth, image quality requirements, data rate, and performance. The conversion enables the encoding of the image data by a hardware encoder using memory-to-memory operation. The conversion can be performed efficiently either using a corresponding processor or a hardware module. Based on the maximum pixel throughput of the hardware encoder, a maximum resolution and frame rate of the image data can be determined.If necessary, the permissible data throughput of the image data can be adjusted by reducing the resolution, either directly in the image sensor of the image capture device or downstream. A constant bit rate when providing the image data can be advantageous to ensure robust, quasi-real-time transmission. For encoding, it can be advantageous to use only I-frames and not B- or P-frames.

[0019] According to one embodiment, the method comprises: capturing image data by means of the image capturing device, in particular a camera, for capturing at least part of the vehicle interior.

[0020] The image capture device may include an image processor, also called an ISP chip, so that color processing of the captured image data, such as noise reduction, white balance, color temperature correction, etc., can be performed in the image capture device. Alternatively or additionally, color processing can also be performed by appropriate means in the first and / or the further computing device. In this case, the ISP chip in the image capture device can be omitted.

[0021] According to one embodiment, the method comprises: providing output data based on processed image data to the further computing device by means of the first computing device.

[0022] According to one embodiment, the method comprises: specifying a context for providing the image data by means of a control device for controlling the image capture device, and providing the captured image data by means of the image capture device according to the specified context. Advantageously, the image capture device provides image data in different contexts, for example, in a specified chronological order and / or for a specified time period, by appropriate control.

[0023] According to one embodiment, it is provided that the first monitoring function of the first computing device and the at least one further monitoring function of the further computing device are carried out at least temporarily simultaneously, and the image capture device, depending on the first monitoring function, at least temporarily provides image data in a first context, wherein in the first context the captured image data are output according to a first output format, and the image capture device, depending on the further monitoring function, at least temporarily provides image data in at least one further context, wherein in the further context the captured image data are output according to a further output format.

[0024] A respective output format includes, for example, a resolution and / or a frame rate and / or, if applicable, an image range. The respective output format is adapted to a respective monitoring function in a function-oriented manner. The output format of one context may differ from the output format of another context in terms of resolution and / or frame rate and / or image range.

[0025] According to one embodiment, switching between the first context and the at least one further context occurs according to a predeterminable pattern depending on the monitoring functions to be performed. The predeterminable pattern includes, for example, the order in which image data is provided in a respective context and / or a duration for which image data is provided in a respective context.

[0026] According to one embodiment, at least one encryption and / or authentication method is provided to secure the communication between the first computing device and the further computing device. For example, to protect the transmission of image data, in particular facial data, a TLS handshake method can be provided. TLS is an encryption and authentication protocol. A TLS handshake is the process that starts a communication session with TLS encryption. During a TLS handshake, the two communicating parties exchange messages to confirm each other, verify each other, specify the cryptographic algorithms they use, and agree on session keys. The image data is only subsequently transmitted after the TLS handshake has been successfully completed.An alternative or complementary encryption method is scrambling the image data, particularly by adding randomly generated image segments. Alternatively, standard methods such as AES128 or AES256 or the CSA Common Scrambling Algorithm can also be used.

[0027] A further security measure provides for image data to be transmitted anonymously, for example by obscuring it with censorship bars or similar.

[0028] Further advantages will become apparent from the description and the accompanying drawings. Exemplary embodiments of the invention are illustrated in the drawings and explained in more detail in the following description. Identical reference numerals in different figures designate identical or at least functionally comparable elements. When describing individual figures, reference may also be made to elements from other figures. These show, in schematic form: Fig. 1 shows a system for monitoring a vehicle interior according to a first embodiment; Fig. 2 a vehicle interior; Fig. 3 a system for monitoring a vehicle interior according to another embodiment; Fig. 4 shows a system for monitoring a vehicle interior according to another embodiment; Fig. 5 shows a system for monitoring a vehicle interior according to another embodiment; Fig. 6 a system for monitoring a vehicle interior according to another embodiment; Fig. 7 and Fig. 8 different embodiments of an encryption method.

[0029] Fig. 1 shows a system 10 for monitoring a vehicle interior. An exemplary vehicle interior 12 is shown in Fig. 2. The vehicle interior is the interior of a motor vehicle. The motor vehicle can also be a semi-autonomous vehicle. By way of example, a driver 16 of the vehicle is shown in the vehicle interior 12. Additional occupants (not shown) of the vehicle can be provided on a passenger seat 18 and / or on the rear seat 20 of the vehicle.

[0030] In the example, the system 10 comprises an image capture device xMC, in particular a camera, for capturing at least a part of the vehicle interior 12 in the form of image data 14. The image data 14 are, for example, image data streams, for example comprising video data streams or single image data streams.

[0031] In the example, the image capture device xMC comprises an image sensor IM, also called an imager, an image processor ISP, also called an ISP chip, an infrared module IR and a serializer SER for the serial transmission of image data 14.

[0032] The system 10 comprises a first computing device ECU-1 for providing at least one first monitoring function F-1 of the system based on image data 14 acquired with the image acquisition device xMC.

[0033] The system 10 comprises, for example, a further computing device ECU-2 for providing at least one further monitoring function F-2 of the system based on the image data 14 acquired by the image acquisition device xMC.

[0034] According to the example, the image data 14 acquired with the image acquisition device xMC are provided by the image acquisition device xMC to the first computing device ECU-1 via a suitable interface, and at least a portion 14' of the acquired image data 14 is provided by the first computing device ECU-1 to the further computing device ECU-2, wherein the at least a portion 14' of the acquired image data 14 comprises image data required by the further computing device ECU-2 to provide the further monitoring function F-2.

[0035] The system may also include an additional computing device ECU-x for providing additional monitoring functions Fx. The forwarding of image data may be performed by the first computing device ECU-1 and / or the additional computing device ECU-2.

[0036] According to the example shown, the image capture device xMC is part of system 10. This is not mandatory. The system can also include the computing devices ECU-1, ECU-2, and possibly additional computing devices, with image data being provided to the first computing device ECU-1 via a suitable interface. The image capture device xMC is then not necessarily part of system 10.

[0037] The first computing unit ECU-1 is, for example, an ADAS control unit (Advanced Driver Assistance Systems). The first computing unit ECU-1 provides, for example, driver monitoring functions F-1a, such as detecting a gaze direction and / or a head position, also called eye tracking or head tracking, and / or drowsiness detection, and / or distraction detection. As shown in Fig. 3, the first computing device ECU-1 also provides occupant monitoring function(s) F1-b.

[0038] The additional computing unit ECU-2 is, for example, an in-vehicle infotainment (IVI) control unit. The additional computing unit ECU-2 provides, for example, monitoring functions F-2 such as video telephony function F-2a, facial recognition function F-2b, and / or intruder detection F-2c in the vehicle interior. As shown in Fig. 4, the further computing unit ECU-2 also provides occupant monitoring function(s) F2-d.

[0039] According to one embodiment, the system 10 is configured such that the first computing device ECU-1 provides output data based on processed image data 14 to the further computing device. In addition to the forwarded image data 14', processed output data 22, for example a detected face of the driver, can also be used as input data for functions of the further computing device ECU-2. This advantageously eliminates the need for redundant processing of image data 14', the processing of which is relevant for the function of the further computing device ECU-2 but is already covered by processing of the image data 14 by the first computing device ECU-1.

[0040] According to one embodiment, the system 10 comprises a control device 24 for controlling the image capture device xMC, and the image capture device xMC is designed such that image data 14 can be provided in different contexts A, B, C, D, wherein the system 10 is designed such that a respective context A, B, C, D can be specified by means of the control device 24. The specification of the context takes place, for example, by a context-oriented configuration of the image sensor IM of the image capture device xMC by means of the control device 24. According to the example, cf. Fig. 6, the control device is implemented in the first computing device ECU-1.

[0041] Based on the Fig. 3 to 5 explain the context-oriented configuration of the image sensor IM and the provision of image data in different contexts A, B, C, D.

[0042] The image sensor can, for example, provide a maximum resolution of 5.1 Mpix (2560x1980) with a frame rate of 60 Hz. The image sensor can advantageously be a CFA (Color Filter Array) image sensor. The image sensor can, for example, be configured according to four different output formats: 1. RGBIR 4x4 (raw), 2. IR upscaled, 3. IR ¼ of full resolution, 4. BGGR 2x2 Bayer (IR substracted). The latter is an on-sensor RGBIR processing in which the RGBIR 4x4 Bayer matrix is ​​converted to a BGGR 2x2 Bayer matrix, and IR pixels replace neighboring RGB color pixels through interpolation. Alternatively, this processing does not have to be performed internally within the sensor, but can also be performed on the ISP chip.

[0043] To provide driver monitoring functions F-1a, the first computing device ECU-1 requires, for example, image data in context A. Image data in context A in the example refer to a driver-related image section drvROl, for example the driver's head, with a resolution of, for example, 1.6 Mpix and a frame rate of, for example, 60 Hz, for example monochrome IR image data.

[0044] According to an exemplary application, it is intended that a video call F-2a is to be conducted simultaneously using the additional computing device ECU-2, at least for a certain period of time. For the F-2a video telephony function, the additional computing device ECU-2 requires, for example, image data in context B. In the example, image data in context B refers to a color video data stream in BGGR with a resolution of, for example, 3.7 Mpix and a frame rate of, for example, 24 Hz.

[0045] Switching between context A and context B can occur according to a predeterminable pattern, depending on the monitoring functions to be performed. For example, the image data in context A and context B can be provided by the image acquisition device xMC according to a time-division multiplexing method, for example, according to a pattern AABAAB or according to a pattern ABAB.

[0046] If an additional occupant monitoring function F-1b or F-2d is to be provided by the first computing device ECU-1 or by the further computing device ECU-2, image data in context C, for example, is required. In the example, image data in context C refers to image data with reduced VGA resolution and a frame rate of, for example, 10 Hz. An image section occROI can be directed at the front passenger or at passengers in the rear seat, depending on the seat occupancy. For example, the image data in context A, in context B, and in context can be provided by the image acquisition device xMC according to a time-division multiplexing method, for example, according to a pattern AABAA C.

[0047] If additional facial recognition F-2b is to be provided by the additional computing device ECU-2, image data in context D, for example, is required. Image data in context D in the example refers to image data with a focused image area on the passenger's face, for example, as a single image or as a video data stream with a low frame rate, e.g., a 1-s cycle. Focusing on the passenger's face can be achieved either by statically defining the facial area or by dynamically detecting the facial area, for example, using skeleton detection and thus recognizing the face's position in the image.

[0048] In the example in Fig. 5, image data in context B, C are forwarded from the first computing device ECU-1 to the further computing device ECU-2. The forwarding occurs, for example, by transmission via Gigabit Ethernet, GiE. The first computing device ECU-1 can previously perform a format conversion 26 of the image data, for example, to NV12 or YUV420, and an H.264 or H.265 encoding 28. The configuration of the conversion 26 and the encoding 28 can be tailored to limiting factors such as interfaces, bandwidth, image quality requirements, data rate, and performance. The conversion 26 enables the encoding 28 of the image data to be performed by a hardware encoder using a memory-to-memory operation. The conversion 26 can be performed efficiently either by means of a corresponding processor or a hardware module.Based on the maximum pixel throughput of the hardware encoder, a maximum resolution and frame rate of the image data can be determined. If necessary, the permissible data throughput of the image data can be adjusted by reducing the resolution, either directly in the image sensor of the image capture device or downstream. A constant bit rate when providing the image data can be advantageous to ensure robust, quasi-real-time transmission.

[0049] According to the example, the first computing device ECU-1 and / or the further computing device ECU-2 can receive data from further sensors 30 that may be relevant for providing the respective functions.

[0050] In Fig. 6 shows a further illustration of the system 10. The first computing device ECU-1 and the further computing device ECU-2 include a GiE transceiver 32 to provide secure point-to-point Ethernet communication.

[0051] In addition to the components already described in the example, the first computing device includes a driver 34 for the infrared module IR and a deserializer 36.

[0052] Various measures can be provided to secure communication between the first computing device ECU-1 and the further computing device ECU-2.

[0053] For example, to protect the transmission of image data, especially facial data, a TLS handshake procedure may be implemented. TLS is an encryption and authentication protocol. A TLS handshake is the process that initiates a communication session with TLS encryption. During a TLS handshake, the two communicating parties exchange messages to confirm and verify each other, specify the cryptographic algorithms they use, and agree on session keys. Only after the TLS handshake has been successfully completed is the image data transmitted.

[0054] An alternative or supplementary encryption method is, for example, scrambling of the image data, in particular with the addition of randomly generated image parts, cf. Fig. 7 and Fig. 8. Alternatively, standard methods such as AES128 or AES256 or CSA Common Scrambling Algorithm can be used.

[0055] According to the Fig. 7 and Fig. 8, an original image O1 is divided into several sections 1.1 to 4.3. The sections 1.1 to 4.3 are rearranged (scrambled) using lookup tables (LUT), so that a new image O1' is generated. The sections are transferred from the first computing unit ECU-1 to the further computing unit ECU-2 according to the arrangement in the image O1'. The sections are then descrambled again using lookup tables (LUT) and arranged according to the original image O1. According to Fig. 8, additional pseudo-image sections p1 to p4 are added and then removed again.

[0056] Another security measure involves anonymizing image data, for example, by obscuring it with censorship bars or similar. For example, a censorship bar can be provided by default. It can be provided that the censorship bar can be removed with appropriate authorization, for example, by the driver.

Claims

[1] System (10) for monitoring a vehicle interior (12), the system (10) comprising at least one interface for receiving image data (14) acquired with an image acquisition device (xMC), and a first computing device (ECU-1) for providing at least one first monitoring function (F-1) of the system based on the image data (14), as well as a further computing device (ECU-2) for providing at least one further monitoring function (F-2) of the system (10) based on the image data (14), wherein the system (10) is designed such that the image data (14) are provided to the first computing device (ECU-1) via the interface, and that at least a portion (14') of the image data (14) is provided from the first computing device (ECU-1) to the further computing device (ECU-2), wherein the at least one portion (14') of the image data (14) comprises image data,which are required for the further computing device (ECU-2) to provide the further monitoring function (F-2). [2] System according to claim 2, comprising an image capture device (xMC) for capturing image data (14), in particular a camera, for capturing at least a part of the vehicle interior (12), wherein the image capture device (xMC) is designed to provide the captured image data (14) via the interface to the first computing device (ECU-1). [3] System (10) according to one of claims 1 or 2, wherein the first monitoring function (F-1) and / or a further monitoring function which can be provided with the first computing device (ECU-1), and / or the or a further monitoring function (F-2) which can be provided with the further computing device (ECU-2) is one of the following functions: a) a driver monitoring function, in particular for monitoring the driver's state of attention, in particular by detecting a posture, a direction of gaze and / or a position of a head or a face orof the driver's eyes, an opening of the eyes, a blinking frequency of the driver, for example fatigue detection, distraction detection, detection of driver vital signs, gesture detection; b) an occupant monitoring function, for example detection of occupant vital signs, gesture detection, detection of activities in the vehicle interior, detection of situations in the vehicle interior, detection of the presence / absence of occupants in the vehicle interior, detection of the presence / absence of objects in the vehicle interior, c) video telephony function; d) facial recognition function; e) intruder detection in the vehicle interior. [4] System (10) according to one of the preceding claims, wherein the system (10) is designed such that output data (22) based on processed image data are provided by the first computing device (ECU-1) to the further computing device (ECU-2). [5] System (10) according to one of the preceding claims, wherein the system (10) comprises a control device (24) for controlling the image capture device (xMC), and the image capture device (xMC) is designed such that image data (14, 14') can be provided in at least two different contexts (A, B, C, D), wherein the system (10) is designed such that a respective context (A, B, C, D) can be specified by means of the control device (24). [6] Method for operating a system (10) for monitoring a vehicle interior (12) according to one of claims 1 to 5, wherein and with the first computing device (ECU-1) at least one first monitoring function (F-1) of the system (10) is provided based on the image data (14), and with the further computing device (ECU-2) at least one further monitoring function (F-2) of the system (10) is provided based on the image data (14), the method comprising the following steps: Providing image data (14) via the interface to the first computing device (ECU-1), and providing at least a portion (14') of the image data (14) from the first computing device (ECU-1) to the further computing device (ECU-2), wherein the at least one portion (14') of the image data (14) comprises image data required by the further computing device (ECU-2) to provide the further monitoring function (F-2). [7] Method according to claim 6, wherein the method comprises: capturing image data (14) by means of the image capturing device (xMC), in particular a camera, for capturing at least a part of the vehicle interior (12). [8] Method according to one of claims 6 or 7, wherein the method comprises: providing output data based on processed image data by means of the first computing device (ECU-1) to the further computing device (ECU-2). [9] A method according to any one of claims 6 to 8, wherein the method comprises: Predetermining a context (A, B, C, D) for providing the image data (14) by means of a control device (24) for controlling the image capture device (xMC) and providing the captured image data (14) via the interface by means of the image capture device (xMC) according to the predetermining context (A, B, C, D). [10] Method according to claim 9, wherein the first monitoring function (F-1) of the first computing device (ECU-1) and the at least one further monitoring function of the further computing device (ECU-2) are executed at least temporarily simultaneously, and the image capture device (xMC) provides image data (14, 14') in a first context (A) at least temporarily as a function of the first monitoring function (F-1), wherein in the first context (A) the captured image data (14, 14') are output according to a first output format, and the image capture device (xMC) provides image data (14') in at least one further context (B) at least temporarily as a function of the further monitoring function (F-2), wherein in the further context (B) the captured image data (14') are output according to a further output format. [11] Method according to claim 10, wherein switching between the first context (A, B, C, D) and the at least one further context (A, B, C, D) takes place according to a pattern that can be predetermined depending on the monitoring functions (F-1, F-2) to be carried out. [12] Method according to one of claims 6 to 11, wherein at least one encryption and / or authentication method is provided for securing the communication between the first computing device (ECU-1) and the further computing device (ECU-2).

Images