System and procedure for monitoring the interior of a vehicle

Abstract

System (100) for monitoring the interior of a vehicle (10), comprising: at least one first optical sensor unit (112) and at least one second optical sensor unit (122); at least one first illumination unit (114) which is assigned to the at least one first optical sensor unit (112); at least one second illumination unit (124) associated with at least one second optical sensor unit (122); wherein the at least one first illumination unit (114) and the at least one first optical sensor unit (112) form a first subsystem and the at least one second illumination unit (124) and the at least one second optical sensor unit (122) form a second subsystem; and at least one control unit (116, 126) configured to control the at least one first illumination unit (114) and the at least one second illumination unit (124) such that illumination for data acquisition by the at least one first optical sensor unit (112) and the at least one second optical sensor unit (122) is asynchronous, wherein the at least one control unit (116, 126) is configured to perform a reinitialization of the asynchronous illumination under certain circumstances if disturbances caused by the illumination unit (114, 124) of one subsystem are detected in the data of the optical sensor unit (122, 124) acquired by the at least one first optical sensor unit (112) and / or the at least one second optical sensor unit (122).112) of the other subsystem as an indication of an undesired overlap of lighting times of the first lighting unit (114) and the second lighting unit (124) that exceed a threshold, wherein the at least one control unit (116, 126) is further configured to control the at least one first lighting unit (114) and the at least one second lighting unit (124) with a predetermined time offset (Δ) for the lighting and / or based on a timestamp that is used in the vehicle (10) for sensor synchronization, wherein the at least one control unit (116, 126) is further configured to perform the reinitialization of the asynchronous lighting as a restart with respect to the timestamp and / or the predetermined time offset (Δ), wherein the restart with respect to the timestamp and / or the predetermined time offset eliminates this overlap.

Description

[0001] The present disclosure relates to a system for monitoring the interior of a vehicle, a vehicle with such a system, a method for monitoring the interior of a vehicle, and a storage medium for carrying out the method. The present disclosure relates in particular to avoiding interference between camera systems with active illumination. State of the art

[0002] Nowadays, vehicles increasingly use interior monitoring systems that can be configured to monitor occupants and / or detect objects. Such systems can, for example, provide information about the driver's condition and behavior, which can then be used as input for driver assistance systems. Examples of driver assistance systems that can operate using this driver condition and behavior information include emergency braking systems, lane keeping assist, and emergency call systems.

[0003] To ensure the reliable functioning of the driver assistance systems, it is advantageous if the data collected by the interior monitoring systems is of high quality.

[0004] German patent application DE 10 2017 207 206 A1 discloses a method for controlling a driver monitoring system for observing a driver of a vehicle. The driver monitoring system comprises at least a first module with a first sensor device and a first lighting device, and a second module with a second sensor device and a second lighting device. In a first method step, image analysis data, representing the result of an analysis of at least one image of the driver captured by the first sensor device and / or the second sensor device, and / or operating status data, representing an operating state of the driver monitoring system, are received.Subsequently, in a second process step, the image analysis data and / or the operating state data are used to activate a first state and / or a second state of the driver observation system, in each of which switching between a first recording mode and a second recording mode takes place.

[0005] German patent application DE 10 2014 118 387 A1 discloses a detection device for recognizing a gesture and / or a gaze direction of an occupant of a motor vehicle, comprising a first sensor device and at least a second sensor device, wherein each of the sensor devices has a light unit for emitting light, a receiver unit for receiving the light reflected by the occupant and a processing unit for controlling the light unit and for recognizing the gesture and / or the gaze direction based on the reflected light, wherein the processing units of the sensor devices are designed to control the light units synchronously depending on a synchronization signal.

[0006] Publication DE 10 2018 201 154 A1 discloses a method for calibrating at least one sensor or sensors and / or at least one sensor arrangement and / or sensor arrangements, in particular sensors and / or sensor arrangements of motor vehicles, wherein a calibration based on at least one predefined calibration is carried out.

[0007] German patent application DE 10 2019 130 372 A1 discloses a method for synchronizing sensors configured to transmit measurement signals and generate sensor signals based on reflected components of the transmitted measurement signals. During successive measurement cycles, a first measurement is performed using a first sensor and a second measurement is performed using a second sensor. Clock signals for the sensors are generated and synchronized. A first trigger process is performed, according to which a first trigger signal is generated that defines the respective times for triggering the first measurements. A first adjustment process is performed, according to which the first trigger process is temporarily interrupted depending on the clock frequency of the first clock signal. Disclosure of the invention

[0008] It is an objective of the present disclosure to specify a system for monitoring the interior of a vehicle, a vehicle with such a system, a method for monitoring the interior of a vehicle, and a storage medium for executing the method, all of which are capable of providing improved acquisition data. In particular, it is an objective of the present disclosure to improve the reliability of driver assistance systems.

[0009] This problem is solved by the subject matter of the independent claims. Advantageous embodiments are specified in the dependent claims.

[0010] According to an independent aspect of the present disclosure, a system for monitoring the interior of a vehicle, in particular a motor vehicle, is specified according to claim 1. The system comprises at least one first optical sensor unit and at least one second optical sensor unit; at least one first lighting unit associated with the at least one first optical sensor unit; at least one second lighting unit associated with the at least one second optical sensor unit; wherein the at least one first lighting unit and the at least one first optical sensor unit form a first subsystem and the at least one second lighting unit and the at least one second optical sensor unit form a second subsystem; and at least one control unit configured to control the at least one first lighting unit and the at least one second lighting unit in such a manner as tothat illumination for data acquisition is carried out asynchronously by the at least one first optical sensor unit and the at least one second optical sensor unit, wherein the at least one control unit is configured to perform a reinitialization of the asynchronous illumination under certain circumstances when disturbances are detected in the data acquired by the at least one first optical sensor unit and / or the at least one second optical sensor unit, disturbances which are generated by the illumination unit of one subsystem in the data of the optical sensor unit of the other subsystem as an indication of an undesired overlap of illumination times of the first illumination unit and the second illumination unit, and which exceed a threshold, wherein the at least one control unit is further configuredto control at least one first lighting unit and at least one second lighting unit with a predetermined time offset for the lighting and / or based on a timestamp used in the vehicle for sensor synchronization, wherein the at least one control unit is further configured to perform the reinitialization of the asynchronous lighting as a restart with respect to the timestamp and / or the predetermined time offset, wherein the restart with respect to the timestamp and / or the predetermined time offset eliminates this overlap.

[0011] Preferably, the at least one control unit is configured to control the at least one first lighting unit, the at least one second lighting unit, the at least one first optical sensor unit, and the at least one second optical sensor unit in such a way that: - the data acquisition by the at least one first optical sensor unit and the illumination by the at least one first illumination unit are synchronous, and - the data acquisition by at least one second optical sensor unit and the illumination by at least one second lighting unit are performed synchronously.

[0012] For example, the corresponding optical sensor unit can set the pace so that the lighting is activated shortly before the start of data acquisition and deactivated shortly after the end of data acquisition.

[0013] The term "asynchronous," as used in the context of this disclosure, refers to a non-simultaneous illumination by the at least one first illumination unit and the at least one second illumination unit. In other words, the illumination times of the at least one first illumination unit and the at least one second illumination unit do not overlap; rather, the illumination occurs sequentially.

[0014] The first and second optical sensor units operate with active illumination, e.g., in the IR range. This illumination is pulsed, meaning it is only active when the sensor unit is acquiring data. Pulsed control of the sensor units and the corresponding illumination can reduce heat generation and / or energy consumption of the system.

[0015] The at least one first lighting unit and the at least one first optical sensor unit form a first subsystem, and the at least one second lighting unit and the at least one second optical sensor unit form a second subsystem. The present disclosure is not limited to the first and second subsystems, and the indoor monitoring system according to the invention can comprise at least one further subsystem. This at least one further subsystem can comprise at least one further lighting unit and at least one further optical sensor unit.

[0016] Preferably, the at least one control unit comprises a first control unit configured for controlling the at least one first lighting unit, and a second control unit configured for controlling the at least one second lighting unit. in particular wherein the first control unit and the second control unit are implemented on different control units of the vehicle, or wherein the first control unit and the second control unit are implemented on the same control unit of the vehicle.

[0017] Preferably, the at least one first optical sensor unit is a camera, in particular an infrared camera; and / or at least one second optical sensor unit, a camera, in particular an infrared camera.

[0018] Preferably, the at least one first lighting unit is an infrared lighting unit, and / or wherein the at least one second lighting unit is an infrared lighting unit.

[0019] According to a further independent aspect of the present disclosure, a vehicle, in particular a motor vehicle, is specified according to claim 6. The vehicle comprises the interior monitoring system according to the embodiments of the present disclosure.

[0020] The term "vehicle" includes cars, trucks, buses, motorhomes, motorcycles, etc., used for the transport of people, goods, etc. In particular, the term includes motor vehicles for passenger transport.

[0021] According to another independent aspect of the present disclosure, a method for monitoring the interior of a vehicle, in particular a motor vehicle, is specified according to claim 7.

[0022] According to another independent aspect of the present disclosure, a storage medium is specified. The storage medium may include a software program configured to run on one or more processors and thereby execute the method for monitoring the interior of a vehicle described in this document. Brief description of the drawings

[0023] Examples of the manifestation of the revelation are shown in the figures and are described in more detail below. They show: Fig. 1 schematically a system for monitoring the interior of a vehicle according to embodiments of the present disclosure, Fig. 2 a graph of the asynchronous illumination according to embodiments of the present disclosure, Fig. 3 schematically a vehicle with a driver assistance system according to embodiments of the present disclosure, and Fig. 4 a flowchart of a method for monitoring the interior of a vehicle according to embodiments of the present disclosure. Implementations of the revelation

[0024] Unless otherwise noted, the same reference symbols are used for identical and equivalent elements in the following.

[0025] Fig. Figure 1 schematically shows a system 100 for monitoring the interior of a vehicle according to embodiments of the present disclosure.

[0026] System 100 comprises a first subsystem 110 and a second subsystem 120. The first subsystem 110 comprises at least one first optical sensor unit 112, at least one first illumination unit 114 associated with the at least one first optical sensor unit 112, and a first control unit 116. The second subsystem 120 comprises at least one second optical sensor unit 122, at least one second illumination unit 124 associated with the at least one second optical sensor unit 122, and a second control unit 126.

[0027] In some embodiments, the at least one first optical sensor unit 112 is a camera, such as an infrared camera. Additionally or alternatively, the at least one second optical sensor unit 122 can be a camera, such as an infrared camera. In further embodiments, the at least one first illumination unit 114 can be an infrared illumination unit. Additionally or alternatively, the at least one second illumination unit 124 can be an infrared illumination unit.

[0028] The first control unit 116 can be implemented in a first control unit of the vehicle or can be the first control unit of the vehicle. The second control unit 126 can be implemented in a second control unit of the vehicle or can be the second control unit of the vehicle. The first control unit 116 and the second control unit 126 can be different control units, in particular different control units. In particular, the first control unit 116 and the second control unit 126 can be implemented in separate software and / or hardware modules.

[0029] However, the present disclosure is not limited to the example of separate control units and in alternative embodiments the same control unit can be used to control both subsystems 110, 120.

[0030] The first control unit 116 and the second control unit 126 can be communicatively connected to each other. For example, the first control unit 116 and the second control unit 126 can be networked via a data bus 130 and thus communicate bidirectionally with each other.

[0031] The first control unit 116 is configured to control the at least one first optical sensor unit 112 and the at least one first illumination unit 114. In particular, the first control unit 116 can control the at least one first optical sensor unit 112 to acquire sensor data, and can control the at least one first illumination unit 114 so that it exposes the image while the sensor data is being acquired by the at least one first optical sensor unit 112.

[0032] The second control unit 126 is configured to control the at least one second optical sensor unit 122 and the at least one second illumination unit 124. In particular, the second control unit 126 can control the at least one second optical sensor unit 122 to acquire sensor data, and can control the at least one second illumination unit 124 so that it exposes the image while the sensor data is being acquired by the at least one second optical sensor unit 122.

[0033] The illumination by the at least one first lighting unit 114 and the at least one second lighting unit 124 can be pulsed, i.e., the illumination is only active when the corresponding sensor unit is receiving data. Pulsed control of the sensor units and the corresponding illumination can reduce heating and / or energy consumption of the system.

[0034] System 100 ensures that the illumination for data acquisition by the at least one first optical sensor unit 112 and the at least one second optical sensor unit 122 is asynchronous. In other words, the at least one first illumination unit 114 is active and the at least one second illumination unit 124 is inactive, while the at least one first optical sensor unit 112 is active and acquiring sensor data. Similarly, the at least one second illumination unit 124 is active and the at least one first illumination unit 114 is inactive, while the at least one second optical sensor unit 122 is active and acquiring sensor data.

[0035] This prevents interference from the lighting unit of one subsystem in the sensor data of the other subsystem. As a result, high-quality sensor data can be provided. For example, this can be advantageous in occupant detection, as the high-quality sensor data supports real-time capability of the system; that is, there is no delay in the detection functions due to, for example, insufficient sensor data.

[0036] Fig. Figure 2 shows a graph of the asynchronous lighting according to embodiments of the present disclosure. In particular, pulses indicating active lighting are shown as a function of time t for a first subsystem A and a second subsystem B of an interior monitoring system according to the invention.

[0037] The active illumination time of the first subsystem (i.e., the active pulse duration or pulse width) can be equal to or shorter than the inactive illumination time (i.e., the inactive pulse duration or pulse width) of the second subsystem. Furthermore, the active illumination time of the second subsystem (i.e., the active pulse duration or pulse width) can be equal to or shorter than the inactive illumination time (i.e., the inactive pulse duration or pulse width) of the first subsystem. This prevents the exposure times of the subsystems from overlapping.

[0038] Preferably, the active illumination time of the first subsystem is substantially equal to the active illumination time of the second subsystem. Additionally or alternatively, the inactive illumination time of the first subsystem can be substantially equal to the inactive illumination time of the second subsystem. However, the present disclosure is not limited to this, and the active and / or inactive illumination times of the subsystems can be of different lengths.

[0039] In some embodiments, the illumination can be based on a timestamp. The timestamp can ensure that both subsystems operate within the same timeframe or run synchronously. For example, the timestamp could be a timestamp used in the vehicle for sensor synchronization. In other words, the timestamp could be a timestamp already present in the vehicle, used, for example, to synchronize sensors as part of sensor data fusion.

[0040] Illumination by means of at least one first illumination unit and at least one second illumination unit can be performed with a predetermined and / or variably configurable time offset Δ. For example, the first subsystem and the second subsystem can be initialized based on the timestamp, whereby the predetermined time offset Δ starting from the initialization time is applied to both subsystems to enable asynchronous illumination.

[0041] In some embodiments, the system can perform a reinitialization of the asynchronous lighting, particularly with respect to the timestamp and / or the predetermined time offset. This reinitialization can occur, for example, if a time drift develops in the lighting control over the system's operating time. This can lead to an overlap of lighting times, which can be prevented by restarting with respect to the timestamp and / or the predetermined time offset.

[0042] In some embodiments, the asynchronous lighting can be reinitialized regularly at predetermined time intervals and / or under certain circumstances. For example, the asynchronous lighting can be reinitialized if disturbances exceeding a threshold are detected in the data acquired by the at least one first optical sensor unit and / or the at least one second optical sensor unit. These disturbances may be generated by the lighting unit of one subsystem in the sensor data of the other subsystem, which may indicate an undesired overlap of the lighting times. This overlap can be eliminated by restarting with respect to the timestamp and / or the predetermined time offset.

[0043] Fig. Figure 3 schematically shows a vehicle 10 with a driver assistance system 300 for automated driving according to embodiments of the present disclosure.

[0044] The driver assistance system 300 can be configured for automated driving. In automated driving mode, the longitudinal and / or lateral control of the vehicle 10 is performed automatically. The driver assistance system 300 thus takes over vehicle control. For this purpose, the driver assistance system 300 controls the drive 20, the transmission 22, the hydraulic service brake 24, and the steering 26 via intermediate units not shown.

[0045] For the planning and execution of automated driving, environmental information from an environmental sensor system that monitors the vehicle's surroundings is received by the driver assistance system 300. In particular, the vehicle 10 can include at least one environmental sensor 12, which is configured to record environmental data specifying the vehicle's surroundings. The at least one environmental sensor 12 can, for example, include one or more LiDAR systems, one or more radar systems, one or more laser scanners, and / or one or more cameras.

[0046] The vehicle 10 further comprises the interior monitoring system 100 according to the embodiments described in this disclosure. The system 100 can be communicatively connected to the driver assistance system 300. Alternatively, the interior monitoring system 100 can be included in or integrated into the driver assistance system 300.

[0047] The interior monitoring system 300 can, for example, generate driver status and behavior information that is provided to the driver assistance system 300. The driver assistance system 300 can then activate the appropriate assistance function based on this interior monitoring information.

[0048] Examples of driver assistance systems that can use interior monitoring include emergency braking systems, lane keeping assist systems, and emergency call assist systems. However, the embodiments of the present disclosure are not limited to these, and other types of driver assistance systems that can use interior monitoring are conceivable.

[0049] Fig.Figure 4 schematically shows a flowchart of a method 400 for monitoring the interior of a vehicle according to embodiments of the present disclosure. The method 400 can be implemented by suitable software that can be executed by one or more processors (e.g., a CPU).

[0050] Method 400 comprises in block 410 controlling at least one first lighting unit to illuminate the interior for data acquisition by at least one first optical sensor unit; and in block 420 controlling at least one second lighting unit to illuminate the interior for data acquisition by at least one second optical sensor unit, wherein the illumination of the interior by the at least one first lighting unit and the at least one second lighting unit is asynchronous.

[0051] According to the invention, the vehicle's interior is illuminated at different times to record sensor data, thereby avoiding interference. In particular, it prevents the lighting unit of one subsystem from causing disturbances in the sensor data of another subsystem. As a result, high-quality sensor data can be provided. This can be advantageous, for example, in occupant detection, as the high-quality sensor data supports the system's real-time capability; that is, there is no delay in the detection functions, e.g., due to insufficient sensor data quality. Furthermore, a software-based measure is provided, so that no physical filtering of disturbances in the sensor data is required.

[0052] Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims

[1] System (100) for monitoring the interior of a vehicle (10), comprising: at least one first optical sensor unit (112) and at least one second optical sensor unit (122); at least one first illumination unit (114) which is assigned to the at least one first optical sensor unit (112); at least one second illumination unit (124) associated with at least one second optical sensor unit (122); wherein the at least one first illumination unit (114) and the at least one first optical sensor unit (112) form a first subsystem and the at least one second illumination unit (124) and the at least one second optical sensor unit (122) form a second subsystem; and at least one control unit (116, 126) configured to control the at least one first illumination unit (114) and the at least one second illumination unit (124) such that illumination for data acquisition by the at least one first optical sensor unit (112) and the at least one second optical sensor unit (122) is asynchronous, wherein the at least one control unit (116, 126) is configured to perform a reinitialization of the asynchronous illumination under certain circumstances if disturbances caused by the illumination unit (114, 124) of one subsystem are detected in the data of the optical sensor unit (122, 124) acquired by the at least one first optical sensor unit (112) and / or the at least one second optical sensor unit (122).112) of the other subsystem as an indication of an undesired overlap of lighting times of the first lighting unit (114) and the second lighting unit (124) that exceed a threshold, wherein the at least one control unit (116, 126) is further configured to control the at least one first lighting unit (114) and the at least one second lighting unit (124) with a predetermined time offset (Δ) for the lighting and / or based on a timestamp that is used in the vehicle (10) for sensor synchronization, wherein the at least one control unit (116, 126) is further configured to perform the reinitialization of the asynchronous lighting as a restart with respect to the timestamp and / or the predetermined time offset (Δ), wherein the restart with respect to the timestamp and / or the predetermined time offset eliminates this overlap. [2] The system (100) according to claim 1, wherein the at least one control unit (116, 126) is configured to control the at least one first lighting unit (114), the at least one second lighting unit (124), the at least one first optical sensor unit (112) and the at least one second optical sensor unit (122) such that: - the data acquisition by the at least one first optical sensor unit (112) and the illumination by the at least one first illumination unit (114) are carried out synchronously, and - the data acquisition by at least one second optical sensor unit (122) and the illumination by at least one second illumination unit (124) are carried out synchronously. [3] The system (100) according to one of claims 1 or 2, wherein the at least one control unit (116, 126) comprises a first control unit (116) configured for controlling the at least one first lighting unit (114) and a second control unit (126) configured for controlling the at least one second lighting unit (124), in particular wherein the first control unit (116) and the second control unit (126) are implemented on different control units of the vehicle (10), or wherein the first control unit (116) and the second control unit (126) are implemented on the same control unit of the vehicle (10). [4] The system (100) according to any one of claims 1 to 3, wherein at least one first optical sensor unit (112) is a camera, in particular an infrared camera; and / or wherein at least one second optical sensor unit (122) is a camera, in particular an infrared camera. [5] The system (100) according to any one of claims 1 to 4, wherein the at least one first illumination unit (114) is an infrared illumination unit, and / or wherein the at least one second illumination unit (124) is an infrared illumination unit. [6] Vehicle (10), in particular motor vehicle, comprising the interior monitoring system (100) according to any one of claims 1 to 5. [7] Method (400) for monitoring the interior of a vehicle (10), comprising: Controlling (410) at least one first lighting unit (114) to illuminate the interior for data acquisition by at least one first optical sensor unit (112); and Controlling (420) at least one second lighting unit (124) to illuminate the interior for data acquisition by at least one second optical sensor unit (122), wherein the at least one first lighting unit (114) and the at least one first optical sensor unit (112) form a first subsystem and the at least one second lighting unit (124) and the at least one second optical sensor unit (122) form a second subsystem, wherein the illumination of the interior by the at least one first lighting unit (114) and the at least one second lighting unit (124) is asynchronous, wherein the at least one control unit (116, 126) performs a reinitialization of the asynchronous illumination at predetermined time intervals and / or under certain circumstances when disturbances generated by the lighting unit (114, 124) of one subsystem are detected in the data of the optical sensor unit (122, 112) of the other subsystem as an indication of an undesired overlap of illumination times of the first lighting unit (114) and the second lighting unit (124) that exceed a threshold,wherein the at least one first lighting unit (114) and the at least one second lighting unit (124) are controlled with a predetermined time offset (Δ) for the lighting and / or based on a timestamp which is used in the vehicle (10) for sensor synchronization, wherein the reinitialization of the asynchronous lighting is carried out as a restart with respect to the timestamp and / or the predetermined time offset (Δ), wherein the restart with respect to the timestamp and / or the predetermined time offset eliminates this overlap. [8] Storage medium comprising a software program configured to run on one or more processors and thereby to perform the method (400) according to claim 7.

Images