Method for capturing images of the interior of a vehicle and vehicle interior camera system

Abstract

TITLE: Method for capturing images of the passenger compartment of a vehicle and vehicle passenger compartment camera system. Method for capturing images in a vehicle (102) with a camera (124), a first and a second lighting installation (126), and a lighting installation (128) consisting of controlling: - the first installation (126) to illuminate a first area (140) according to a first mode (220) depending on a first application or for the images to be taken by the camera (124) in the first area (140), - the second installation (128) to illuminate a second area (142) according to a second lighting mode (230) depending on a second application, and - controlling the camera (124) to take images of the passenger compartment (102) of the vehicle. Figure 1

Description

Title of the invention: Method for capturing images of the passenger compartment of a vehicle and vehicle passenger compartment camera system FIELD OF THE INVENTION

[0001] The present invention relates to a method for capturing images of a vehicle interior with a camera, a first lighting installation and a second lighting installation as well as a computing unit, a vehicle interior camera system and a computer program for applying the method. STATE OF THE ART

[0002] In vehicles, cameras are used to detect the passenger compartment. For this purpose, driver observation systems are used, for example, which consist of illuminating the driver with a lighting installation which is, typically, infrared light. The light reflected by the driver is detected by the camera or by a sensor (detector) to be used by image processing and thus make it possible to determine the position of the driver's head and the direction of his gaze.

[0003] In general, such systems can also be used for monitoring the passenger compartment. This is how, for example, it makes it possible to detect behavior that does not comply with regulations in the case of shared vehicles (for example, smoking which is not permitted) or to analyze the state of the passenger compartment of the vehicle (for example, forgetting objects or dirt).

[0004] DISCLOSURE AND ADVANTAGES OF THE INVENTION

[0005] The present invention relates to a method for capturing images in a vehicle interior as well as a computing unit, a camera for capturing images in the vehicle interior and a computer program for carrying out the method.

[0006] The invention relates to systems comprising a camera (or sensor or detector) for capturing or taking single images or also a succession of images, i.e. video films and thus in particular for monitoring or detecting the vehicle interior. A particular objective is that of lighting for such cameras. Two lighting installations can be used for a determined lighting zone of the interior. This can be a first lighting installation for a first zone of the interior and a second lighting installation for illuminating a second zone of the interior. The two zones can be different, but also overlap at least partially. The first zone is, for example, that of the driver, i.e. the first lighting installation only illuminates the driver and the second zone, on the other hand, illuminates, for example, the passenger zone and / or the rear seat zone.

[0007] Typically, such lighting installations emit infrared light so that images can be taken with the camera, even at night, without disturbing the occupants of the vehicle. The camera (or detector) is designed to capture, in particular, infrared light and visible light.

[0008] The two lighting installations can be, for example, next to each other (one to the left, the other to the right) of the camera; the camera and the two lighting installations can be housed in a common housing and form part of the vehicle interior camera system. Such a vehicle interior camera system or each time the camera and, if applicable, the two lighting installations can be integrated in particular in the rear view mirror or in a central display or in the dashboard or in a module installed on the roof. Advantageously, however, the camera is installed in the middle or centrally in the vehicle to have a good view of the vehicle interior. In principle, the lighting installations can be further apart than the camera.

[0009] In this context, one can also have a single camera system. One can also envisage a two-camera or multiple camera system with at least one other camera installed at a suitable location in the vehicle or the vehicle interior.

[0010] The use of two lighting installations makes it possible to effectively illuminate two different areas of the passenger compartment. But as has been observed, the lighting required for different areas may be different depending on the areas, for example, for different applications. Thus, for example, for the driver the lighting would be different from that for the other parts; in other words, the same lighting is not always available for each area.

[0011] In this context, a first lighting installation is proposed for illuminating the first area according to a first lighting mode selected according to the first application. The first application is an application in which the camera takes images of the first area. Correspondingly, the second lighting installation illuminates the area with a second lighting mode selected according to the second application. The first lighting mode and / or the second lighting mode can be changed depending on the situation. In particular, it is also possible to control the two lighting installations separately and independently of each other, i.e. to control them individually.This is how, for example, the driver will be illuminated in the first zone differently, more intensely or more frequently than the passenger because, in the case of the driver, his attention must be monitored while for the passenger, monitoring of the occupancy is sufficient.

[0012] Preferably, the first lighting mode consists of emitting im light pulses at a first lighting frequency and / or a first pulse duration and / or the second lighting mode consists of emitting light pulses at a second lighting frequency and / or a second pulse duration. The term "light pulse" means that light is emitted, for example, a brief activation of the light source for a determined pulse duration, for example, a few milliseconds; this can be done at a determined frequency (the lighting frequency) repeatedly. In the case of an identical mode and duration of the light pulses, this can be repeated. For a constant mode and duration of the light pulses, by reducing the lighting frequency, the overall lighting duration can be reduced and thus, in particular, the heat generated in the lighting installation or in the control or driver circuit.To control lighting systems, for example, a pulse width modulation (PWM) control can be used with a correspondingly selected detection ratio.

[0013] Similarly, the camera can be controlled for taking pictures at a picture frequency (frame rate) that corresponds at least to the maximum of the first and second lighting frequencies. If, for example, the first lighting frequency is 60 Hz, the second lighting frequency being only, for example, 30 Hz, 10 Hz or 1 Hz, then the camera's picture frequency can also be 60 Hz. This provides a sufficiently high picture frequency with sufficiently illuminated images of the driver while the energy input and heat generated by the passenger lighting system will be significantly reduced.

[0014] If another camera is used, one of the two lighting systems can be associated with a camera. Then, for example, the recording frequency of the first camera can be matched to the (possibly higher) lighting frequency of the first lighting system and the recording frequency of the other camera can be matched to a (possibly lower) lighting frequency of the second lighting system. In this case, the other camera will also operate more efficiently.

[0015] This principle can also be transposed to other lighting installations and / or other cameras and to any combinations.

[0016] Preferably, the first lighting frequency and / or the pulse duration are at least occasionally different from the second lighting frequency or the pulse duration to reduce the temperature rise of the second lighting installation. In principle, it is also possible to select, for example, both lighting frequencies or pulse durations equal to the extent that this corresponds to a certain situation or to suitable applications which are not necessarily identical, hands need the same type of lighting. According to the invention, in the case of using another camera, this other camera is controlled to take images of the passenger compartment of the vehicle with a shooting frequency which corresponds at least to the minimum of the first lighting frequency and the second lighting frequency.

[0017] The first application is at least one of the following applications: driver observation, tracking of the driver's gaze direction and observation of the driver's eyelids. For the second application, it is at least one of the following applications: presence monitoring, videoconferencing, gesture control and data recording. The data recording is, in particular, an "event data recording" so that in the event of an accident, the characteristic data, in particular the images concerning the state and / or the number of passengers, can be recorded in a non-volatile memory for the respective phases of the accident (before, during and after the accident, also called pre-crash, crash, post-crash).

[0018] A computing unit according to the invention, for example, the vehicle control device or a control unit of a camera or a vehicle interior camera system, is designed in particular using programming technology to apply the method of the invention.

[0019] The invention also relates to a vehicle interior camera system comprising a camera, a lighting installation, a second lighting installation and a computing unit according to the invention. Likewise, if necessary, another camera can be provided. The camera, the first lighting installation and the second lighting installation can be housed in a common housing, the computing unit being installed elsewhere than in the housing and is connected in particular by power supply and signal transmission lines, i.e., in particular, to the camera and the lighting installations in order to control them and receive the images from the camera. The control circuits and the driver circuits of at least one of the cameras, the first lighting installation and the second lighting installation can also be in the computing unit, if necessary also in the housing.

[0020] The method according to the invention can also be implemented in the form of a computer program or a computer program product with program code for carrying out all the method steps; advantageously, since this results in particularly low costs, in particular if the control device is also used for other applications and exists anyway. Finally, a machine-readable storage medium is provided having the recording of the computer program as described above. Suitable storage media or data carriers for the programs Computer memory devices include magnetic, optical or electrical memory devices such as, for example, hard drives, flash memory, EEPROM memory, DVDs or others. Downloading a program via a computer network (internet, intranet and others) is also possible. Such downloading can be done by cable or without a cable, for example, via a WLAN network, a 3G, 4G, 5G or 6G network. Brief description of the drawings

[0021] The present invention will be described below in more detail using examples of methods and calculation units according to the invention represented in the appended drawings in which:

[0022] [Fig.l] diagram of a vehicle equipped with a passenger compartment camera system according to a preferred embodiment,

[0023] [Fig.2] shooting and lighting sequence according to a preferred embodiment relative to the method of the invention,

[0024] [Fig.3] diagram of another preferred embodiment of a camera system vehicle interior according to the invention.

[0025] DESCRIPTION OF EMBODIMENTS OF THE INVENTION

[0026] [Fig. 1] schematically shows a vehicle 100 equipped with a vehicle interior camera system 120 according to the invention corresponding to a preferred embodiment. The vehicle interior camera system 120 serves to capture or take views of the interior 102 of the vehicle; it comprises a camera 124, a first lighting installation 126 and a second lighting installation 128 housed in a common housing 122. The housing with its components is installed in the middle of the vehicle, for example, on the rearview mirror or integrated into the rearview mirror.

[0027] The cabin camera system 120 comprises a computing unit 130 which is, for example, a special control device for the cabin camera system 120 or the vehicle control device 100 used for other applications. The computing unit 130 is connected to the camera 124 by power supply and signal transmission lines 132a, 132b, 132c connected to the camera 124, to the first lighting installation 126 and to the second lighting installation 128 in order to be able to control them and to receive the images captured by the camera 124.The supply and / or signal lines 132b, 132c can be grouped and simplified for a common control line, if an additional coded signal is available for controlling the appropriate lighting installation 126 or 128, in order to be able to reactivate in the camera 124 an electronic switch and, for example, a MOSFET or a MOSFET switch to activate the desired lighting installation 126 or 128.

[0028] The first lighting installation 126 illuminates a first zone 140 (simply indicated here); the first zone 140 comprises, for example, the driver's area with his seat 104. The second lighting installation 128 is provided to illuminate a second zone 142 (and which is only indicated here); the second zone 142 comprises, for example, the passenger's area and his seat 106. The second zone 142 could also additionally or alternatively cover the rear bench seat 108.

[0029] The camera 124 has a reception area 144 which covers, for example, the driver's seat 104, the passenger seat 106 and the rear bench seat 108 in particular, i.e. mainly the entire passenger compartment 102 of the vehicle in any case to the extent that these locations are occupied by a person.

[0030] [Fig.2] shows shooting and lighting sequences applied by the method of invention in a preferred embodiment. All of this will be described below, in more detail with reference to the cabin camera system 120 of [Fig.l].

[0031] The shooting mode 210 of the camera is represented as a function of time or the time axis 200 with a first lighting mode 220 of the first lighting installation and a second lighting mode 230 of the second lighting installation.

[0032] The camera capture mode 210 comprises the illumination durations or length 240, for example, 4 ms. The intervals 242 between the start times of the separate illumination durations are, for example, 16.67 ms. This corresponds to a capture frequency of 60 Hz, i.e., the camera detector will be illuminated 60 times per second for 4 ms each time. The camera is controlled appropriately. It could also be envisaged that the camera or the vehicle interior camera system and the image processing carried out downstream, work at higher image repetition frequencies than 60 Hz, for example, 90 Hz; for this, reference is not made to a complete image, but only to partial regions characteristic of the image, i.e. regions of interest (ROI) to transmit while respecting the maximum transmission capacity of the image band of the image sensor.

[0033] The lighting mode 220 of the first lighting installation comprises lighting durations 240, for example, also equal to 4 ms; this also concerns the lighting pause with a pulse duration of 4 ms. The interval 242 of the start times of the separate light pulses is, for example, also 16.67 ms. This results in a first lighting frequency of 60 Hz, i.e. the first zone will be illuminated 60 times per second for 4 ms each time. For this, the first lighting installation is controlled appropriately. In this way, for each separate illumination of the camera detector, the first zone, i.e., for example, the driver, will be illuminated with the first lighting installation to obtain an (infrared) image of the driver.

[0034] The lighting mode 230 of the second lighting installation comprises the lighting durations which correspond to a duration or length 240, for example of 4 ms; these are thus light pulses with a pulse duration of 4 ms. The intervals 244 between the starting times of the separate light pulses which follow one another are thus, however, for example, 33.333 ms. This can correspond to a second lighting frequency of 30 Hz, that is to say that the second zone will be illuminated 30 times per second for a respective duration of 4 ms. The second lighting installation can be controlled appropriately. Thus, for each second illumination of the camera detector, for the second zone, that is to say for the passenger, there will be illumination by the second lighting installation to have an image (in infrared light) of the passenger.

[0035] As indicated, both the first and second lighting frequencies can be modified by ensuring that the camera's image capture frequency then corresponds to the higher of the two lighting frequencies (insofar as these frequencies are different). It is possible, for example, to consider setting the second lighting frequency only to 10 Hz or only to 1 Hz if this is sufficient for a second application, i.e. an application for which the camera takes images of the second area for which the second lighting installation is provided. Similarly, both lighting frequencies can also be set at the same level, i.e., if necessary only from time to time. This image capture frequency can also be chosen as the two lighting frequencies.

[0036] Due to functional requirements, for example, for driver observation as a first application, it may be necessary to have a higher detection rate and a higher processing rate of up to 60 Hz so that the first lighting installation or driver-side lighting installation will be activated in synchronism for each image capture of the camera, i.e. its integration (illumination) time.Typically, the second lighting system will then also be activated at a frequency of 60 Hz, which would have the consequence that although the second area, i.e., for example, the right side of the vehicle interior, will be illuminated at a frequency of 60 Hz, the second characteristic applications or characteristic operations, such as, for example, a video conference, would only require 30 Hz or presence monitoring (presence detection) would only require a frequency of 10 Hz or even 1 Hz.

[0037] An advantage of the proposed procedure such as that described in relation to [Fig.2] is in particular a demand-dependent activation of the (infrared) lighting so that the lighting and the camera - but then with a reduced thermal stress in the camera, remains available even at high ambient temperatures. Likewise, fewer computing resources or ECU resources (in the case of the computing unit mentioned) will be required in the case of a so-called SOC (system on chip), preferably through intelligent distribution of computing capacity. Such distribution includes, in particular, the distribution of the so-called perception stack and the video stack. These are two different image streams for the different applications, namely, for example, driver observation (perception stack) and video conferencing and / or event data recording (video stack). This also allows for cost savings.

[0038] [Fig. 3] schematically shows another preferred embodiment of a vehicle interior camera system 320. This system comprises a camera 324, a first lighting installation 326 and a second lighting installation 328 installed in a common housing 322. A computing unit 330 is provided which is, for example, connected by power supply lines 332 (there may be several) for the lighting installation and a current and signal line 332 for the camera.

[0039] The computing unit 330 comprises a microcontroller 331 implemented as a system-on-chip which comprises a RAM memory 332 and a ROM memory 333; it is supplied by a current line and a signal line 332b and comprises, in a deserializer 336 (in this, for example, the serial data blocks are converted into a parallel data block) processed images (the data) and transmits them in the form of a video stack 334 and a perception stack 335. The video stack 334 is transmitted via the Ethernet line 340 to the control device 350 of the vehicle; this is provided, for example, to control the human-machine interface and is thus available for a video conference as a second application 360.The perception stack 335 is transmitted via a CAN connection 341 and another bus, a CAN portal 351 and thus to a control device 352 of the vehicle; this is used, for example, to control partially automated driving or other safety-related functions and thus requires, for example, driver observation as the first application 361.

[0040] In addition, the microcontroller 331 can control the lighting installations by an appropriate control circuit 337 and a pilot circuit 338.

[0041] The possible steps of the method will be described below by way of example for the perception stack and the video stack. For the perception stack, a frame rate of 60 Hz is required; for the video stack, however, a rate of 30 Hz is sufficient.

[0042] First, the color data and infrared data of the camera images are divided. This is a processing step performed in the SoC element or in a dedicated ISP (image signal processing) chip and which divides the original input image into a first image, or a color image, for example in RGB format and in a second image or IR image; these two images are extracted. The color image with the RGB pixels (e.g., the detector used for capture) is itself also sensitive to IR infrared radiation and is intended to subtract the IR image from the color image to compensate for the effect of a "washed-out color image".

[0043] The perception stack may comprise, for example, different computer vision functions such as, for example, head or eye tracking, facial recognition or face identification (face-ID) and also attention recognition, body attitude recognition and driver or passenger reaction detection, etc. and not only evoked observation of the driver. These different functions may be encompassed in the first application.

[0044] As an input signal to the processing path of the perception stack, the extracted IR image can be used, as discussed, which typically has a quarter of the original resolution of the original input image. In addition, the original input image can be used for eye tracking to achieve greater accuracy in eye tracking and gaze direction. For example, a face found by head tracking can be used as an initial search range in time and space in the current image for eye detection in the next image, since eye tracking has not yet been performed.

[0045] Functions like, for example, head tracking also operate at lower frequencies, for example, 30 Hz while eye tracking should be performed at higher frequencies, for example, 60 Hz. Thus, the frequency of 60 Hz would be interesting for the perception stack.

[0046] The processing path of the video stack, i.e. the second application, is activated in particular if a video conference is activated by the user, for example, the driver or the passenger. The color image is, for example, processed by an image signal processing pipeline (ISP) so that in the following steps, the image quality, its color and the image intelligence and contrast will be improved. In addition, the background can, for example, be replaced at the request of the driver by masking it, or with a background image. Thus, the image can be encoded and / or compressed and transmitted via Ethernet as indicated to another control device for outputting the image via a human-machine interface.

[0047] A particular camera control can also be provided that controls the appropriate camera, in particular with camera parameters such as analog amplification, digital amplification and illumination time depending on the requirements for the applications, in particular, the first application. For computer vision functions (the second application) a control can be provided. or regulation with good image brightness and good contrast and sharpness, for example, for a maximum lighting duration of 4 ms to avoid sharpness defect during movement.

[0048] A color image (in RGB) requires a good representation on the human-machine interface (e.g., a display) and possibly a longer illumination time of up to 6 ms; this also applies if, for example, the user activates the video conference function. Depending on the environment, in particular the lighting situation, one can switch, depending on the situation, for example, in favor of better image quality, from the IR image to the RGB color image. If, for example, the tracking confidence is precisely high, i.e., if eye tracking or other tracking with IR is not safe, one can increase the quality of the color image or vice versa.

[0049] Thus, in this way, different applications that require different frame rates are made possible by different lighting installations while avoiding excessive heating of the vehicle interior camera system.

[0050] NOMENCLATURE OF MAIN ELEMENTS

[0051] 100 Vehicle

[0052] 102 Passenger compartment

[0053] 104 Driver's seat

[0054] 106 Passenger seat

[0055] 108 Rear bench seat

[0056] 120 Cabin camera system

[0057] 122 Housing

[0058] 124 Camera

[0059] 126 First lighting installation

[0060] 128 Second lighting installation

[0061] 132 Signal transmission and power supply line

[0062] 140 First lighting zone / shooting zone

[0063] 142 Second lighting zone

[0064] 200 Time axis

[0065] 210 Shooting mode

[0066] 220 First lighting mode

[0067] 230 Second lighting mode

[0068] 240 Duration of a lighting time

[0069] 242 Interval between lighting times

[0070] 320 Cabin camera system

[0071] 322 Common housing

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085] 324 Camera 326 First lighting system 328 Second lighting system 331 Microcontroller 332 Power and signal line / RAM memory 333 ROM memory 334 Computing unit / video stack 335 Perception stack 336 Deserializer 340 Ethernet link 341 CAN bus 350 Control unit 351 CAN portal 361 First application

Claims

Claims

1. A method for capturing images in a vehicle interior (102) using a camera (124, 324), a first lighting installation (126, 326) and a second lighting installation (128, 328) comprising the steps of: - controlling the first lighting installation (126, 326) to illuminate a first area (140) of the vehicle interior according to a first lighting mode (220) chosen as a function of a first application (361) or provided for the images to be taken by the camera (124, 324) in the first area (140), - controlling the second lighting installation (128, 328) to illuminate a second area (142) of the vehicle interior according to a second lighting mode (230) chosen as a function of a second application (360) for the images to be taken with the camera (124, 324) in the second zone (142), and - control the camera (124, 324) to take images of the passenger compartment (102) of the vehicle,- the first lighting mode (210) comprises the emission of light pulses at a first lighting frequency / pulse duration and / or the second lighting mode (230) comprises the emission of the light pulses with a second lighting frequency, - additionally using another camera further comprising the step of: - controlling the other camera to take images of the passenger compartment (102) of the vehicle with a shooting frequency which corresponds at least to the minimum of the first lighting frequency and the second lighting frequency.,

2. The method of claim 1, wherein the control of the camera (124, 324) for taking images is done at a shooting frequency which corresponds at least to the maximum of the first illumination frequency and the second illumination frequency.

3. A method according to claim 1 or 2, wherein the first illumination frequency is at least occasionally different from the second illumination frequency, and preferably the first lighting frequency is 60 Hz and the second lighting frequency is set to a maximum of 30 Hz.

4. Method according to one of the preceding claims, according to which the first lighting mode (210) consists of emitting light pulses with a first pulse duration and / or the second lighting mode (230) consists of emitting light pulses with a second pulse duration.

5. Method according to one of the preceding claims, wherein the first lighting installation (126, 326) and / or the second lighting installation (128, 328) are designed to emit infrared (IR) light.

6. Method according to one of the preceding claims, according to which the first application (361) comprises at least one of the following applications: observation of the driver, tracking of the direction of the driver's gaze and observation of the driver's eyelids.

7. A method according to one of the preceding claims, wherein the second application (360) comprises at least one of the following: presence monitoring, video conferencing, gesture control and data recording.

8. Method according to one of the preceding claims, according to which the first lighting mode (220) and / or the second lighting mode (320) are modified depending on the situation.

9. A method according to any preceding claim, wherein the first area (140) comprises a driver's area of ​​the vehicle and the second area (142) comprises at least a passenger area and a rear seat area.

10. Computing unit (130, 330) designed to perform all steps of a method according to one of the preceding claims.

11. Vehicle interior camera system (120, 320) comprising a camera (124, 324), a first lighting installation (126, 362), a second lighting installation (128, 328) and a computing unit (130, 330) according to claim 12 and preferably a further camera.

12. A computer program which controls the computing unit (130, 330) to execute all steps of a method according to one of claims 1 to 11 when this method is executed by the computing unit (130, 330).

13. A machine-readable storage medium comprising the recording of a computer program according to claim 12.

Images