Method and device for carrying out a colour calibration of an interior camera of a vehicle

Abstract

In a method for carrying out a colour calibration of an interior camera (102) of a vehicle (104), at least one image is captured by the interior camera (102). The image comprises a representation of at least one reference element (108, 110, 112) arranged in the vehicle interior (106). The reference element (108, 110, 112) actively emits light of a known colour or is illuminated with light of a known colour by at least one light source (114) arranged in the vehicle interior (106). Image data corresponding to the image are generated by the interior camera (102). The image data are received and processed by a processing unit (124) in order to determine colour information relating to the reference element (108, 110, 112). Furthermore, a colour calibration of the interior camera (102) is carried out by a processing unit (124) taking into account the known colour and the determined colour information.

Description

[0001] Method and apparatus for performing a color calibration of an interior camera of a vehicle

[0002] The invention relates to a method for performing a color calibration of an interior camera of a vehicle. The invention further relates to a device for performing a color calibration of an interior camera of a vehicle.

[0003] Interior cameras in vehicles enable a variety of functions, from monitoring the driver's attention to detecting other occupants and objects inside the vehicle. However, the effectiveness of these cameras depends significantly on their ability to deliver precise and reliable image information under varying lighting conditions. For the purposes of this document, "vehicles" refers specifically to motor vehicles such as passenger cars and trucks.

[0004] German patent DE 10 2022 001 562 A discloses a method for the dynamic calibration of a camera. In this method, standardized objects, materials, and component shapes arranged in a vehicle interior are used as a reference for dynamic calibration to perform a white balance.

[0005] From DE 10 2022 119 259 A1, a method is known in which white balance and color correction are performed for a vehicle interior camera. In this method, cabin features of the vehicle interior are used, which fulfill the function of a predefined color chart, for example, the colors of a vehicle seat cover.

[0006] German patent application DE 10 2018 220 597 A1 discloses a method for the self-calibration of an optical sensor of a vehicle. The method uses predefined calibration points inside or outside the vehicle. The calibration points should be as invisible as possible to an occupant. For example, the calibration points are infrared markers.

[0007] The object of the invention is to provide a method and a device for performing a color calibration of an interior camera of a vehicle, which are improved compared to the known prior art.

[0008] This problem is solved by a method having the features of claim 1 and by a device having the features of the independent device claim. Advantageous embodiments are specified in the dependent claims.

[0009] In the proposed method for color-calibrating a vehicle's interior camera, the camera captures at least one image. This image depicts at least one reference element located in the vehicle's interior. The reference element actively emits light of a known color or is illuminated by at least one light source of a known color located in the vehicle's interior. The interior camera generates image data corresponding to the image. A processing unit receives and processes this image data to determine color information related to the reference element. Furthermore, a processing unit performs a color calibration of the interior camera, taking into account the known color and the determined color information.

[0010] The proposed method ensures that images captured by the interior camera exhibit high color fidelity and accuracy, regardless of varying lighting conditions inside the vehicle. The interior camera can thus capture precise image information independently of external influences such as changing daylight or artificial lighting. To enable more accurate color calibration than previously possible, the proposed method uses light of a known color generated by a vehicle-integrated source. This vehicle-integrated source can be the reference element itself, for example, an LED element located within the vehicle interior and within the detection range of the interior camera.If the light source emitting light of the known color is not within the detection range of the interior camera, an element of the vehicle interior illuminated by the light source, particularly a surface, can serve as the reference element. For example, the light source could be indirect lighting of the vehicle's headliner. The light source itself is not visible in an image captured by the camera, but the illuminated headliner is, forming the reference element in such an embodiment. While materials can, for example, fade and thus change color, the color of the light emitted by the reference element or the light source does not change, even over extended periods.This makes using light of a known color as a reference advantageous compared to using non-self-illuminating materials, such as a color chart placed in the vehicle interior.

[0011] The method can use two or more reference elements. Each reference element either actively emits light of the known color or is illuminated by at least one light source of a known color located in the vehicle interior. The light of the known color preferably comprises light from the optical wavelength range.

[0012] In one embodiment, the reference element is positioned in a location within the vehicle interior that is not visible to a vehicle occupant seated in a vehicle seat. Preferably, the reference element is positioned in a location within the vehicle interior that is not visible to a driver seated in the driver's seat. For example, the reference element can be located on the side of the steering wheel facing away from the driver. Since the interior camera is directed towards the vehicle occupants, locations that are not visible to the occupants are often clearly visible to the interior camera. Furthermore, this arrangement reduces the risk of the reference element being obscured by an occupant. The method is therefore particularly reliable.Furthermore, such a reference element does not disturb the overall aesthetic impression of the vehicle interior, which makes this arrangement particularly attractive for vehicle manufacturers.

[0013] In another embodiment, the reference element is a lighting element in the vehicle interior that actively emits light of the known color. The reference element could, for example, be a light source located in a side door, on the center console, or in the vehicle's headliner. Such light sources are often already installed in modern vehicles and have a known color characteristic. These light sources can therefore serve as the reference element particularly easily, without requiring complex modifications to the vehicle interior. The process is thus particularly easy to implement.

[0014] In another embodiment, the reference element is a vehicle display element that actively emits light of the known color. This display element could be, for example, an instrument cluster, a component of an instrument cluster such as the speedometer or tachometer, an infotainment display, a central information display located in the center console, a warning light in the cockpit, or another display in the vehicle interior. Display elements are often already installed at the factory and have known color characteristics. Just like the lighting element, these can therefore easily serve as the reference element, making the process particularly simple to implement. In yet another embodiment, the reference element is a vehicle display. The display can show a reference pattern to actively emit light of the known color.The image includes a representation of the reference pattern. In particular, the processing unit controls the display to show the reference pattern. The reference pattern shown by the display includes at least one element with the known color, so the display emits light of the known color. For example, the display can show a rectangle in the known color as the reference pattern.

[0015] In another embodiment, an image of an occupant is captured using the interior camera, which includes a picture of at least one vehicle occupant. The interior camera generates occupant image data corresponding to this image. Based on this occupant image data, the processing unit determines whether the vehicle occupant's gaze is currently directed at the reference element. The reference element only actively emits light of a known color if it has been determined that the vehicle occupant's gaze is not currently directed at the reference element. Preferably, an occupant image is captured that includes a picture of the vehicle's driver. In such an embodiment, it is therefore determined whether the driver's gaze is currently directed at the reference element.In this embodiment, the reference element is only used for color calibration of the interior camera when the vehicle occupant is not currently looking at it. This prevents distraction of the vehicle occupant by the color calibration of the interior camera and increases road safety. Preferably, in the present embodiment, the reference element is a display element of the vehicle. For example, a vehicle display is only activated by the processing unit to display the reference pattern when the driver is not looking at it. Display elements often perform important functions in a vehicle cockpit, for example, they show the driver important information. By using the display element for color calibration of the interior camera only when the vehicle occupant is not currently looking at it, distraction is further reduced.Furthermore, no important information is withheld from the vehicle occupant.

[0016] In another embodiment, the reference element is illuminated by the internal light source with light of a known color. The reference element can have a reference pattern that includes at least one element with another known color. The color calibration of the interior camera can be performed taking this other known color into account. The reference element is, in particular, a surface of the vehicle interior, for example, the headliner or a seat surface. The reference pattern is preferably a surface colored in the other known color. The light source is, for example, direct or indirect lighting of the vehicle interior that emits light of a predefinable color. Such light sources and surfaces with reference patterns are already factory-installed in most vehicles.This means the procedure requires no changes to the vehicle interior and is therefore particularly easy to carry out.

[0017] In another embodiment, the interior camera or an additional optical sensor is used to determine the color of light from at least one external light source located outside the vehicle. The interior camera's color calibration can then be performed taking the color of the external light source into account. External light sources, such as artificial light sources like streetlights or the sun as a natural light source, can distort the perceived color of the reference element. By considering the color of the external light source during the interior camera's color calibration, the calibration can be performed even more accurately. For example, the color of the external light source is subtracted from the known color to achieve a more precise color calibration of the interior camera.In another embodiment, white balance and / or color correction of the interior camera is performed as color calibration. Precise white balance and careful color correction of the interior camera are essential to minimize color distortions in images captured by the interior camera, which are caused by varying lighting conditions inside the vehicle. These measures ensure accurate color reproduction and improve, for example, the reliability of image-based driver assistance systems, as well as the quality of functions in which the image from the interior camera is offered to the occupants or other persons (e.g., video telephony).

[0018] The invention further relates to a device for performing a color calibration of a vehicle's interior camera. The device comprises at least one reference element arranged in the vehicle interior. The reference element actively emits light of a known color. Alternatively, the reference element is illuminated by at least one light source arranged in the vehicle interior with light of a known color. The device also comprises the interior camera, which is arranged and configured to capture at least one image that includes a representation of the reference element and to generate image data corresponding to the image. Furthermore, the device comprises a processing unit configured to receive and process the image data.The processing unit is further designed to determine color information related to the reference element based on the image data, and to perform a color calibration of the indoor camera taking into account the known color and the determined color information.

[0019] The device can, in particular, comprise two or more reference elements, each of which either actively emits light of a known color or is illuminated by at least one light source arranged in the vehicle interior with light of a known color. The device has the same advantages as the claimed method. In particular, the device can be further developed with the features of the dependent claims directed to the method. Furthermore, the method described above can be further developed with the features described in this document in connection with the device.

[0020] Exemplary embodiments of the invention are explained in more detail below with reference to the figures. These show:

[0021] Figure 1 shows a schematic representation of a device for performing a color calibration of an interior camera of a vehicle according to an exemplary embodiment; and

[0022] Figure 2 shows a flowchart of a method for performing a color calibration of an interior camera of a vehicle according to an exemplary embodiment.

[0023] Figure 1 shows a schematic representation of a device 100 for performing a color calibration of an interior camera 102 of a vehicle 104.

[0024] The interior camera 102 of the vehicle 104 is arranged and oriented such that it can capture an image of the vehicle interior 106. The portion of the vehicle interior 106 captured by the interior camera 102 is hereinafter also referred to as the detection range of the interior camera 102. From the captured image, the interior camera 102 generates image data that can be further processed, for example, by driver assistance systems, safety monitoring functions, or for analyzing driver behavior. In the illustrated embodiment, the interior camera 102 is permanently installed in the vehicle interior 106 and is designed to capture the image as a two-dimensional image in the optical spectrum. The interior camera 102 can also be designed to capture light in the infrared spectrum.Capturing light in the infrared spectrum has the additional advantage that the indoor camera 102 can also capture images in poor lighting conditions, for example at night.

[0025] The device 100 serves, for example, to perform a white balance of the interior camera 102 in order to ensure a uniform representation of colors in images captured by the interior camera 102, even under different lighting conditions in the vehicle interior 106. As a basis for the color calibration, the device 100 comprises at least one reference element 108, 110, 112, which either emits light of a known color itself or is illuminated by a light source 114 that emits light of a known color. The self-illuminating reference elements 108, 110 can, for example, be decorative elements already present in the vehicle interior 106 or elements specifically installed for the color calibration of the interior camera 102.Reference elements 108, 110, 112, in particular those expressly installed for color calibration, can be arranged so that they are not visible to a vehicle occupant 116 sitting in a vehicle seat 118. As a result, these reference elements 108, 110, 112 do not significantly alter the overall aesthetic impression of the vehicle interior 106. The non-self-illuminating reference elements 112 are, for example, surfaces in the vehicle 104 that are illuminated by a light source 114, which is part of the vehicle 104's interior lighting.

[0026] Figure 1 shows three reference elements 108, 110, and 112 by way of example. A first reference element 108 is an LED element arranged on the interior trim of a side door of the vehicle 104 such that it lies within the detection range of the interior camera 102. The first reference element 108 is controllable and configured to emit light of a first known color, for example, blue. The first reference element 108 can be part of the interior lighting of the vehicle 104 and is arranged outside the field of vision of the vehicle occupant 116 when the occupant is seated in the vehicle seat 118 and facing forward.

[0027] A second reference element 110 is a display element of the vehicle 104, located within the detection range of the interior camera 102. In Figure 2, the second reference element 110 is shown, purely by way of example, as a display in the cockpit of the vehicle 104, for instance, a graphic instrument cluster or a central information display. The second reference element 110 can be controlled to output graphic information. In particular, the second reference element 110 can be controlled to output a predefined reference pattern and thus emit light with a second known color.

[0028] In the embodiment shown in Figure 1, a third reference element 112 is formed by a portion of the vehicle headliner 120, which is illuminated by a light source 114 that is part of the interior lighting of the vehicle 104. The third reference element 112 is located within the detection range of the interior camera 102 and has a known reference pattern that includes at least one other known color. For example, the portion of the vehicle headliner 120 is uniformly colored in the other known color. The light source 114 illuminates the third reference element 112 with light of a third known color.

[0029] In the illustrated embodiment, the device 100 comprises a further optical sensor 122, which is configured, purely by way of example, as an external camera of the vehicle 104. The optical sensor 122 is configured to detect an area outside the vehicle 104, for example, the roadway directly in front of the vehicle 104, an area to the side of the vehicle 104, and / or an area behind the vehicle 104. The optical sensor 122 generates sensor data that can be further processed, for example, by a parking assistant. The device 100 also comprises a processing unit 124, which is shown in Figure 1, purely by way of example, as a part of the vehicle 104. However, the processing unit 124 of the device 100 can also be a processing unit arranged wholly or partially remote from the vehicle 104.For example, individual functions of the processing unit 124 can be performed by units located remotely from the vehicle 104, such as servers or cloud services. The processing unit 124 is configured to receive and process the image data from the interior camera 102 and to control the first reference element 108, the second reference element 110, and the light source 114. The processing unit 124 is further configured to perform a method for color calibration of the interior camera 102, which is described in more detail below with reference to Figure 2.

[0030] Figure 2 shows a flow chart of the procedure for performing the color calibration of the interior camera 102 of the vehicle 104 according to an exemplary embodiment.

[0031] The process is started in step S200. In the optional step S202, the processing unit 124 controls the controllable reference elements 108 and 110 to emit light of the known color. In step S202, the processing unit 124 can also control the light source 114 to illuminate the non-self-illuminating reference elements 112. In the embodiment of the device 100 shown in Figure 1, the processing unit 124, for example, controls the first reference element 108 and the second reference element 110. The second reference element 110 is controlled in particular to display the reference pattern. The processing unit 124 also controls the light source 114 to illuminate the third reference element 112 with light of the third known color. Controlling the second reference element 110 in particular, i.e.,The display calibration can only be performed if it has been determined that the gaze of the vehicle occupant 116 is not directed at the respective reference element 108, 110, 112. This prevents the vehicle occupant 116 from being distracted by the color calibration.

[0032] In step S204, the processing unit 124 controls the interior camera 102 to capture an image of the vehicle interior 106. The image includes the vehicle interior 106 and the reference elements 108, 110, 112 arranged within the vehicle interior 106. The interior camera 102 generates corresponding image data and transmits it to the processing unit 124. In the optional step S206, the processing unit 124 controls the optical sensor 122 to detect the area outside the vehicle 104. The optical sensor 122 detects, in particular, an external light source, such as street lighting. The optical sensor 122 generates corresponding sensor data and transmits it to the processing unit 124. Steps S204 and S206 can be performed simultaneously or in any sequence.

[0033] In step S208, the processing unit 124 determines color information for each of the reference elements 108, 110, and 112 based on the image data. For example, the processing unit 124 determines a color value for each of the reference elements 108, 110, and 112, i.e., the color in which the respective reference element 108, 110, and 112 appears in the image. If the sensor data was transmitted in step S206, the processing unit 124 determines the light color of the external light source in the optional step S210. This can be done, for example, based on a database stored on a memory element of the processing unit 124. Many artificial and natural light sources have a known light color that can be queried from the database.

[0034] In step S212, the processing unit 124 then performs the color calibration of the indoor camera 102 based on the previously determined color information. For this purpose, the processing unit 124 compares, for example, the determined color values ​​of the reference elements 108, 110, and 112 with the known colors emitted by the respective reference elements 108 and 110, or with which the reference elements 112 are illuminated. From such a comparison, the processing unit 124 can, for example, generate a filter with which the image data of the indoor camera 102 can be color-corrected or with which a white balance of the image data of the indoor camera 102 can be performed. For the non-self-illuminating reference elements 112, the processing unit 124 also takes into account the other known color, i.e., the color of the illuminated reference element 112.The expected color of the reference element 112 thus results from the known color of light with which the reference element 112 is illuminated, and the color of the reference element 112 itself.

[0035] Once the color temperature of the external light source has been determined in step S210, processing unit 124 performs the color calibration of the internal camera 102 taking this into account. For example, processing unit 124 subtracts the color temperature of the external light source from the determined color values ​​to correct them with respect to the external light source. This is particularly important for the non-self-illuminating reference elements 112, since the expected color of the reference element 112 also depends on the color temperature of the external light source.

[0036] The procedure is then terminated in step S214.

[0037] In the embodiment described with reference to Figures 1 and 2, at least one of the reference elements 108, 110, 112, the interior camera 102, and the processing unit form the device 100 for performing a color calibration of an interior camera 102 of a vehicle 104. Further elements and features shown in Figures 1 and 2 and mentioned in the preceding description may be part of the device 100. Likewise, process steps described with reference to the device 100 may be part of the claimed method.

[0038] List of reference symbols

[0039] 100 device

[0040] 102 Interior camera

[0041] 104 vehicles

[0042] 106 Vehicle interior

[0043] 108, 110, 112 Reference element

[0044] 114 Light source

[0045] 116 vehicle occupants

[0046] 118 vehicle seats

[0047] 120 vehicle headliners

[0048] 122 Optical Sensor

[0049] 124 processing units

Claims

Claims 1. A method for performing a color calibration of an interior camera (102) of a vehicle (104), wherein the interior camera (102) captures at least one image comprising a representation of at least one reference element (108, 110, 112) arranged in the vehicle interior (106), the reference element (108, 110, 112) actively emits light of a known color or is illuminated by at least one light source (114) arranged in the vehicle interior (106) with light of a known color; image data corresponding to the image are generated by the interior camera (102); the image data are received and processed by a processing unit (124) to determine color information related to the reference element (108, 110, 112); and a color calibration of the indoor camera (102) is carried out by a processing unit (124) taking into account the known color and the determined color information.

2. Method according to claim 1, wherein the reference element (108) is arranged at a location within the vehicle interior (106) which is not visible to a vehicle occupant (116) sitting in a vehicle seat (118).

3. Method according to claim 1 or 2, wherein the reference element (108) is a lighting element of the vehicle interior (106) that actively emits light of the known color.

4. Method according to one of the preceding claims, wherein the reference element (110) is a display element of the vehicle (104) that actively emits light of the known color.

5. Method according to any of the preceding claims, wherein the reference element (110) is a display of the vehicle (104); wherein the display shows a reference pattern to actively emit the light with the known color; and wherein the image comprises a representation of the reference pattern.

6. A method according to any of the preceding claims, wherein an occupant image is captured using the interior camera (102), which includes an image of at least one vehicle occupant (116); wherein the interior camera (102) generates occupant image data corresponding to the occupant image; wherein the processing unit (124) determines, based on the occupant image data, whether the gaze of the vehicle occupant (116) is currently directed towards the reference element (108, 110); and wherein the reference element (108, 110) only actively emits light of a known color when it has been determined that the gaze of the vehicle occupant (116) is currently not directed towards the reference element (108, 110).

7. A method according to any of the preceding claims, wherein the reference element (112) is illuminated by the internal light source (114) with light of the known color; wherein the reference element (112) has a reference pattern comprising at least one element of a further known color; and wherein the color calibration of the internal camera (102) is carried out taking additional account of the other known color.

8. A method according to any of the preceding claims, wherein the color of light of at least one external light source located outside the vehicle (104) is determined using the interior camera (102) or a further optical sensor (122); and wherein the color calibration of the interior camera (102) is carried out taking into account the color of light of the external light source.

9. Method according to one of the preceding claims, wherein the color calibration is performed by white balance and / or color correction of the internal camera (102).

10. Device (100) for performing a color calibration of an interior camera (102) of a vehicle (104), comprising at least one reference element (108, 110, 112) arranged in the vehicle interior (106), wherein the reference element (108, 110, 112) actively emits light of a known color or is illuminated by at least one light source (114) arranged in the vehicle interior (106) with light of a known color; the interior camera (102), which is arranged and configured to capture at least one image comprising a representation of the reference element (108, 110, 112) and to generate image data corresponding to the image; and a processing unit (124), which is configured to receive and process the image data, wherein the processing unit (124) further is trained to make a decision based on the image data. to determine the color information related to the reference element (108, 110, 112) and taking into account the known color and the determined To perform a color calibration of the interior camera (102) using color information.

Images