METHOD FOR IMPROVING THE IMAGE QUALITY OF IMAGES TAKEN INSIDE A VEHICLE

Abstract

A method for improving the image quality of images captured in a vehicle is disclosed. The method comprises at least one first image acquisition device (3) and at least one second image acquisition device (4), the fields of view of which at least partially overlap. The first image acquisition device (3) is configured to acquire first image data in the visible spectrum (RGB), and the second image acquisition device (4) is configured to acquire second image data in the infrared spectrum (IR). In this method, the first image acquisition device (3) acquires first image data from an image sequence with a plurality of first frames within a time interval. Furthermore, the second image acquisition device (4) acquires second image data from an image sequence with a plurality of second frames within the same time interval.The first and second image data are combined to obtain image data (10) for an image sequence which is based on at least a part of the first image data which has been processed based on at least a part of the second image data.

Description

[0001] The present invention relates to a method for improving the image quality of images captured in a vehicle. In particular, image data from different image capture devices are combined for this purpose.

[0002] Vehicles can have different image capture devices whose image data can be used for different purposes. These can be infrared (IR) cameras and RGB cameras, for example. IR cameras are used in particular to monitor the vehicle interior to monitor the driver's attention. An active IR light source and, if necessary, a variable exposure time can be used for this purpose. IR cameras are suitable for ensuring robust detection even in difficult lighting conditions, for example in strong sunlight or even in the dark. Monitoring is increasingly being extended to the entire interior, for which systems consisting of several cameras can be used. A three-dimensional model of the vehicle interior can also be created, particularly using so-called time-of-flight (TOF) cameras. However, colors in the visible light range cannot be detected with an IR camera.

[0003] RGB cameras that capture image data in the visible light range can therefore also be installed in the vehicle interior. These can be used, for example, for video telephony. Combined units are also known that deliver a stream of image data in both the IR and RGB ranges.

[0004] The streams from the various cameras can be combined and processed in central control units. However, the simultaneous operation of cameras in the visible and IR ranges is resource-intensive. Providing high resolutions and frame rates requires high-bandwidth data processing. This may require reducing the resolution or frame rate of individual streams to avoid exceeding the overall system bandwidth. At the same time, functions typically only use either RGB or IR streams, so information from other modalities is lost.

[0005] It is an object of the present invention to provide a method for improving the image quality of images captured in a vehicle. In particular, the solution presented here is intended to enable more efficient use of the acquired image data in a system with RGB and IR image data.

[0006] This object is achieved according to the teaching of the independent claims. Various embodiments and further developments of the invention are the subject of the dependent claims.

[0007] A first aspect of the invention relates to a method, in particular a computer-implemented method, for improving the image quality of images taken in a vehicle. In particular, at least one first image capture device and at least one second image capture device are provided in an interior of the vehicle, the fields of view of which at least partially overlap. The first image capture device is configured to capture first image data in the visible light range, and the second image capture device is configured to capture second image data in the infrared light range. In the method, first image data of an image sequence having a plurality of first frames in a time interval are captured by means of the first image capture device. Furthermore, second image data of an image sequence having a plurality of second frames in the time interval are captured by means of the second image capture device.The first and second image data are combined to obtain image data for an image sequence based on at least a portion of the first image data that has been processed based on at least a portion of the second image data.

[0008] The aforementioned method according to the first aspect is therefore based in particular on combining image data from the visible range of light (RGB range) and image data from the infrared range of light (IR range), in particular to improve the image quality of a resulting RGB image (or an RGB image sequence). In other words, the streams of an RGB camera and an IR camera (i.e., the frames from the same time interval) are combined, in particular to utilize differences in the image data to improve image quality. Thus, the advantages of one image modality can be utilized for the other image modality without any loss of information.

[0009] In this way, for example, a video stream for a video call can be provided with increased quality without having to increase the quality (e.g., in terms of resolution, frame rate, exposure, noise, etc.) of the RGB camera itself, which might not be feasible due to limited bandwidth in the vehicle system, for example. The method uses the frames of the IR image data in particular, whereby new image data of higher quality can be calculated based on (multiple) infrared and RGB frames.

[0010] The term “vehicle” as used herein refers in particular to a passenger car, including all types of motor vehicles, hybrid and battery-powered electric vehicles, as well as vehicles such as sedans, vans, buses, trucks, delivery vans and the like.

[0011] The term "image capture device" used here refers in particular to a camera, in particular a digital camera. The camera can capture still images (photos) or moving images (videos), in particular in the visible light range (RGB). An image capture device can capture or record such images and output corresponding image data. An image sequence consisting of several frames represents a moving image, in particular a video. The continuous capture of image data, in particular of consecutive frames, forms a stream. A frame rate is the number of frames in a specific time interval. The frame rate can be specified, for example, in fps ("frames per second"). Typical frame rates can, for example, be in the range from 20 fps to 60 fps.

[0012] An image capture device configured to capture image data in the IR range can also be referred to as a "monitoring device," so the following definition can be applied. The monitoring device can comprise an interior camera, which is already used, for example, to monitor the driver, e.g., for fatigue detection. Such a driver monitoring system (DMS) is a vehicle safety system that evaluates the driver's attention. The monitoring device can comprise an image capture device (a camera). An infrared (IR) camera (e.g., a near-infrared (NIR) camera) can be used. An IR image is well suited for monitoring because it is robust against changing light conditions, for example, even in strong sunlight. Recordings are also possible in the dark. It goes without saying that a suitable IR light source can be present.It is also conceivable to create a three-dimensional model of the driver or the vehicle interior, for example, using a time-of-flight (TOF) camera. Using this three-dimensional model, distances to the camera, and thus distances between objects and their movements, can be calculated.

[0013] The term "modality" used here refers in particular to the type of image capture device, particularly with regard to the range of electromagnetic waves for which corresponding image data is captured. One modality can, for example, refer to the visible range of light ("RGB"). Another modality, however, can refer to the infrared range of light ("IR" or "NIR"). It is understood that the image capture devices (or "cameras") can each have a corresponding image sensor and, if appropriate, also a corresponding light source (particularly for IR).

[0014] The term "field of view" used here refers in particular to an area that is visible from the viewing angle of the image capture device, i.e., an image area of ​​a corresponding image sensor of the image capture device. For example, with respect to an RGB image, the field of view substantially corresponds to an image of the captured image data that is visible to a viewer. In the context of the present invention, the fields of view of an RGB image capture device and an IR image capture device are intended to at least partially overlap. It is understood that the image quality can only be improved for the overlap area. Therefore, the fields of view are preferably identical (congruent) or at least substantially identical.This can be achieved, for example, by an RGB-IR camera module, in which the two modalities are combined in one unit (so that the two sensors are located close to each other, allowing them to have a nearly identical field of view).

[0015] As used herein, the terms "comprises," "includes," "includes," "has," "has," "with," or any other variation thereof are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or has a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or that are inherent in such a method or apparatus.

[0016] Furthermore, unless explicitly stated to the contrary, "or" refers to an inclusive "or" and not an exclusive "or." For example, a condition A or B is satisfied by one of the following conditions: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

[0017] The terms "a" or "an" as used herein are defined as "one or more." The terms "another" and "another," and any other variations thereof, are defined as "at least one other."

[0018] The term “plurality” as used here shall mean “two or more”.

[0019] The term “configured” or “set up” to fulfil a specific function (and respective modifications thereof) is to be understood within the meaning of the invention that the corresponding device is already in a design or setting in which it can carry out the function or is at least adjustable - i.e. configurable - so that it can carry out the function after being set accordingly. The configuration can be carried out, for example, by appropriately setting parameters of a process sequence or of switches or the like for activating or deactivating functionalities or settings. In particular, the device can have a plurality of predetermined configurations or operating modes, so that the configuration can be carried out by selecting one of these configurations or operating modes.

[0020] Preferred embodiments of the method according to the first aspect are described below, which can each be combined with each other as well as with the other aspects of the invention described, unless this is expressly excluded or is technically impossible.

[0021] In some embodiments, combining the first and second image data comprises adjusting at least one image parameter of the first image data based on the second image data. The image parameter can comprise at least one of brightness, contrast, and sharpness. In this way, information from the infrared range can be used to optimize, for example, poorly illuminated images in the RGB range. In particular, this can be carried out by applying a generative model, for example a diffusion model or GAN model. The model receives, for example, a dark, overexposed, or noisy RGB output image and one or more infrared images in which the vehicle interior can be better recognized. The model can derive a uniformly illuminated version of the RGB output image from this.

[0022] In some embodiments, a number of first frames (i.e., in the RGB range) in the time interval is smaller than a number of second frames (i.e., in the IR range) in the time interval. Combining the first and second image data may comprise adding additional frames to the image sequence of the first frames, wherein the additional frames are calculated from the second image data. In other words, an effective frame rate in one stream (in particular the RGB stream) may be increased by transferring frames from other streams (in particular an IR stream) that arrive in the meantime. The transfer from one stream to the other is carried out via a generative model that is trained to transform IR and RGB images with different lighting conditions into one another.

[0023] In some embodiments, a resolution of the captured first image data (i.e. in the RGB range) is higher than a resolution of the captured second image data (i.e. in the IR range). In this case, before the first and second image data are combined, at least one new frame with a higher resolution can be calculated from at least two of the second frames, wherein the second image data comprise the at least one new frame when the first and second image data are combined. In this way, an increase in the image resolution can be achieved by combining a plurality of frames. In particular, the information from a plurality of frames of the same scene recorded in quick succession can be used to calculate an image with a higher resolution than the original images.The calculation can also be performed using a generative model trained to predict the high-resolution originals from compressed image data, thereby generating higher-resolution image data from low-resolution image data. Using multiple frames provides the model with additional information to support the prediction. In particular, the initially lower resolution of the IR image data can be adapted to the higher resolution of the RGB image data, and the IR image data can then be used to improve the image quality of the RGB image data (e.g., to remove noise, improve exposure, and the like, see above).

[0024] A second aspect of the invention relates to a data processing system comprising at least one processor configured to execute the method according to the first aspect. The system may, in particular, comprise a first image capture device and a second image capture device configured, as described above, to capture image data in the visible range of light (RGB range) and in the infrared range of light (IR range), respectively.

[0025] A third aspect of the invention relates to a computer program comprising instructions which, when executed on a system according to the second aspect, cause the system to carry out the method according to the first aspect.

[0026] The computer program can, in particular, be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloadable via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. Furthermore, the computer program can have a plurality of interacting individual program modules.

[0027] The system according to the second aspect can accordingly comprise a program memory in which the computer program is stored. Alternatively, the system can also be configured to access an external computer program, for example, available on one or more servers or other data processing units, via a communication connection, in particular to exchange data with it that is used during the execution of the method or computer program or that represents outputs of the computer program.

[0028] The features and advantages explained with respect to the first aspect of the invention also apply accordingly to the further aspects of the invention.

[0029] Further advantages, features and possible applications of the present invention will become apparent from the following detailed description in conjunction with the drawings.

[0030] It shows: Fig. 1 is a schematic view of a vehicle interior with a first and a second image capture device; and Fig. 2 a flowchart of a method according to an embodiment.

[0031] Throughout the figures, the same reference numerals are used for the same or corresponding elements of the invention.

[0032] In Fig. 1 schematically shows a vehicle interior 1, while in Fig. 2 schematically illustrates a method according to one embodiment. In the vehicle interior 1, here in the base 6 of the interior mirror 5, an RGB-IR camera module 2 is installed, which can capture infrared and RGB images ("frames") of the interior 1, in particular of the driver 11. For this purpose, a first image capture device in the form of an RGB camera 3 and a second image capture device in the form of an IR camera 4 are provided in the camera module 2. Due to their joint arrangement in the camera module 2, the fields of view of the RGB camera 3 and the IR camera 4 are largely identical or at least largely overlap.

[0033] The RGB camera 3 captures images (frames) in the visible range of light (RGB), while the IR camera 4 captures images (frames) in the infrared range of light (IR). This is also Fig. 2. The recorded frames are transmitted to a control unit 13 for further processing, in particular to improve image quality. The frames can arrive at the control unit 13 alternately and can also be used individually by driver monitoring (IR) and video telephony (RGB) functions. An image enhancement module 12 is implemented in the control unit 13. In particular, an image is to be enhanced for a video call, as explained below.

[0034] In this embodiment, a frame rate of the IR stream is greater than a frame rate of the RGB stream (illustrated in the camera module 2 in Fig.2). In contrast, the resolution of the RGB frames can be higher than that of the IR frames. The additional frames from the IR stream are now to be used to improve the image quality of the RGB stream. First, an improved IR frame 8 is generated from the IR frames in a generative model 7, such as a GAN model, which has a higher resolution than the IR frames captured by the IR camera 4. In particular, the resolution of the IR frames can be increased so that it corresponds to the resolution of the RGB frames. This improved IR image 8 is then combined with an RGB image by another generative model 9, such as a GAN model, to predict a higher-resolution, better-illuminated RGB image 10. This process can run continuously, which can, for example, achieve a better user experience in a video call.

[0035] While at least one exemplary embodiment has been described above, it should be appreciated that a wide variety of variations exist. It should also be understood that the described exemplary embodiments are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide a guide to implementing at least one exemplary embodiment, with the understanding that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from the subject matter as defined in the appended claims, as well as their legal equivalents. LIST OF REFERENCE SYMBOLS RGB RGB frames IR IR frames 1 vehicle interior 2 RGB IR camera modules 3 RGB camera 4 IR cameras 5 interior mirrors 6 feet of the interior mirror 7 Model 8 Improved IR image 9 Model 10 Improved RGB image 11 drivers 12 Image enhancement module 13 Control unit

Claims

[1] Method for improving the image quality of images taken in a vehicle, wherein at least one first image capture device (3) and at least one second image capture device (4) are provided, the fields of view of which at least partially overlap, wherein the first image capture device (3) is configured to capture first image data in the visible range of light (RGB), and the second image capture device (4) is configured to capture second image data in the infrared range of light (IR), the method comprising: - capturing, by means of the first image capturing device (3), first image data of an image sequence having a plurality of first frames in a time interval; - capturing, by means of the second image capturing device (4), second image data of an image sequence having a plurality of second frames in the time interval; and - Combining the first and second image data to obtain image data (10) for an image sequence based on at least a portion of the first image data which has been processed based on at least a portion of the second image data. [2] The method of claim 1, wherein combining the first and second image data comprises adjusting at least one image parameter of the first image data based on the second image data. [3] The method of claim 2, wherein the image parameter comprises at least one of brightness, contrast and sharpness. [4] Method according to one of the preceding claims, wherein a number of the first frames (RGB) in the time interval is smaller than a number of the second frames (IR) in the time interval. [5] The method of claim 4, wherein combining the first and second image data comprises adding additional frames to the image sequence of the first frames (RGB), the additional frames being calculated from the second image data. [6] Method according to one of the preceding claims, wherein a resolution of the acquired first image data is higher than a resolution of the acquired second image data. [7] Method according to claim 6, wherein before combining the first and second image data from at least two frames of the second frames (IR), at least one new frame (8) is calculated with a higher resolution, wherein the second image data comprises the at least one new frame (8) when combining the first and second image data. [8] A data processing system comprising at least one processor (13) configured to carry out the method according to any one of the preceding claims. [9] System according to claim 8, further comprising at least one first image capture device (3) and at least one second image capture device (4), whose fields of view at least partially overlap, wherein the first image capture device (3) is configured to capture first image data in the visible range of light (RGB), and the second image capture device (4) is configured to capture second image data in the infrared range of light (IR). [10] A computer program comprising instructions which, when executed on a system according to claim 8 or 9, cause the system to carry out the method according to any one of claims 1 to 7.

Images