Image pickup system and operating method thereof

Abstract

The invention relates to a camera system (1) comprising a camera module (2) for generating photos of a scene (3) and a computing unit (4) for controlling the camera module (2) and processing the photos. The invention is characterized by an infrared light source (7), in which the computing unit (4) is further adapted to:-control the camera module (2) to expose the image sensor (5) in at least one first acquisition process (EV1) with a first exposure time, in which a first RGB image (RGB) is generated by means of the RGB exposure device (6.1); controlling the infrared light source (7) in order to additionally illuminate the scene (3) with infrared light in at least one second acquisition process (EV2) and simultaneously controlling the camera module (2) in order to expose the image sensor (5) in the second acquisition process (EV2) with a second exposure time different from the first exposure time, wherein a second RGB image (RGB) is generated by means of the RGB exposure device (6.1) and an IR image (IR) is generated by means of the IR exposure device (6.2); and fusing the at least one first RGB image (RGB) and the at least one second RGB image (RGB) into an HDR image (HDR).

Description

Technical Field

[0001] The invention relates to a camera system according to the type defined in more detail in the preamble of claim 1 and to a method for operating such a camera system. Background Art

[0002] As is known, camera systems include analog and digital camera systems. To generate digital photos, the corresponding camera systems usually use image sensors, such as CMOS sensors. Such image sensors include a large number of pixels arranged in a two-dimensional grid, which output electronic signals when exposed. To generate color photos, color filters (for example, red, green and blue filters) are placed in front of the pixels. The filters are arranged in a regular pattern. The filters can be, for example, Bayer filters or interference filters. By setting suitable components, individual pixels can also be sensitive to infrared light. This enables the image sensor to capture color images in the visible spectrum as well as images in the infrared spectrum.

[0003] If necessary, bracketed exposures are usually generated from a sequence of color images and infrared images. In a camera system with an image acquisition rate (frame rate) of 60 frames per second, if color images and infrared images are recorded alternately, color images and infrared images can be recorded at an acquisition rate of 30 frames per second, respectively.

[0004] In order to increase the dynamic range, you can also record exposures with different exposure times in bracketing, so that you can generate high dynamic range images, the so-called High-Dynamik-Range (HDR). For example, using a camera with a sampling rate of 60 frames per second, three consecutive exposures with short, medium and long exposure times can generate an HDR video with a frequency of 20 frames per second.

[0005] The simultaneous generation of infrared and HDR images reduces the frame rate of the respective HDR and infrared video streams generated by the camera system, since the camera must continuously record color images with different exposure times and additionally the infrared image. If the frame rate of the video stream is too low, for example below 30 frames per second, the corresponding video will be perceived as "shaky" or "unsmooth", which impairs the viewing comfort of the user. Motion artifacts can also occur due to long exposure times or large time differences between the images in the bracketed exposures. If a person moves while recording the respective bracketed exposures, this will appear as a "blur" in the video stream.

[0006] Camera systems are also used in vehicles to provide comfort functions and safety functions. For example, the interior camera can be used for driver status monitoring or seat occupancy detection. Vehicle occupants can also use the camera system to take selfies or make video calls. Infrared images are recorded in particular in order to provide safety-related functions such as driver status monitoring and to ensure adequate image quality even in poor lighting conditions, for example in the dark. In this case, the vehicle interior can be actively illuminated with an infrared light source while the corresponding infrared image is being recorded. For this purpose, the camera system and the infrared light source are synchronized.

[0007] In order to reduce costs and use the available installation space in the vehicle as efficiently as possible, as few cameras as possible, preferably just a single camera, should be used. At the same time, photos or videos must be recorded in the visible and infrared spectrum with the highest possible dynamic range, high image quality and high frame rates.

[0008] In order to increase the frame rate when generating HDR videos, a corresponding vehicle camera system is known from DE 10 2018 112 345 A1. The vehicle camera system includes a circular buffer, into which the photos generated by the camera system are fed. The circular buffer has a certain size, so that the photos generated in the past are overwritten by the photos currently generated. Here, the HDR image is generated from the exposure group. Such an exposure group includes, for example, three separate exposures, each with different exposure times, such as short, medium and long exposure times. The vehicle camera system disclosed in this document is able to generate an HDR image after each recording of a separate exposure, without having to first shoot a complete exposure group. In this way, the currently recorded separate exposure will overwrite the separate exposure with the same exposure time in the exposure group used to generate the HDR image. In the case of three different exposure times, each separate exposure is used to generate three HDR images. Summary of the invention

[0009] The object of the present invention is to propose an improved camera system and method for operating the same, by means of which photos or videos can be recorded in the visible spectrum and in the infrared spectrum simultaneously and a high frame rate can be achieved for both color images and infrared images despite the relatively high dynamic range of the color images.

[0010] This object is achieved according to the invention by a camera system having the features of claim 1 and a method for operating the camera system having the features of claim 13. Advantageous embodiments and developments are given in the dependent claims.

[0011] A camera system of the type mentioned at the outset has a camera module for generating photos of a scene and a computing unit for controlling the camera module and processing the photos, wherein the camera module has an image sensor, which is adapted to collect light in the visible spectrum by means of an RGB exposure device and to collect light in the infrared spectrum by means of an IR exposure device, the camera module is adapted to record at least two single-frame images of the scene by means of the exposure of the image sensor in at least two acquisition processes with different exposure times, and the computing unit is adapted to fuse the at least two single-frame images into an HDR image. According to the invention, the camera system is improved by providing an infrared light source, wherein the computing unit is further adapted according to the invention to:

[0012] - controlling the camera module to expose the image sensor with a first exposure time in at least one first acquisition process, wherein a first RGB image is generated by an RGB exposure device;

[0013] - controlling the infrared light source to additionally illuminate the scene with infrared light during at least one second acquisition process, and simultaneously controlling the camera module to expose the image sensor during the second acquisition process at a second exposure time different from the first exposure time, wherein the second RGB image is generated by the RGB exposure device and the IR image is generated by the IR exposure device; and

[0014] - merging at least one first RGB image and at least one second RGB image into an HDR image.

[0015] By means of the camera system according to the invention, it is possible to record both RGB images, i.e. color images (the letters stand for example for red, green and blue), and IR images, i.e. images recorded in the infrared spectrum, at a relatively high image acquisition rate, while ensuring a high dynamic range for the color images. The core idea is to generate an RGB image and an IR image simultaneously in a second acquisition process. In order to be able to generate an HDR image (i.e. a high dynamic range image) from a first RGB image and a second RGB image, the respective exposure times must be different. In general, it is not important whether the first exposure time is longer or shorter than the second exposure time. However, the second exposure time is preferably shorter than the first exposure time, since, in order to generate the IR image, a relatively short exposure time is used when the scene is illuminated simultaneously by an infrared light source. Typically, the second exposure time can be about 100 μs to 1 ms.

[0016] If the camera system according to the invention is to be used, for example, to detect persons, in particular to track the direction of their gaze, the infrared illumination of persons must be limited to a specified maximum duration per illumination in order to ensure eye safety. If the infrared illumination is too long, the illuminated surface will also heat up. This must be avoided. In addition, too long exposure times can have an adverse effect on the service life of the image sensor or the infrared light source. The exposure time must be long enough to prevent underexposure of the exposure device (e.g. the corresponding pixels of a CMOS sensor), but not too long to avoid overexposure of the pixels. If the exposure time is too short, so-called image noise will also increase.

[0017] The camera module and the computing unit can be integrated in a common device or arranged in a decentralized manner. A single frame image is generated during the acquisition process. Therefore, the acquisition process can also be called an exposure. The corresponding series of exposures can also be called an acquisition group. The infrared light source can be arranged arbitrarily relative to the camera module and the computing unit and can have any number of infrared lamps. The camera system according to the invention can be integrated into a vehicle, for example. In this case, the camera module can be arranged, for example, in the dashboard of the vehicle, for example in the area of ​​the instrument cluster or the head unit, and facing the interior of the vehicle. The infrared light source can be arranged laterally next to the camera module. In the case where the camera system is integrated into the vehicle, the computing unit can also be formed by the control unit of the vehicle subsystem.

[0018] An advantageous improvement of the camera system provides that the camera module and the computing unit are further adapted to continuously capture a scene in a sequence consisting of a plurality of at least a first capture process and a second capture process, wherein at least a plurality of first RGB images and a second RGB image recorded continuously in time are combined into an RGB video stream, a plurality of IR images recorded continuously in time are combined into an IR video stream and / or a plurality of HDR images generated continuously in time are combined into an HDR video stream. Therefore, the camera system according to the present invention is capable of generating a video stream, i.e., a video with a relatively high frame rate. For example, if the sampling rate of the camera system is 60 frames per second, an RGB video stream (i.e., a color video) with a frame rate of 60 frames per second, an IR video stream with a frame rate of 30 frames per second, and an HDR video stream with a frame rate of 30 frames per second can be generated. This means that a color video can be generated at the same frame rate as the image acquisition rate of the camera module, and an IR video and an HDR video with a frame rate equivalent to half of the sampling rate of the camera module can be generated.

[0019] The corresponding video stream can be used in the vehicle to provide a variety of functions. For example, the photos and / or videos can be processed by the computing device, and the information obtained thereby or directly obtained can be read as input variables by the driver assistance system. For example, the IR video stream can be used to track the driver's gaze direction and / or perform fatigue detection based on blinking frequency.

[0020] The camera system according to the present invention accordingly performs the steps of generating and outputting RGB video stream, IR video stream and / or HDR video stream. To this end, the camera system continuously captures scenes in the order of each capture process.

[0021] According to a further advantageous embodiment of the camera system, the computing unit is further adapted to:

[0022] - controlling the camera module to expose the image sensor with a third exposure time different from the first exposure time and the second exposure time in at least one third acquisition process, wherein a third RGB image is generated by the RGB exposure device;

[0023] - setting the order of the acquisition process to: a first acquisition process, a second acquisition process, and a third acquisition process; and

[0024] - merging at least one first RGB image, at least one second RGB image and at least one third RGB image into an HDR image.

[0025] Therefore, the camera system according to the invention also allows to consider more than two different exposures to generate an HDR image. The third exposure time is different from the first exposure time and the second exposure time and is in particular greater than the second exposure time. The third exposure time may be greater than or less than the first exposure time.

[0026] Therefore, the camera system can perform the following steps, namely, perform a third acquisition process with a third exposure time, and accordingly generate a third RGB image, and additionally consider the third RGB image for fusion of the HDR image, which further increases the dynamic range of the HDR image.

[0027] The more single-frame images with different exposure times are generated and the more corresponding video streams are output, the more the frame rate of the corresponding video stream is reduced. If the camera system has a sampling rate of 60 frames per second, for example, when the first acquisition process, the second acquisition process and the third acquisition process are performed, the frame rate of the IR video stream and the HDR video stream will be 20 frames per second respectively. In general, it is conceivable to set up additional acquisition processes, such as the third acquisition process, the fourth acquisition process or the fifth acquisition process, or even more acquisition processes, each of which has a separate exposure time. Accordingly, the dynamic range of the HDR image can be improved at the expense of the frame rate.

[0028] In order to generate an HDR image, an acquisition group consisting of a first acquisition process, a second acquisition process and a third acquisition process is used. Therefore, before the first acquisition process, the second acquisition process and the third acquisition process are executed again, the camera system generates a sequence of the first acquisition, the second acquisition and the third acquisition. The difference in the observation position of the acquisition process sequence also corresponds to the order of the acquisition processes: the second acquisition process, the third acquisition process and the first acquisition process or the third acquisition process, the first acquisition process and the second acquisition process. Therefore, the corresponding acquisition processes can be rearranged as needed, and the only relevant factor is that each of the three different acquisition processes is executed exactly once before the corresponding acquisition process is executed again, that is, before a single frame image is recorded again with the corresponding exposure time.

[0029] According to another advantageous embodiment of the camera system according to the invention, it is further provided that the computing unit is further adapted to:

[0030] - controlling the camera module to expose the image sensor with a third exposure time different from the first exposure time and the second exposure time in at least one third acquisition process, wherein a third RGB image is generated by the RGB exposure device;

[0031] - setting the order of the acquisition process to: the second acquisition process, the first acquisition process, the second acquisition process and the third acquisition process; and

[0032] - Merging at least one first RGB image with at least one second RGB image, and at least one third RGB image with at least one second RGB image, or at least one first RGB image, at least one second RGB image and at least one third RGB image into an HDR image.

[0033] Unlike the previous embodiment, the acquisition process of generating only RGB images and the acquisition process of generating both RGB images and IR images are now performed alternately. This makes it possible to output the corresponding video stream with an increased frame rate despite the setting of the third acquisition process with a third exposure time. In the case of a sampling rate of 60 Hz, a frame rate of 30 frames per second for the IR video stream and the HDR video stream can be guaranteed respectively. This can be achieved by inserting a second acquisition process between the first and third acquisition processes or between the third and first acquisition processes respectively and using two different exposures to generate HDR images respectively, that is, once for the combination of the first acquisition process and the second acquisition process, and once for the combination of the third acquisition process and the second acquisition process. It is also possible to use all three acquisition processes to generate HDR images. This will reduce the frame rate of the corresponding HDR video stream to 20 frames per second. In addition, one of the two second acquisition processes for generating HDR images is omitted.

[0034] Within an acquisition group for generating an HDR image, each acquisition process for generating only an RGB image occurs only once, wherein these acquisition processes are respectively separated from one another by an acquisition process for generating both an RGB image and an IR image simultaneously.

[0035] If further acquisition processes are accordingly carried out, for example a fourth, fifth or sixth acquisition process, in which only one RGB image is recorded with a respective separate exposure time, then a second acquisition process is respectively inserted between these acquisition processes.

[0036] The camera system according to the invention performs corresponding steps for this purpose.

[0037] According to another advantageous design of the camera system of the present invention, the computing unit comprises a circular buffer, which is adapted to sequentially cache single-frame images generated continuously in time, so that by adding each latest single-frame image to the circular buffer, the corresponding oldest single-frame image is deleted from the circular buffer, and the circular buffer has at least a storage capacity capable of permanently storing at least one first RGB image, one second RGB image and one IR image in the circular buffer; and wherein the computing unit is also adapted to fuse at least each latest first RGB image and second RGB image into an HDR image to generate an HDR image. This can generate an HDR video stream with a frame rate that is the same as the sampling rate of the camera module.

[0038] The camera system then performs the corresponding steps for this purpose.

[0039] This means that it is no longer necessary to record a complete acquisition group consisting of at least the first RGB image and the second RGB image to generate an HDR image. An HDR image can be generated after each recording of an RGB image (i.e., for example, after recording the first RGB image and after recording the second RGB image).

[0040] According to an advantageous development of the camera system according to the invention, the computing unit can also be adapted to determine the following acquisition process sequence:

[0041] -First acquisition process;

[0042] - a third acquisition process; and

[0043] - The second acquisition process is to fuse the latest first RGB image, second RGB image and third RGB image into an HDR image.

[0044] Thus, compared with using the first acquisition process and the second acquisition process alone, the dynamic range of the HDR image can be increased. Thanks to the use of the ring buffer, a frame rate equivalent to the sampling rate of the camera module can also be guaranteed.

[0045] To this end, the camera system performs the following steps: a first acquisition process, a third acquisition process, and then a second acquisition process, and then the computing unit fuses the respectively generated RGB images into an HDR image, wherein the latest images are read from the ring buffer.

[0046] According to another advantageous embodiment of the camera system according to the invention, the computing unit is further adapted to determine the following acquisition process sequence:

[0047] - Second acquisition process;

[0048] -First acquisition process;

[0049] - a second acquisition process; and

[0050] - A third acquisition process; and

[0051] The latest first RGB image, second RGB image and third RGB image are fused into an HDR image.

[0052] This makes it possible to increase the frame rate of the IR video stream at the expense of the frame rate of the HDR video stream. In the case of the sequence "first acquisition process, third acquisition process and second acquisition process", the frame rate of the IR video stream corresponds to one third of the sampling rate of the camera module and the frame rate of the HDR video stream corresponds to the sampling rate of the camera module, while the sequence "second acquisition process, first acquisition process, second acquisition process and third acquisition process" makes the frame rate of the IR video stream correspond to half of the sampling rate of the camera module, which is correspondingly applicable to the frame rate of the HDR video.

[0053] The camera system accordingly executes the following steps: performing acquisition processes that only record one RGB image each, separating these acquisition processes by executing a second acquisition process, and then the computing unit fuses the RGB images into an HDR image, wherein the latest images are read from the circular buffer.

[0054] According to another advantageous design of the camera system of the present invention, it is also proposed that the calculation unit is also adapted to determine the moving speed of the object present in at least two consecutive single-frame images, compare the moving speed with a predetermined threshold, and if the moving speed is greater than the threshold, omit the RGB image generated by the corresponding acquisition process when fusing into an HDR image. In order to determine the moving speed, the calculation unit can evaluate the position change of the corresponding object in the two single-frame images. The camera system can also include other sensors (for example, radar systems, laser scanners, etc.), which can be used to generate depth information. In this way, in particular, considering the time-of-flight principle, its moving speed can be determined by the position of the object changing over time. If the corresponding object (for example, a vehicle occupant) moves "too fast", the corresponding object appears "blurred" in the HDR image generated by the respective corresponding RGB images. However, by omitting the corresponding RGB image when fusing into an HDR image, it is possible to prevent the object from appearing unclear or blurred in the generated HDR image. In addition to blurring, the corresponding object that moves too fast may also be deformed. In order to remedy this, the proven homography transformation method and / or ghost removal method in the prior art can be used.

[0055] The camera system accordingly performs the following steps: the camera system determines the moving speed of the object in at least two consecutive single-frame images, compares the moving speed with a predetermined threshold, and omits these RGB images when fusing them into an HDR image if the moving speed exceeds the threshold.

[0056] According to another advantageous design of the camera system of the present invention, the computing unit is further adapted to identify at least one edge of an object and / or a surface structure of the object present in at least two consecutive single-frame images, compare the edges and / or surface structures in each single-frame image to identify differences, and when the identified difference is greater than a predetermined threshold, omit the RGB image generated by the corresponding acquisition process when fusing into an HDR image.

[0057] For this purpose, the camera system executes corresponding steps.

[0058] The materials captured by the camera system may have different reflectivities in the visible and infrared spectrum. This can cause, for example, the upper clothing of a vehicle occupant to appear transparent in the infrared spectrum, or the pattern of a plaid shirt to be unrecognizable in the infrared spectrum. Such RGB images can be omitted when merging to form an HDR image.

[0059] According to another advantageous design of the camera system of the present invention, it is also proposed that the computing unit is also adapted to execute a machine learning model. When the difference is greater than a threshold, at least two single-frame images can be read into the machine learning model as input variables, and then the machine learning model generates a corrected image from the at least two single-frame images, in which the edges and / or surface structures are highlighted and / or displayed in different positioning methods to reduce the difference to below the threshold.

[0060] The camera system executes corresponding steps for this purpose.

[0061] The machine learning model can in particular be or can use an artificial neural network. Thanks to sufficient training, the machine learning model can recognize surface structures, clothing, patterns, facial parts, eyes, teeth, etc. in the corresponding single-frame images and associate these with each other in the two single-frame images. If the corresponding surface structure and / or edge is missing in one of the single-frame images, the machine learning model then inserts it into the corresponding single-frame image or positions it in a different way.

[0062] The at least two consecutive single-frame images are preferably at least one RGB image and at least one IR image. In this way, for example, a pattern of a plaid shirt can be recognized in the RGB image and transferred to the corresponding IR image for display. Similarly, for example, a transparent clothing layer can be made opaque, or, for example, an edge that cannot be recognized in the RGB image due to darkness can be transferred from the IR image to the corresponding overly dark RGB image and highlighted therein.

[0063] Preferably, the computing unit is also adapted to control the camera module and the infrared light source so as to change the second exposure time, the operating time of the infrared light source and / or the sensitivity of the IR exposure device during the second acquisition process based on image statistical data obtained by analyzing at least one RGB image, IR image and / or HDR image that has been generated.

[0064] The camera system executes corresponding steps for this purpose.

[0065] The sensitivity of the IR exposure device relates, for example, to the ISO sensitivity. As image statistics, for example, a brightness histogram can be generated or an evaluation can be generated as to whether the bottom or top bins are filled, i.e. whether the individual pixels are underexposed or overexposed. The corresponding image statistics can be generated for the corresponding entire single-frame image (e.g., the entire RGB image) or for one or more regions of interest of the image. It is also possible to weight the different regions of interest. This makes it possible to make the relevant image regions particularly easy to identify. In this way, by influencing the second exposure time, the operating time of the infrared light source and the sensitivity of the IR exposure device, the recognizable brightness of the individual structures of the scene in the IR image can be influenced. The region of interest can then be defined as the image portion of the IR image in which, for example, the eyes of the vehicle driver are located. The image acquisition can then be adjusted so that the eyes of the vehicle driver can be identified in the corresponding IR image with sufficient quality. The corresponding region of interest can also be moved adaptively, for example due to the vehicle driver moving or adjusting his seat.

[0066] By taking into account the image statistics of the RGB image or the HDR image, the exposure of the corresponding IR image can be adjusted so that, if possible, the best available RGB image is also generated. However, the image indicators of the IR image are given priority, since the use of IR images in the vehicle environment is important for safety-related functions. For example, the second exposure time can be increased in order to prevent the corresponding second RGB image from being underexposed, since the second RGB image and the IR image are recorded with the same exposure time. However, the second exposure time can only be increased to such an extent that it does not lead to an overexposure of the IR image. In addition, for example, the operating time of the infrared light source during the second acquisition process can be reduced in order to additionally prevent an overexposure of the IR image. The second exposure time can thus be further increased. Similarly, the sensitivity of the IR exposure device can be reduced so that the corresponding IR image also appears darker. This has a favorable effect on the noise ratio of the IR image. The second exposure time must also not be increased to such an extent that the eye safety of the detected person is at risk, the surface illuminated by the infrared light source is overheated and / or the service life of the infrared light source is too greatly affected.

[0067] The method for operating the aforementioned camera system according to the present invention proposes:

[0068] - exposing the image sensor of the camera module with a first exposure time in at least one first acquisition process, wherein a first RGB image is generated by an RGB exposure device;

[0069] - turning on the infrared light source to additionally illuminate the scene with infrared light in at least one second acquisition process, and exposing the image sensor of the camera module with a second exposure time different from the first exposure time in the second acquisition process, wherein a second RGB image is generated by the RGB exposure device and an IR image is generated by the IR exposure device; and

[0070] - The computing unit fuses at least one first RGB image and at least one second RGB image into an HDR image. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Further advantageous embodiments of the camera system according to the invention and of the method for operating the same are also apparent from the exemplary embodiments which are described in more detail below with reference to the drawings.

[0072] in:

[0073] Figure 1 A schematic diagram of a camera system according to the present invention is shown;

[0074] Figure 2 shows the sequence of the acquisition process performed by the camera system according to the first embodiment;

[0075] Figure 3 shows the sequence of the acquisition process performed by the camera system according to the second embodiment;

[0076] Figure 4 shows the sequence of the acquisition process performed by the camera system according to the third embodiment;

[0077] Figure 5 shows the sequence of an acquisition process performed by a camera system according to a fourth embodiment;

[0078] Figure 6 shows the sequence of an acquisition process performed by a camera system according to a fifth embodiment; and

[0079] Figure 7 The sequence of the acquisition process performed by the camera system according to the sixth embodiment is shown. DETAILED DESCRIPTION

[0080] Figure 1The camera system 1 according to the invention is shown, which is integrated into a vehicle 9. The camera system 1 comprises a camera module 2, a computing unit 4 and an infrared light source 7. The camera module 2 and the infrared light source 7 are arranged on the vehicle 9 and oriented relative to the vehicle in such a way that the interior of the vehicle 9 is captured as a scene 3 by the camera module 2 or illuminated with infrared light by the infrared light source 7. This enables the vehicle occupants to be captured. The computing unit 4 can be arranged arbitrarily relative to the camera module 2 and the infrared light source 7, for example, it can be integrated into a common unit or it can be arranged in a decentralized manner, so that the computing unit 4 is formed by a computing unit of the vehicle 9, for example, by a control unit of a vehicle subsystem.

[0081] The camera module 2 includes an image sensor 5 having an RGB exposure device 6.1 and an IR exposure device 6.2. The individual exposure devices 6.1 and 6.2 are arranged in a two-dimensional matrix on the image sensor 5. When the image sensor 5 is exposed, a color image, also called an RGB image RGB, can be generated by the RGB exposure device 6.1, see the figure below. Similarly, the IR exposure device 6.2 can be used to detect infrared light and thereby generate an IR image IR.

[0082] The camera system 1 can combine continuously recorded single-frame images into a video stream, wherein RGB images (i.e., color images) can be concatenated to generate a color video, and IR images (i.e., images recorded in the infrared spectrum) can be concatenated to generate an infrared video. Vehicle occupants can use the color images or color videos for entertainment, such as taking selfies or making video calls. The IR images IR or IR videos are processed by the vehicle 9 to provide safety-related functions, such as monitoring the vital signs or attention of the vehicle driver. In particular, thanks to the illumination provided by the infrared light source 7, photos of sufficient quality for evaluation can be taken even under conditions of poor visibility, such as at night.

[0083] The following figures show different embodiments of the sequence of acquisition processes EV1, EV2, EV3 performed by the camera system 1. The camera system 1 continuously records a continuous stream of single-frame images, which represents a sequence 8 of corresponding acquisition processes EV1, EV2, EV3.

[0084] Figure 2 A first embodiment is shown, in which a first acquisition process EV1 and a second acquisition process EV2 are performed alternately. KAMERA For example, the sampling rate f KAMERA60 frames per second, i.e. n=60 Hz. In the first acquisition process EV1, the camera system 1 generates a first RGB image RGB with a first exposure time, which is represented by 0. In the second acquisition process EV2, the camera system 1 simultaneously generates an IR image IR and a second RGB image RGB with a second exposure time through the RGB exposure device 6.1 and the IR exposure device 6.2, and the second exposure time is especially less than the first exposure time, represented by the number -1. The IR image IR and the second RGB image RGB are both generated with the same exposure time (i.e., the second exposure time).

[0085] In order to generate the HDR image HDR, at least two RGB images with different exposure times are merged by the calculation unit 4. For this purpose, every two acquisition processes are combined into an acquisition group EG, which is represented by a dotted frame in the figure.

[0086] It is not important here whether the first acquisition process EV1 or the second acquisition process EV2 is performed first in the sequence 8. Figure 2 In , the boxes for the corresponding acquisition group EG are shown for both variants.

[0087] The corresponding acquired or generated images can then be combined into a video stream. Figure 2 In the embodiment, the frame rate f of the IR video stream IR-VID IR is the sampling rate f of camera module 2 KAMERA The frame rate of the HDR video stream HDR-VID is half of HDR is the sampling rate f of camera module 2 KAMERA half of that of the RGB video stream RGB-VID, and the frame rate f RGB is the sampling rate f of camera module 2 KAMERA。 However, the RGB video stream RGB-VID will "flicker" because the concatenated images are alternately bright or dark due to different exposure times. Therefore, the visual output of the RGB video stream RGB-VID is not a preferred embodiment, but is generally feasible.

[0088] Figure 3 An alternative embodiment is shown, in which a third acquisition process EV3 is additionally performed. Therein, the image sensor 5 is exposed with a third exposure time, and a third RGB image RGB is recorded by the RGB exposure device 6.1. The third exposure time can be greater than or less than one of the other two exposure times, for example, greater than the first exposure time, and the third exposure time is represented by the number +1. Here, the numbers +1, 0 and -1 represent the exposure times of the corresponding acquisition processes, for example, compared to 0, +1 is a longer exposure time, and -1 is a shorter exposure time, preferably a much shorter exposure time suitable for infrared imaging. However, in Figure 5In the example, the third exposure time of acquisition process EV3 is less than the first exposure time of acquisition process EV1, so the two numbers 0 and +1 are exchanged accordingly, that is, EV1 is Figure 5 has the number 0 in Figure 3 has the number 1, EV3 is in Figure 5 has the number +1 and Figure 3 By providing a third acquisition process, the dynamic range of the HDR image HDR can be expanded, but at the expense of the corresponding frame rate.

[0089] exist Figure 3 In the illustrated embodiment, each acquisition group EG includes exactly one first acquisition process EV1 , exactly one second acquisition process EV2 , and one third acquisition process EV3 .

[0090] Figure 4 Another alternative design is shown, in which RGB images RGB with exposures of 0 and +1 are generated alternately, wherein after each acquisition process, a second acquisition process EV2 with an exposure time of -1 (i.e., an exposure time for infrared imaging) is inserted therein. This can increase the corresponding frame rate f of the individual video streams IR-VID, HDR-VID IR 、f HDR .

[0091] Here, in order to generate a directly continuous HDR image HDR, two RGB images RGB with different exposure time combinations are used, i.e. once with 0 and -1 and once with +1 and -1. In general, it is also possible to generate an HDR image HDR using all three exposure times (i.e. -1, 0 and +1), while the frame rate of the HDR video stream HDR-Vid is reduced to n / 3. This variant is not shown in any figure.

[0092] Figure 5 Another alternative design is shown, in which the single-frame images generated by the camera system 1 are at least temporarily cached in a ring buffer. In each acquisition cycle, the older single-frame images are overwritten by the newer single-frame images. Figures 5 to 7 In the figure, the acquisition groups EG that are consecutive to each other in the ring buffer for generating the HDR image HDR are shown in an increasingly larger form, that is, the smallest acquisition group EG (for example, at time t0) is followed by the next larger acquisition group EG (for example, at time t1), which is followed by the next larger acquisition group (for example, at time t3), and so on. Therefore, the data of a single frame image located to the left of any acquisition group can be removed from the ring buffer, and the free storage locations in the ring buffer are filled with new data on the opposite side. As already mentioned, in Figure 5 In the embodiment, the exposure time of the third acquisition process EV3 is shorter than the exposure time of the first acquisition process EV1.

[0093] The HDR image HDR is always generated using the most recent RGB image RGB in the ring buffer, which in the present case may preferably have only three storage locations. Figure 5 As shown, this is equivalent to a sliding acquisition group EG. Therefore, in order to generate the HDR image HDR, it is no longer necessary to fully record the corresponding acquisition group EG. Instead, it is sufficient to perform another acquisition process EV1, EV2 or EV3 to form a new acquisition group and generate the HDR image HDR. For example, in the case of a circular buffer with three storage locations, the first HDR image HDR is generated from the acquisition processes EV1, EV3 and EV2 shown on the left outer edge. After the memory is updated, EV1 on the left edge of the memory is removed, and a new EV1 is stored on the right side, and an HDR image HDR is generated from the acquisition processes EV3, EV2 and the new EV1. In the next update, EV2 on the left is removed, and a new EV3 is stored on the right side, and then an HDR image HDR is generated from EV2, EV1 and the new EV3. The circular buffer is continuously updated, and the next step corresponds to the first step described above. Here, Figure 5 Shown with Figure 3 A similar situation occurs, where each different acquisition process EV1, EV2, EV3 is executed only once before being executed again. The HDR image is generated from the acquisition processes EV1, EV2, EV3, respectively, i.e. the exposure times +1, 0 and -1 are alternately performed in sequence. Figure 3 Compared with the embodiment in the embodiment, the frame rate f of the HDR video stream HDR-VID HDR It can be increased to the sampling rate f of the camera module 2 KAMERA =n, the frame rate of the infrared image is fir=n / 3.

[0094] Figure 6 Another alternative design is shown, which is Figure 4 The embodiment shown is equivalent to that shown in FIG. 1 . In which, each time after or before performing the acquisition process of recording only the RGB image RGB alternately with an exposure time of +1 or 0, a second acquisition process EV2 is performed (i.e., the RGB image RGB and the IR image IR are simultaneously acquired while the scene 3 is illuminated by the infrared light source 7). Here, the acquisition group EG for generating the HDR image is not as follows. Figure 5 The ring buffer is overwritten in each cycle. The ring buffer preferably also has three storage locations. The basic operation of the ring buffer corresponds to Figure 5 This makes the frame rate of the IR video stream IR-VID f IR It can be increased from one third to n / 2, but the frame rate of the HDR video stream HDR-VID is f HDRReduce from n to n / 2.

[0095] You can also Figure 4 As shown, two different exposure times are always merged into one HDR image HDR, namely once for 0 and -1 and once for +1 and -1. However, this reduces the dynamic range unnecessarily.

[0096] Figure 7 Another possible implementation is shown, in which the frame rate of the HDR video stream HDR-Vid can be increased to the value of the image acquisition rate of the camera module. Therein, the order of the acquisition processes stored in a circular scrolling manner in the ring buffer is: the second acquisition process EV2 (exposure time -1), followed by the third acquisition process EV3 (exposure time +1), then the second acquisition process again, followed by the first acquisition process EV1 (exposure time 0). Here, the HDR image HDR is fused in each acquisition cycle from images stored in the ring buffer, which are recorded with three exposure times -1, 0 and +1, respectively. As a result, the acquisition group EG alternately spans the last three or the last four stored acquisition processes EV per cycle, so that fusion can be performed with the three exposure times mentioned. Therefore, the ring buffer preferably has four storage locations. Figure 7 In the first state, the four storage locations are sequentially the acquisition process EV2 shown by the dotted line on the left and the subsequent EV1, EV2 and EV3, wherein the HDR image HDR is generated from the acquisition group including the acquisition processes EV3, EV2 and EV1 shown by the dotted line. After the new data is updated and saved, the new acquisition process EV2 is added to the right side of the circular buffer and the EV2 on the left side shown by the dotted line is removed, so now EV1, EV2, EV3 and EV2 are stored in the circular buffer in sequence from the left side. The HDR image HDR is generated from the latest EV2, EV3 and EV1 in the acquisition group EG shown by the dotted line. After the new data is saved, the new acquisition process EV1 is added to the right side of the circular buffer and the old EV1 on the left side is removed, and the order of the acquisition processes from the left side is EV2, EV3, EV2 and EV1. The HDR image HDR is generated from the continuous acquisition processes EV3, EV2 and EV1 from the left side in the acquisition group EG shown by the dotted line. In the next cycle, the new EV2 is stored in the circular buffer and the old one is deleted. The ring buffer again includes EV3, EV2, EV1 and EV2 in order from the left. The HDR image HDR is generated from the latest EV2, EV1 and EV3 in the acquisition group EG shown by the dotted line.

[0097] In the next storage process, EV3 is stored and the old EV3 is deleted from the buffer, so that the buffer has the above-mentioned first state again, that is, from the left side, there are acquisition processes EV2, EV1, EV2 and EV3, where the HDR image HDR is generated from EV3, EV2 and EV1 in the acquisition group shown by the dotted line. Starting from this first state, the storage process after the first state and the generation of the HDR image HDR are repeated as described above.

[0098] An IR image is recorded during each acquisition process EV2.

[0099] By means of the camera system 1 according to the invention and the corresponding operating method thereof, an IR video stream IR-VID and an HDR video stream HDR-VID with a sufficient frame rate and a relatively high dynamic range can be generated simultaneously based on the currently available image sensor 5 or camera module 2. Thanks to the active illumination used during the recording of the infrared images, sufficiently recognizable single-frame images can be recorded, especially in poor lighting conditions. By using a ring buffer, the frame rate of the HDR video stream HDR-VID can be further increased while maintaining the same dynamic range. The frame rate f of the HDR video stream HDR-VID HDR Corresponding to f of camera module 2 KAMERA =n, the frame rate of the infrared image is fir=n / 2.

Claims

1. A camera system (1), comprising a camera module (2) for generating a photo of a scene (3) and a computing unit (4) for controlling the camera module (2) and processing the photo, wherein the camera module (2) has an image sensor (5), the image sensor being adapted to collect light in the visible spectrum through an RGB exposure device (6.1) and to collect light in the infrared spectrum through an IR exposure device (6.2), the camera module (2) being adapted to record at least two single-frame images of the scene in at least two acquisition processes (EV1, EV2) with different exposure times through the exposure of the image sensor (5), and the computing unit (4) being adapted to fuse the at least two single-frame images into an HDR image (HDR), The computing unit (4) is further adapted to: - controlling the camera module (2) to expose the image sensor (5) with light in the visible spectrum for a first exposure time in at least one first acquisition process (EV1), wherein a first RGB image (RGB) is generated by the RGB exposure device (6.1); - controlling an infrared light source (7) to illuminate the scene (3) with infrared light in addition to light in the visible spectrum in at least one second acquisition process (EV2), and simultaneously controlling the camera module (2) to expose the image sensor (5) in the second acquisition process (EV2) with a second exposure time different from the first exposure time, It is characterized in that - the image sensor (5) generates a second RGB image (RGB) through the RGB exposure device (6.1) and an IR image (IR) through the IR exposure device (6.2) simultaneously during the second acquisition process; and The calculation unit fuses at least one first RGB image (RGB) and at least one second RGB image (RGB) into an HDR image (HDR).

2. The camera system (1) according to claim 1, It is characterized in that The camera module (2) and the computing unit (4) are also adapted to continuously capture the scene (3) in a sequence (8) consisting of a plurality of at least first acquisition processes and second acquisition processes (EV1, EV2), wherein at least a plurality of first RGB images (RGB) and a second RGB image (RGB) recorded continuously in time are combined into an RGB video stream (RGB-VID), a plurality of IR images (IR) recorded continuously in time are combined into an IR video stream (IR-VID) and / or a plurality of HDR images (HDR) generated continuously in time are combined into an HDR video stream (HDR-VID).

3. The camera system (1) according to claim 1 or 2, It is characterized in that The computing unit (4) is further adapted to: - Control the camera module (2) to capture the image in at least one third acquisition process (EV3) exposing the image sensor (5) with a third exposure time different from the first exposure time and the second exposure time, wherein a third RGB image (RGB) is generated by the RGB exposure device (6.1); - setting the order of the acquisition process to: a first acquisition process (EV1), a second acquisition process (EV2) and a third acquisition process (EV3); as well as - merging the at least one first RGB image (RGB), the at least one second RGB image (RGB) and the at least one third RGB image (RGB) into an HDR image (HDR).

4. The camera system (1) according to claim 1 or 2, It is characterized in that The computing unit (4) is further adapted to: - Control the camera module (2) to capture the image in at least one third acquisition process (EV3) exposing the image sensor (5) with a third exposure time different from the first exposure time and the second exposure time, wherein a third RGB image (RGB) is generated by the RGB exposure device (6.1); - setting the order of the acquisition process to: the second acquisition process (EV2), the first acquisition process (EV1), the second acquisition process (EV2) and the third acquisition process (EV3); as well as -Merging the at least one first RGB image (RGB) with the at least one second RGB image (RGB), and the at least one third RGB image (RGB) with the at least one second RGB image (RGB), or merging the at least one first RGB image (RGB), the at least one second RGB image (RGB) and the at least one third RGB image (RGB) into an HDR image (HDR).

5. The camera system (1) according to any one of claims 1 to 4, It is characterized in that The computing unit (4) comprises a circular buffer adapted to sequentially cache single-frame images generated successively in time, so that by adding each latest single-frame image to the circular buffer, the corresponding oldest single-frame image is deleted from the circular buffer, the circular buffer having at least a storage capacity capable of permanently storing at least one first RGB image (RGB), one second RGB image (RGB) and one IR image (IR) in the circular buffer; and wherein, The calculation unit (4) is also adapted to fuse at least the latest first RGB images (RGB) and second RGB images (RGB) into an HDR image (HDR) to generate an HDR image (HDR).

6. The camera system (1) according to claim 5, It is characterized in that The calculation unit (4) is further adapted to determine the following acquisition process sequence: -First acquisition process (EV1); - a third acquisition process (EV3); and - a second acquisition process (EV2), and The latest first RGB image (RGB), second RGB image (RGB) and third RGB image (RGB) are fused into an HDR image (HDR).

7. The camera system (1) according to claim 5, It is characterized in that The calculation unit (4) is further adapted to determine the following acquisition process sequence: - Second acquisition process (EV2); -First acquisition process (EV1); - Second acquisition process (EV2); and - a third acquisition process (EV3), and The latest first RGB image (RGB), second RGB image (RGB) and third RGB image (RGB) are fused into an HDR image (HDR).

8. The camera system (1) according to any one of claims 1 to 7, It is characterized in that The calculation unit (4) is also adapted to determine the moving speed of an object present in at least two consecutive single-frame images, compare the moving speed with a predetermined threshold, and when the moving speed is greater than the threshold, omit the RGB image (RGB) generated by the corresponding acquisition process when fusing into an HDR image (HDR).

9. The camera system (1) according to any one of claims 1 to 8, It is characterized in that The computing unit (4) is further adapted to identify at least one edge of an object and / or a surface structure of the object present in at least two consecutive single-frame images, compare the edges and / or the surface structures in the single-frame images to identify differences, and when the identified differences are greater than a predetermined threshold, omit the RGB image (RGB) generated by the corresponding acquisition process when fusing into an HDR image (HDR).

10. The camera system (1) according to claim 9, It is characterized in that The computing unit (4) is also adapted to execute a machine learning model. When the difference is greater than the threshold, the at least two single-frame images can be read into the machine learning model as input variables, and then the machine learning model generates a corrected image from the at least two single-frame images, in which the edges and / or the surface structures are highlighted and / or displayed in different positioning methods to reduce the difference to below the threshold.

11. The camera system (1) according to any one of claims 8 to 10, It is characterized in that The at least two consecutive single-frame images include at least one RGB image (RGB) and at least one IR image (IR).

12. The camera system (1) according to any one of claims 1 to 11, It is characterized in that The computing unit (4) is also adapted to control the camera module (2) and the infrared light source (7) so as to change the second exposure time, the operating time of the infrared light source (7) and / or the sensitivity of the IR exposure device (6.2) in the second acquisition process (EV2) according to image statistics obtained by analyzing at least one generated RGB image (RGB), IR image (IR) and / or HDR image (HDR).

13. A method for operating a camera system (1) according to claim 1, It is characterized in that - exposing the image sensor (5) of the camera module (2) to light in the visible light range for a first exposure time in at least one first acquisition process (EV1), wherein a first RGB image (RGB) is generated by the RGB exposure device (6.1); - turning on the infrared light source (7) to illuminate the scene (3) with infrared light in addition to light in the visible spectrum in at least one second acquisition process (EV2), and exposing the image sensor (5) of the camera module (2) with a second exposure time different from the first exposure time in the second acquisition process (EV2), wherein in the second acquisition process a second RGB image (RGB) is generated by the RGB exposure device (6.1) and an IR image (IR) is generated by the IR exposure device (6.2); and - the calculation unit (4) fuses the at least one first RGB image (RGB) and the at least one second RGB image (RGB) into an HDR image (HDR).

Images