Camera system and method for operating a camera system

Abstract

The present invention relates to a camera system (1) comprising a camera module (2) for generating a camera image of a scene (3) and a computing unit (4) for controlling the camera module (2) and processing the camera image. The present invention is characterized in that an infrared light source (7) is provided, and the computing unit (4) further - exposes the image sensor (5) with a first exposure time during at least one first detection process (EV1) and controls the camera module (2) such that a first RGB image (RGB) is generated using RGB exposure means (6.1); - controls the infrared light source (7) to illuminate the scene additionally with infrared light during at least one second detection process (EV2), and simultaneously, during the second detection process (EV2), exposes the image sensor (5) with a second exposure time different from the first exposure time and controls the camera module (2) such that a second RGB image (RGB) is generated using RGB exposure means (6.1) and an IR image (IR) is generated using IR exposure means (6.2); - is configured to merge at least one first RGB image (RGB) and at least one second RGB image (RGB) into one HDR image (HDR).

Description

Technical Field

[0001] The present invention relates to a camera system as more particularly defined in the preamble of claim 1 and to a method for operating such a camera system.

Background Art

[0002] Camera systems are widely known in both analog and digital formats. To generate a digital camera image, a corresponding camera system typically uses an image sensor such as a CMOS sensor. Such an image sensor includes a large number of pixels arranged in a two-dimensional grid, and these pixels output an electronic signal when exposed. To generate a color camera image, for example, color filters of red, green, and blue are arranged in front of the pixels. For this purpose, the filters are arranged in a regular pattern. The filter may be, for example, a Bayer filter or an interference filter. By providing appropriate components, individual pixels can be made sensitive to infrared light as well. Thereby, using the image sensor, it is possible to detect a color image in the visible spectrum or an image in the infrared spectrum.

[0003] If this is desired, typically an exposure series consisting of a sequence of consecutive color and infrared images is generated. Thus, in a camera system having an image detection rate (frame rate) of 60 images / second, when color and infrared images are alternately taken, the color and infrared images can be taken at a detection rate of 30 images / second each.

[0004] To expand the dynamic range, in an exposure series, it is also possible to perform shooting with exposures at different exposure times, which realizes the generation of high-dynamic-range images well known as high-dynamic-range (HDR). In a camera having a sampling rate of 60 images / second, when three successive exposures are performed with a short exposure time, an intermediate exposure time, and a long exposure time, for example, the generation of an HDR video at a frequency of 20 images / second is realized.

[0005] By simultaneously generating infrared images and HDR images, the camera sequentially records a plurality of color images with different exposure times, and accordingly, it becomes necessary to further shoot infrared images, so the frame rates of the corresponding HDR video stream and infrared video stream generated by the camera system decrease. When the frame rate of the video stream is excessively low, for example, below 30 images / second, the corresponding video is recognized as "jerky" or "intermittent", which reduces the user comfort during viewing. Furthermore, if the exposure time is excessively long, or the time difference between individual images in the exposure series is excessively large, it leads to the occurrence of motion artifacts. When a person moves during the shooting of the corresponding exposure series, the person appears "blurry" in the video stream.

[0006] The camera system is also used in vehicles to provide functions regarding comfort and safety. That is, for example, using an in-vehicle camera, it is possible to perform driver state monitoring or seat occupancy detection. Also, vehicle occupants can use the camera system for taking "selfies" and for conducting video calls. In particular, to provide an important function such as driver state monitoring where safety is crucial, infrared images are taken, which ensures sufficient image quality even, for example, when the light conditions are poor in a dark place. When the corresponding infrared images are taken, the interior of the vehicle can be actively illuminated using an infrared light source. For this purpose, the camera system and the infrared light source are synchronized.

[0007] To reduce costs and to utilize as efficiently as possible the structural space available in the vehicle, it is desirable to use as few cameras as possible, and if possible only one camera. At the same time, it is necessary to capture camera images or videos with as high a dynamic range, high image quality, and high frame rate as possible in both the visible spectrum and the infrared spectrum.

[0008] To improve the frame rate when generating HDR video, a corresponding vehicle camera system is known from Patent Document 1. The vehicle camera system includes a ring buffer to which camera images generated by the camera system are supplied. Since the ring buffer has a predetermined size, the camera images generated in the past are overwritten by the latest generated camera image. At this time, the HDR image is generated from the exposure group. Such an exposure group includes, for example, three individual exposures with different exposure times, for example, a short exposure time, an intermediate exposure time, and a long exposure time. With the vehicle camera system disclosed in this publication, instead of having to first capture a complete exposure group, an HDR image can already be generated each time an individual exposure is captured. That is, the latest captured individual exposure overwrites the individual exposure of the exposure group having the same exposure time that was used to generate the HDR image. Therefore, for three different exposure times, each individual exposure is utilized to generate three HDR images.

Prior Art Documents

Patent Documents

[0009]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0010] The problem underlying the present invention is to provide an improved camera system that can simultaneously capture a camera image or video in both the visible spectrum and the infrared spectrum, and in doing so, can achieve a high frame rate for both the color image and the infrared image, even though a relatively high dynamic range is provided in the color image, and a method for operating such a camera system.

Means for Solving the Problem

[0011] According to the present invention, this problem is solved by a camera system having the features of claim 1 and a method for operating a camera system having the features of claim 13. Advantageous configurations and developments will become apparent from the claims that are dependent on those claims.

[0012] A camera system as described at the beginning has a camera module for generating a camera image of a scene and a computing unit for controlling the camera module and processing the camera image. The camera module has an image sensor designed to detect light in the visible spectrum using RGB exposure means and to detect light in the infrared spectrum using IR exposure means. The camera module is designed to capture at least two individual images of the scene by exposing the image sensor during at least two detection processes using different exposure times. The computing unit is designed to merge at least two individual images into one HDR image. According to the present invention, the camera system is further developed by providing an infrared light source, and according to the present invention, the computing unit is further designed as follows: - Exposing the image sensor with a first exposure time during at least one first detection process and controlling the camera module so that a first RGB image is generated using the RGB exposure means; - Control an infrared light source to additionally illuminate the scene with infrared light during at least one second detection process. At the same time, during the second detection process, expose the image sensor with a second exposure time different from the first exposure time, and control the camera module so that a second RGB image is generated using RGB exposure means and an IR image is generated using IR exposure means; - Merge at least one first RGB image and at least one second RGB image into one HDR image.

[0013] Using the camera system according to the present invention, both an RGB image, which is a color image (the letters represent, for example, red, green, and blue), and an IR image, which is an image taken in the infrared spectrum, can be taken with a relatively high image detection rate, and it is realized to guarantee a high dynamic range in the color image. Here, the core idea is to simultaneously generate an RGB image and an IR image during the second detection process. In order to be able to generate one HDR image, that is, a high dynamic range image, from the first RGB image and the second RGB image, the respective exposure times must be different. Generally, whether the first exposure time is longer or shorter than the second exposure time is not important. However, in order to generate an IR image when simultaneously illuminating the scene with an infrared light source, a relatively short exposure time is used, so preferably the second exposure time is shorter than the first exposure time. Typically, the second exposure time may be on the order of 100 μs to 1 ms.

[0014] When using the camera system according to the present invention, for example, when a human is detected, especially when the line-of-sight direction of the human is tracked, in order to ensure the safety of the human eyes, the infrared exposure of the human needs to be limited to a specified maximum duration for each exposure. Furthermore, if the infrared exposure is too long, there is a risk that the exposed surface will be heated. This must be avoided. In addition, an overly long exposure time adversely affects the lifespan of the image sensor or the infrared light source. In order to prevent insufficient exposure of the exposure means, that is, insufficient exposure of the corresponding pixels of, for example, a CMOS sensor, the exposure time needs to be long enough. However, it must not be too long in order to prevent overexposure of the pixels. Also, when the exposure time is excessively short, so-called image noise also increases.

[0015] The camera module and the calculation unit may be integrated into a common device or may be locally distributed. In one detection process, one individual image is generated. Therefore, the detection process can also be referred to as an exposure. A corresponding series of exposures can also be referred to as a detection group. The infrared light source can be arranged at an arbitrary position with respect to the camera module and the calculation unit and can have an arbitrary number of infrared light sources. The camera system according to the present invention can be integrated into, for example, a vehicle. In this case, the camera module can be arranged, for example, in the area of the vehicle dashboard, such as the instrument cluster or the head unit, and can be directed towards the interior of the vehicle. The infrared light source can be arranged on the side of the camera module. When the camera system is integrated into a vehicle, the calculation unit can also be formed by the control device of the vehicle subsystem.

[0016] In an advantageous development of the camera system, the camera module and the computing unit are further designed to continuously detect the scene in a sequence of a plurality of detection processes including at least a first detection process and a second detection process. At this time, at least a first RGB image and a second RGB image taken continuously in time a plurality of times are integrated into one RGB video stream, IR images taken continuously in time a plurality of times are integrated into one IR video stream, and / or HDR images generated continuously in time a plurality of times are integrated into one HDR video stream. Therefore, the camera system according to the present invention enables the generation of a video stream, that is, the generation of a video having a relatively high frame rate. For example, when the camera system has a scanning speed of 60 images / second, a color video having a frame rate of 60 images / second, an IR video stream having a frame rate of 30 images / second, and an HDR video stream having a frame rate of 30 images / second can be generated. That is, a color video having the same frame rate as the frame rate of the camera module can be generated, and an IR video and an HDR video having a frame rate corresponding to half of the scanning speed of the camera module can be generated.

[0017] The corresponding video stream can be used to provide various functions in a vehicle. For example, the camera image and / or video can be processed by a computing device, and can also be directly read by a driving assistance system, or the information obtained therefrom can be read as input variables. For example, the IR video stream can be used to track the line-of-sight direction of a person driving a vehicle and / or perform fatigue identification based on the frequency of blinking.

[0018] The camera system according to the present invention correspondingly executes a processing step of generating and outputting an RGB video stream, an IR video stream, and / or an HDR video stream. For this purpose, the camera system continuously detects the scene using the sequence of each detection process.

[0019] According to another advantageous configuration of the camera system, the computing unit is further designed as follows: - Exposing the image sensor with a third exposure time different from the first exposure time and the second exposure time during at least one third detection process, and controlling the camera module so that a third RGB image is generated using the RGB exposure means; - Setting the order of the detection processes to the first detection process, the second detection process, and the third detection process; - Merging at least one first RGB image, at least one second RGB image, and at least one third RGB image into one HDR image.

[0020] That is, the camera system according to the present invention can also consider more than two different exposures to generate an HDR image. The third exposure time is different from the first exposure time and the second exposure time, and is particularly longer than the second exposure time. Here, the third exposure time may be longer or shorter than the first exposure time.

[0021] Therefore, the camera system can perform a third detection process with the third exposure time, generate a third RGB image accordingly, and execute method steps to additionally consider this third RGB image for merging into the HDR image. Thereby, the dynamic range of the HDR image can be further expanded.

[0022] As the number of individual images generated using different exposure times increases, and as the corresponding number of video streams output increases, the frame rate of the corresponding video stream decreases. When the camera system has a sampling rate of, for example, 60 images per second, if the aforementioned first detection process, second detection process, and third detection process are carried out, the frame rates for the IR video stream and the HDR video stream will each be 20 images per second. Generally, it is also conceivable to provide a further plurality of detection processes, that is, for example, a third detection process, a fourth detection process, or a fifth detection process, or even more detection processes, each having a different exposure time. Accordingly, the dynamic range of the HDR images can be improved at the expense of the frame rate.

[0023] To generate HDR images, a detection group is used from the first detection, second detection, and third detection. Accordingly, the camera system generates a sequence consisting of the first detection, second detection, and third detection before the first detection process, second detection process, and third detection process are newly carried out. Depending on where the sequence of detection processes is considered, this also corresponds to the following order of the detection processes: the second detection process, the third detection process, and the first detection process, or the third detection process, the first detection process, and the second detection process. That is, each detection process can be arbitrarily interchanged, and it is important that each detection process of the three different detection processes is carried out again before each detection process is newly carried out, that is, before individual images are newly taken with each exposure time.

[0024] In another advantageous configuration of the camera system according to the invention, furthermore, the computing unit is designed further as follows: - During at least one third detection process, control the camera module so that the image sensor is exposed with a third exposure time different from the first exposure time and the second exposure time, and a third RGB image is generated using the RGB exposure means; - Set the order of the detection processes to the second detection process, the first detection process, the second detection process, and the third detection; - Merge at least one first RGB image with at least one second RGB image, merge at least one third RGB image with at least one second RGB image, or merge at least one first RGB image, at least one second RGB image, and at least one third RGB image into one HDR image.

[0025] Different from the above configuration, a detection process that generates only RGB images and a detection process that generates both RGB images and IR images are alternately performed. As a result, even though a third detection process using a third exposure time is provided, a corresponding video stream with an improved frame rate can be output. Therefore, at a sampling frequency of 60 Hz, frame rates of 30 images / second can be guaranteed for the IR video stream and the HDR video stream, respectively. This is achieved by inserting one second detection process between the first detection process and the third detection process, or between the third detection process and the first detection process, and using two different exposures for generating the HDR image, i.e., one is a combination of the first detection process and the second detection process, and the other is a combination of the third detection process and the second detection process. Also, all three detection processes may be used to generate the HDR image. As a result, the frame rate decreases to 20 images / second for the corresponding HDR video stream. Further, one of the two second detection processes is omitted for generating the HDR image.

[0026] Therefore, within the detection group for generating the HDR image, each detection process that generates only RGB images is performed only once, and those detection processes are separated from each other by a detection process that generates both RGB images and IR images.

[0027] Correspondingly, more detection processes, for example a fourth detection process, a fifth detection process or a sixth detection process, are carried out. In each detection process, if only RGB images are taken with different exposure times respectively, one second detection process is inserted between these detection processes respectively.

[0028] For this purpose, the camera system according to the invention executes corresponding method steps.

[0029] According to another advantageous configuration of the camera system according to the invention, the computing unit includes a ring buffer, and the ring buffer is designed to sequentially temporarily store individual images generated successively in time, and each time the latest individual image is added to the ring buffer, the oldest individual image at that time is deleted from the ring buffer, and at least the first RGB image, the second RGB image and the IR image can be held in the ring buffer. The computing unit is further designed to merge at least the latest first RGB image and the second RGB image into one HDR image in order to generate an HDR image. Thereby, an HDR video stream having the same frame rate as the scanning speed of the camera module can be generated.

[0030] The camera system executes corresponding method steps for this purpose.

[0031] Therefore, in order to generate an HDR image, it is not necessary to capture a complete detection group consisting of at least the first RGB image and the second RGB image. That is, an HDR image can be generated each time an RGB image is captured, for example, after the first RGB image is captured or after the second RGB image is captured.

[0032] According to an advantageous development of the camera system according to the invention, the computing unit is further designed to define the order of the detection processes as follows: - The first detection process; - The second detection process; and - The second detection process merges the latest first RGB image, second RGB image, and third RGB image each time into an HDR image.

[0033] In this way, the dynamic range of the HDR image can be expanded compared to the case where only the first detection process and the second detection process are used. By using a ring buffer, the frame rate corresponding to the scanning speed of the camera module can also be guaranteed here.

[0034] For this purpose, the camera system performs the following method steps: the first detection process, the third detection process, and the second detection process are carried out, and then the generated RGB images are merged by the calculation unit into one HDR image, and the latest image is read from the ring buffer each time.

[0035] According to another advantageous configuration of the camera system according to the present invention, the calculation unit is further designed to define the order of the detection processes as follows: - The second detection process; - The first detection process; - The second detection process; and - The third detection process; and The latest first RGB image, second RGB image, and third RGB image each time are merged into one HDR image.

[0036] This sacrifices the frame rate of the HDR video stream to achieve an improvement in the frame rate of the IR video stream. In the order of "the first detection process, the third detection process, and the second detection process", the frame rate of the IR video stream corresponds to one-third of the scanning speed of the camera module, while the frame rate of the HDR video stream corresponds to the scanning speed of the camera module. On the other hand, the order of "the second detection process, the first detection process, the second detection process, and the third detection process" achieves a frame rate of the IR video stream corresponding to half of the scanning speed of the camera module, and the same applies to the frame rate of the HDR video stream.

[0037] Correspondingly, the camera system performs the following method steps: performing each detection process in which only one RGB image is captured, each separated by the implementation of the second detection process, and then merging each RGB image into an HDR image by the calculation unit, with the latest image being read from the ring buffer each time.

[0038] Furthermore, in another advantageous configuration of the camera system according to the invention, the computing unit further determines the speed of movement of an object present in at least two consecutive individual images, compares the speed of movement with a predetermined threshold value, and, if the speed of movement is greater than the threshold value, is designed to exclude the RGB image generated for the corresponding detection process from the merging into the HDR image. In order to determine the speed of movement, the computing unit can evaluate the change in the position of each object in two individual images. However, the camera system can also include other sensors such as a radar system, a laser scanner, etc., and the generation of depth information is realized using those sensors. Thereby, in particular considering the principle of Time-of-Flight, the speed of movement of the object can be determined from the time-dependent change in the position of the object. When a corresponding object, for example a vehicle occupant, moves "too fast", in the HDR image generated from the corresponding RGB image, each object appears "blurred". However, by excluding the corresponding RGB image from the merging into the HDR image, it is possible to prevent the generation of an HDR image in which the object appears unclear or indistinct. Also, in addition to becoming unclear, distortion of an object moving excessively fast correspondingly can also occur. To remove this, established homography techniques and deghosting can be used from the prior art.

[0039] Correspondingly, the camera system executes a method step of determining the speed of movement of an object in at least two consecutive individual images, comparing the speed of movement with a predetermined threshold value, and, if the speed of movement exceeds the threshold value, excluding those RGB images from the merging into the HDR image.

[0040] In another advantageous configuration of the camera system according to the invention, the computing unit further identifies at least one edge and / or the surface structure of an object present in at least two consecutive individual images, compares the edges and / or surface structures in each individual image with each other to identify differences, and, if the identified differences are greater than a predetermined threshold value, excludes the RGB image generated for the corresponding detection process from the merging into the HDR image.

[0041] For this purpose, corresponding method steps are carried out by the camera system.

[0042] Substances detected by the camera system may have different reflection characteristics in the visible and infrared spectra. This can, for example, result in the upper layer of the vehicle occupant's clothing appearing transparent in the infrared spectrum or the check pattern of the shirt not being distinguishable in the infrared spectrum. RGB images of that kind can be excluded from the merge into the HDR image.

[0043] Furthermore, in another advantageous configuration of the camera system according to the invention, the computing unit is further designed to execute a machine learning model in which at least two individual images can be read in as input variables if the difference is greater than a threshold value, and subsequently the machine learning model generates at least two corrected images from the at least two individual images, in which the edges and / or surface structures are emphasized and displayed and / or are displayed at another position, in order to make the difference smaller than the threshold value.

[0044] The camera system thus carries out the corresponding method steps.

[0045] In particular, the machine learning model may be an artificial neural network or such an artificial neural network may be used. The machine learning model can identify surface structures, clothing, patterns, parts of the face, eyes, teeth, etc. in the corresponding individual images by sufficient learning and can correlate them with each other in two individual images respectively. Subsequently, if the corresponding surface structures and / or edges are missing in one of the individual images, the machine learning model inserts them into the corresponding individual image afterwards or places them at another position.

[0046] Preferably, at least two consecutive individual images are at least one RGB image and at least one IR image. That is, for example, the check pattern of a shirt can be identified in the RGB image and transferred to the corresponding IR image for display. Similarly, for example, a transparent layer of clothing can be made opaque, or an edge that cannot be identified in the RGB image due to darkness, for example, can be transferred from the IR image to the corresponding overly dark RGB image and emphasized there.

[0047] Preferably, the computing unit is further designed to control the camera module and the infrared light source in order to change the second exposure time, the driving period of the infrared light source during the second detection process and / or the sensitivity of the IR exposure means according to the image statistics obtained by analyzing at least one RGB image, IR image and / or HDR image already generated.

[0048] The camera system executes corresponding method steps for this purpose.

[0049] The sensitivity of the IR exposure means is, for example, the ISO sensitivity. As image statistics, for example, a histogram regarding luminance, or an evaluation of whether it has the lowest or highest bin, i.e., an evaluation of whether individual pixels are underexposed or overexposed, can be created. The corresponding image statistics can be created for each of all individual images, i.e., for example, for all RGB images, or also for one or more regions of interest of the image. At this time, weighting for different regions of interest can also be performed. Thereby, the relevant image regions can be made particularly well distinguishable. By the second exposure time, the period during which the infrared light source operates, and also by affecting the sensitivity of the IR exposure means, it is possible to affect how brightly the individual structures of the scene can be identified in the IR image. Subsequently, as a region of interest, for example, an image section of the IR image where the eyes of a person driving a vehicle are located can be defined. Subsequently, in the corresponding IR image, image detection can be set so that the eyes of the person driving the vehicle can be identified with sufficient quality. For example, since the person driving the vehicle moves or the seating position shifts, the corresponding region of interest can also be adaptively shifted.

[0050] Since the exposure of the corresponding IR image can be adapted in consideration of the image statistics of the RGB image or the HDR image, the optimally available RGB image is also generated as much as possible. However, in a vehicle environment, the use of IR images is important for functions where safety is crucial, so the image metrics of the IR images are preferred. For example, since the second RGB image and the IR image are captured with the same exposure time, the second exposure time can be increased so as to prevent underexposure of the corresponding second RGB image. However, the second exposure time is only increased to such an extent that there is no risk of overexposure of the IR image. Furthermore, in order to additionally prevent overly strong exposure of the IR image, for example, the period during which the infrared light source is operated during the second detection process can be shortened. As a result, the second exposure time can be further increased. Similarly here, the sensitivity of the IR exposure means can be reduced, whereby the corresponding IR image additionally appears darker. This has a beneficial effect on the noise ratio of the IR image. The second exposure time must not be increased to such an extent that the safety of the detected human eyes is put at risk, the surface exposed to the infrared light source is overly heated, and / or the life of the infrared light source is overly affected.

[0051] In the method according to the invention for operating the aforementioned camera system, the following is done: - During at least one first detection process, the image sensor of the camera module is exposed with a first exposure time, and a first RGB image is generated using the RGB exposure means; - The infrared light source is activated, and during at least one second detection process, the scene is additionally illuminated with infrared light. During the second detection process, the image sensor of the camera module is exposed with a second exposure time different from the first exposure time, a second RGB image is generated using the RGB exposure means, and an IR image is generated using the IR exposure means; - The computing unit merges at least one first RGB image and at least one second RGB image into one HDR image.

[0052] Another advantageous configuration of the camera system according to the present invention and a method for operating the camera system will become apparent from the embodiments described in more detail below with reference to the respective figures.

Brief Description of the Drawings

[0053]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0054] FIG. 1 shows a camera system 1 according to the invention, integrated into a vehicle 9. The camera system 1 includes a camera module 2, a computing unit 4, and an infrared light source 7. The camera module 2 and the infrared light source 7 are arranged on and oriented with respect to the vehicle 9 such that, as a scene 3, it is detected by the camera module 2 or the interior of the vehicle 9 is illuminated with infrared light by the infrared light source 7, thereby enabling the detection of vehicle occupants. The computing unit 4 can be arranged at any position with respect to the camera module 2 and the infrared light source 7. For example, it may be integrated into a common unit or may be arranged dispersedly, whereby the computing unit 4 is formed by a computing unit of the vehicle 9, for example, a control device of a vehicle subsystem.

[0055] The camera module 2 includes an image sensor 5 having RGB exposure means 6.1 and IR exposure means 6.2. Each of the exposure means 6.1 and 6.2 is arranged in a two-dimensional matrix on the image sensor 5. When the image sensor 5 is exposed, a color image can be generated by the RGB exposure means 6.1, and this color image is also referred to as an RGB image RGB. See also the following drawings. Similarly, infrared light can be detected using the IR exposure means 6.2, and an IR image IR can be generated accordingly.

[0056] The camera system 1 can combine individual images that are continuously captured into a video stream, and can generate a color video with a series of RGB images, i.e., color images, and can also generate an infrared video with a series of IR images, i.e., images captured in the infrared spectrum. The color image or color video can be used by the vehicle occupants for entertainment, for example, for selfies or for making video calls. The IR image or IR video is processed by the vehicle 9, in particular, to provide vehicle functions related to safety, for example, to monitor the vital signs or attention of the person driving the vehicle. With illumination using the infrared light source 7, even when the visibility conditions are poor, for example, at night, a camera image with sufficient quality for evaluation can be captured.

[0057] The following figures show various examples of the sequence in which the camera system 1 performs any of the detection processes EV1, EV2, EV3. That is, the camera system 1 captures a continuous stream of individual images representing the sequences 8 of the respective detection processes EV1, EV2, EV3.

[0058] FIG. 2 shows a first example in which the first detection process EV1 and the second detection process EV2 are alternately performed. Here, the camera system 1 has a scanning speed f of n. KAMERA For example, at a scanning speed f of 60 images / second, KAMERA n = 60 Hz. During the first detection process EV1, the camera system 1 generates a first RGB image RGB with a first exposure time symbolically represented by 0. During the second detection process EV2, the camera system 1 simultaneously generates an IR image IR and a second RGB image RGB using the RGB exposure means 6.1 and the IR exposure means 6.2, in particular, with a second exposure time shorter than the first exposure time indicated by the numerical value -1. Here, both the IR image IR and the second RGB image RGB are generated with the same exposure time, i.e., the second exposure time described above.

[0059] To generate an HDR image, at least two RGB images with different exposure times are merged by the computing unit 4. For this purpose, two detection processes are combined to form one detection group EG. The detection group EG is symbolically represented by a dashed square in each figure.

[0060] In sequence 8, it doesn't matter which of the first detection process EV1 or the second detection process EV2 is performed first. In FIG. 2, boxes suitable for both variations are shown for each detection group EG.

[0061] Subsequently, the corresponding detected or generated images can be combined to generate one video stream. In the example in FIG. 2, the frame rate f IR of the IR video stream IR - VID is half of the scanning speed f KAMERA of the camera module 2, the frame rate f HDR of the HDR video stream HDR - VID is half of the scanning speed f KAMERA of the camera module 2, and the frame rate f RGB of the RGB video stream RGB - VID is half of the scanning speed f KAMERA of the camera module 2. However, here, since the images ordered successively alternate in brightness and darkness due to different exposure times, "flickering" occurs in the RGB video stream RGB - VID. Therefore, the visual output of the RGB video stream RGB - VID is not a preferred embodiment, but is generally possible.

[0062] FIG. 3 shows an alternative embodiment in which the third detection process EV3 is additionally implemented. Here, the image sensor 5 is exposed with a third exposure time, and a third RGB image RGB is taken using the RGB exposure means 6.1. The third exposure time may be longer or shorter than one of the other two exposure times. For example, the third exposure time may be longer than the first exposure time indicated by the number +1. The numbers +1, 0, and -1 represent the exposure times of the associated detection processes. For example, for 0, +1 represents a longer exposure time, -1 represents a shorter exposure time, and in particular, a very short exposure time suitable for infrared imaging. In contrast, in FIG. 5, the third exposure time of the detection process EV3 is shorter than the first exposure time of the detection process EV1, and thus, the two numbers 0 and +1 are reversed accordingly. That is, EV1 in FIG. 5 has the number 0 for the number 1 in FIG. 3, and EV3 in FIG. 5 has the number +1 for the number -1 in FIG. 3. By providing the third detection process, the dynamic range of the HDR image HDR can be expanded at the expense of the respective frame rates.

[0063] Here, in the embodiment illustrated in FIG. 3, each detection group EG includes one each of the first detection process EV1, the second detection process EV2, and the third detection process EV3.

[0064] FIG. 4 shows another alternative configuration in which, each time the detection process in which RGB images RGB are alternately generated using the exposure represented by 0 and the exposure represented by +1 is completed, an exposure time of -1, i.e., the exposure time for infrared imaging, is inserted. Thereby, the respective frame rates fIR, fHDR of the respective video streams IR-VID, HDR-VID can be improved.

[0065] To generate directly successive HDR images HDR, combinations of different exposure times are utilized, namely one combination of 0 and -1, and the other combination of +1 and -1. Generally, all three exposure times, namely -1, 0, and +1, are used for generating the HDR image HDR, and in that case, it is also possible to reduce the frame rate of the HDR video stream HDR-Vid to n / 3. This variation is not shown in any of the figures.

[0066] Figure 5 shows another alternative configuration in which the individual images generated by the camera system 1 are temporarily stored in a ring buffer at least temporarily. Older individual images are overwritten by more recent individual images for each detection cycle. For better understanding, in Figures 5 to 7, the detection groups EG for generating directly successive HDR images HDR in the ring buffer are shown gradually larger. That is, for example, the smallest illustrated detection group EG at time t0 is followed by the next larger detection group EG at time t1, for example, and that detection group EG is followed by the next larger detection group EG at time t3, for example, and the detection groups continue in a similar manner thereafter. The data respectively located on the left side of one of the detection groups within the detection groups can be correspondingly deleted from the ring buffer, and the emptied memory space in the ring buffer is filled with new data on the opposite side. As described above, in this Figure 5, the exposure time of the third detection process EV3 is shorter than the exposure time of the first detection process EV1.

[0067] After that, in order to generate the HDR image HDR, the most recent RGB image RGB in the ring buffer is always used each time, and the ring buffer can have only three memory spaces here, among other things. As shown in FIG. 5, this corresponds to the sliding detection group EG. Therefore, in order to generate the HDR image HDR, it is no longer necessary to completely capture each detection group EG. Rather, to form a new detection group and enable the generation of the HDR image HDR, it is sufficient to perform one of the other detection processes EV1, EV2, or EV3. In the case of a ring buffer with three memory spaces, for example, the first HDR image HDR is generated from the detection processes EV1, EV3, and EV2 shown at the left end. After the memory update, EV1 arranged at the far left of the memory is deleted, and instead, a new EV1 is stored on the right side, and the HDR image HDR is formed from the detection processes EV3, EV2, and the new EV1. In the next update, the left-side EV2 is deleted, and instead, a new EV3 is stored on the right side, and the HDR image HDR is formed from EV2, EV1, and the new EV3. Since the ring buffer is continuously updated, the next step corresponds to the first step described above again. Here, FIG. 5 shows a case equivalent to the case of FIG. 3, and each of the different detection processes EV1, EV2, and EV3 is performed only once before they are newly performed. The HDR image is generated from the detection processes EV1, EV2, and EV3 each time, that is, the exposure times of +1, 0, and -1 are alternately repeated. Different from the embodiment in FIG. 3, the frame rate f HDR of the camera module 2 can be improved to f KAMERA =n, and the frame rate of the infrared image is fir=n / 3.

[0068] Figure 6 shows another alternative configuration equivalent to the embodiment illustrated in Figure 4. Here, before and after the detection process in which only the RGB image RGB is captured using the exposure times of +1 and 0 alternately, a second detection process EV2, i.e., the simultaneous detection of the RGB image RGB and the IR image IR when the scene 3 is simultaneously illuminated using the infrared light source 7, is performed each time. Here, the detection group EG for generating the HDR image is obtained every two cycles instead of every one cycle as shown in Figure 5. A ring buffer is newly written every cycle, and again, the ring buffer has three memory spaces in particular, and the principle operation of the ring buffer corresponds to the operation in Figure 5. Thereby, the frame rate f IR of the IR video stream IR-VID can be improved from 1 / 3 to n / 2, but the frame rate f HDR of the HDR video stream HDR-VID decreases from n to n / 2.

[0069] Also, as shown in Figure 4, it is also possible to always merge two different exposure times, i.e., one with exposure times of 0 and -1 and the other with exposure times of +1 and -1, into one HDR image HDR. However, this may unnecessarily reduce the dynamic range.

[0070] FIG. 7 shows another possible embodiment in which the frame rate of the HDR video stream HDR-Vid can be improved to the value of the image detection rate of the camera module. Here, stored in the ring buffer, the series of detection processes that cycle consists of the second detection process EV2 (exposure time -1), followed by the third detection process EV3 (exposure time +1), again the second detection process EV2, and then the first detection process EV1 (exposure time 0). Here, the HDR image HDR is merged from the images taken using the three exposure times -1, 0, and +1, respectively, stored in the ring buffer for each detection cycle. Thus, the detection group EG extends substantially every cycle over the last three or four detection processes EV stored, and this detection group EG realizes the merge using the three aforementioned exposure times. Therefore, the ring buffer has, among other things, four memory spaces. In FIG. 7, in the initial state, the detection processes EV2, followed by EV1, EV2, and EV3, represented by the dashed line shown on the left, are stored in succession in the four memory spaces, and the HDR image HDR is formed from the detection group represented by the dashed line, including the detection processes EV3, EV2, and EV1. After the update and storage of new data, on the right side of the ring buffer, a new detection process EV2 is added and the left detection process EV2 represented by the dashed line is deleted, so in the ring buffer, EV1, EV2, EV3, and EV2 are stored in order from left to right. The HDR image HDR is generated from the latest EV2, EV3, and EV1 in the detection group EG represented by the dotted line. After the storage of new data, on the right side of the ring buffer, a new detection process EV1 is added and the older left detection process EV1 is deleted, and EV2, EV3, EV2, and EV1 are stored when viewed from the left. The HDR image HDR is generated from the detection processes EV3, EV2, and EV1 that are consecutive in succession when viewed from the left in the detection group EG represented by the dashed line. In subsequent cycles, a new EV2 is stored in the ring buffer and the older EV2 is erased. The ring buffer again includes EV3, EV2, EV1, and EV2 in succession from the left.The HDR image HDR is generated from the latest EV2, EV1, and EV3 in the detection group EG represented by the dotted line.

[0071] In the subsequent memory process, EV3 is stored and the older EV3 is deleted from the memory. Thus, the memory again has the first state described above, i.e., the detection processes EV2, EV1, EV2, and EV3 that follow each other from left to right. The HDR image HDR is generated in the detection group represented by the dashed line and consisting of EV3, EV2, and EV1. Starting from this first state, the memory process following the first state is repeated, and the HDR image HDR is generated as described above.

[0072] An IR image is taken each time the detection process EV2 is performed.

[0073] With the camera system 1 according to the present invention and the corresponding method for operating the camera system 1, it is possible to simultaneously generate an IR video stream IR-VID and an HDR video stream HDR-VID having a sufficient frame rate and a relatively high dynamic range based on the currently available image sensor 5 or camera module 2. By means of the active illumination during the taking of the infrared image, it is possible to take sufficiently distinguishable individual images, particularly when the light conditions are poor. By using a ring buffer, the frame rate of the HDR video stream HDR-VID can be further improved while maintaining the dynamic range. The frame rate f HDR of the HDR video stream HDR-VID corresponds to f KAMERA =n of the camera module 2, and the frame rate of the infrared image is fir=n / 2.

Claims

1. A camera system (1) comprising a camera module (2) for generating a camera image of a scene (3), and a computing unit (4) for controlling the camera module (2) and processing the camera image, wherein the camera module (2) is designed to detect light in the visible spectrum using RGB exposure means (6.1) and to detect light in the infrared spectrum using IR exposure means (6.2), and has an image sensor (5), and the camera module (2) is set to capture at least two individual images of the scene by exposing the image sensor (5) during at least two detection processes (EV1, EV2) using different exposure times, and the computing unit (4) is set to merge the at least two individual images into one HDR image, wherein the computing unit (4) further: - controls the camera module (2) to expose the image sensor (5) using light in the visible spectrum with a first exposure time during at least one first detection process (EV1) such that a first RGB image (RGB) is generated using the RGB exposure means (6.1); - controls an infrared light source (7) to illuminate the scene (3) using infrared light in addition to light in the visible spectrum during at least one second detection process (EV2), and at the same time controls the camera module (2) to expose the image sensor with a second exposure time different from the first exposure time during the second detection process (EV2); in the camera system (1), - the image sensor (5) simultaneously generates a second RGB image (RGB) using the RGB exposure means (6.1) and an IR image (IR) using the IR exposure means (6.2) during the second detection process; - the computing unit merges the at least one first RGB image (RGB) and the at least one second RGB image (RGB) into one HDR image (HDR), characterized in that the camera system (1).

2. The camera module (2) and the computing unit (4) are set to continuously detect the scene (3) in a sequence (8) of a plurality of detection processes including at least a first detection process (EV1) and a second detection process (EV2), and at least a first RGB image (RGB) and a second RGB image (RGB) taken continuously in time a plurality of times are integrated into one RGB video stream (RGB-VID), and IR images (IR) taken continuously in time a plurality of times are integrated into one IR video stream (IR-VID), and / or HDR images (HDR) generated continuously in time a plurality of times are integrated into one HDR video stream (HDR-VID). The camera system (1) according to claim 1, characterized in that.

3. The computing unit (4) further: - Exposes the image sensor (5) with a third exposure time different from the first exposure time and the second exposure time during at least one third detection process (EV3), and controls the camera module (2) so that a third RGB image (RGB) is generated using the RGB exposure means (6.1); - Sets the order of the detection processes to a first detection process (EV1), a second detection process (EV2), and a third detection process (EV3); - Merges the at least one first RGB image (RGB), the at least one second RGB image (RGB), and the at least one third RGB image (RGB) into one HDR image (HDR). The camera system (1) according to claim 1 or 2, characterized in that it is set as described above.

4. The computing unit (4) further: - Exposes the image sensor (5) with a third exposure time different from the first exposure time and the second exposure time during at least one third detection process (EV3), and controls the camera module (2) so that a third RGB image (RGB) is generated using the RGB exposure means (6.1); - Sets the order of the detection processes to a second detection process (EV2), a first detection process (EV1), a second detection process (EV2), and a third detection process (EV3); - Merge the at least one first RGB image (RGB) with at least one second RGB image (RGB), merge the at least one third RGB image (RGB) with at least one second RGB image (RGB), or merge the at least one first RGB image (RGB), the at least one second RGB image (RGB), and the at least one third RGB image (RGB) into one HDR image (HDR). The camera system (1) according to claim 1 or 2, characterized in that it is set as such.

5. The calculation unit (4) includes a ring buffer, and the ring buffer sequentially stores individual images generated successively in time. By adding the latest individual image each time to the ring buffer, the oldest individual image each time is deleted from the ring buffer, and at least the first RGB image (RGB), the second RGB image (RGB), and the IR image (IR) are set to be storable in the ring buffer. The calculation unit (4) is further set to merge at least the latest first RGB image (RGB) and second RGB image (RGB) each time into the HDR image in order to generate the HDR image (HDR). The camera system (1) according to any one of claims 1 to 4, characterized in that it is set as such.

6. The calculation unit (4) further sets the order of the detection process to - The first detection process (EV1); - The third detection process (EV3); and - The second detection process (EV2) and is set to merge the latest first RGB image (RGB), second RGB image (RGB), and third RGB image (RGB) each time into the HDR image. The camera system (1) according to claim 5, characterized in that it is set as such.

7. The calculation unit (4) further sets the order of the detection process to - The second detection process (EV2); - The first detection process (EV1); - The second detection process (EV2); and - The third detection process (EV3) and is set to merge the latest first RGB image (RGB), second RGB image (RGB), and third RGB image (RGB) each time into the HDR image. The camera system (1) according to claim 5, characterized in that it is set as such.

8. The calculation unit (4) is further configured to determine the moving speed of an object present in at least two consecutive individual images, compare the moving speed with a predetermined threshold value, and if the moving speed is greater than the threshold value, exclude the generated RGB image (RGB) for the corresponding detection process from the merging into the HDR image (HDR). The camera system (1) according to any one of claims 1 to 7, characterized in that.

9. The calculation unit (4) is further configured to identify at least one edge and / or the surface structure of an object present in at least two consecutive individual images, and compare the edge and / or the surface structure in each individual image with each other to identify differences. If the identified difference is greater than a predetermined threshold value, the generated RGB image (RGB) for the corresponding detection process is set to be excluded from the merging into the HDR image (HDR). The camera system (1) according to any one of claims 1 to 8, characterized in that.

10. The calculation unit (4) is further configured to execute a machine learning model in which at least two individual images can be read as input variables when the difference is greater than the threshold value. Subsequently, the machine learning model generates at least two individual images of a corrected image in which the edge and / or the surface structure are emphasized and displayed and / or displayed at another position in order to make the difference smaller than the threshold value. The camera system (1) according to claim 9, characterized in that.

11. The at least two consecutive individual images include at least one RGB image (RGB) and at least one IR image (IR). The camera system (1) according to any one of claims 8 to 10, characterized in that.

12. The calculation unit (4) is further configured to control the camera module (2) and the infrared light source (7) to change the second exposure time during the second detection process (EV2), the driving period of the infrared light source (7), and / or the sensitivity of the IR exposure means (6.2) according to the image statistics obtained by analyzing at least one already generated RGB image (RGB), IR image (IR), and / or HDR image (HDR). The camera system (1) according to any one of claims 1 to 11, characterized in that it is set.

13. - During at least one first detection process (EV1), with a first exposure time, the image sensor (5) of the camera module (2) is exposed to light in the visible region, and a first RGB image is generated using the RGB exposure means (6.1); - The infrared light source (7) is activated, and during at least one second detection process (EV2), in addition to the light in the visible spectrum, the scene (3) is illuminated with infrared light. During the second detection process (EV2), the image sensor (5) of the camera module (2) is exposed with a second exposure time different from the first exposure time. During the second detection process, the generation of a second RGB image (RGB) using the RGB exposure means (6.1) and the generation of an IR image (IR) using the IR exposure means (6.2) are performed simultaneously; - The method for operating the camera system (1) according to claim 1, characterized in that the calculation unit (4) merges the at least one first RGB image (RGB) and the at least one second RGB image (RGB) into one HDR image (HDR).

Images