System for generating an image stream in a vehicle, computer-readable storage medium

Abstract

System for generating an image stream in a vehicle, wherein the system comprises the following: - an IR radiation source (10); - an IR camera (20); - a windshield (30); and - one image processing unit (40), wherein the IR radiation source (10), the IR camera (20) and the windshield (30) are arranged such that first IR radiation (R1) emitted by the IR radiation source (10) falls into a detection area and second IR radiation (R2) generated by reflection of the first IR radiation (R1) in the detection area is reflected at the windshield (30) and detected by the IR camera (20), wherein the IR camera (20) is configured to generate an IR image stream (A) by detecting the second IR radiation (R2) which contains a plurality of IR images (A1, A2, A3), and wherein the image processing unit (40) is configured to generate an RGB image stream (B) based on the IR image stream (A), which contains a plurality of RGB images (B1, B2, B3), wherein the image processing unit (40) is configured to generate one or more corresponding RGB images (B1, B2, B3) for one or more of the IR images (A1, A2, A3), and wherein the image processing unit (40) is configured to capture position data from one or more reference points (u, v, w, x, y, z), in particular in a face, in at least one of the IR images (A1, A2, A3) and to generate at least one of the RGB images (B1, B2, B3) based on the position data.

Description

[0001] The invention relates to a system for generating a stream of images in a vehicle.

[0002] Modern vehicles often have an interior camera that can capture the person in the driver's seat. The image stream generated by the interior camera can be used in the vehicle for various purposes, such as video telephony or facial recognition for a driver assistance system.

[0003] To avoid obstructing the driver's view with a camera, solutions exist where the camera is mounted in the rearview mirror or center console. However, especially when using the recorded video stream for video calls, such solutions have the disadvantage that the driver is not filmed head-on. At least most of the time, the driver is looking forward at the road through the windshield, meaning that a camera positioned as described above is at an angle to the driver's line of sight. For the most natural video call experience, however, it is desirable for participants to look directly into the camera.

[0004] Furthermore, using a conventional video camera in a vehicle has the disadvantage that the image quality is not optimal due to the fluctuating lighting conditions while driving. For example, direct sunlight can cause an overexposed image, while driving through a tunnel can result in an underexposed image. Moreover, such extreme lighting conditions can change rapidly while driving.

[0005] From DE 10 2018 216 806 A1, a method and a corresponding device for processing an infrared image are known, particularly for use in a vehicle. The device comprises the following: an input module for reading an infrared image from an infrared camera; an image processing unit for generating an output image by applying a neural network to the infrared image, wherein the neural network has been trained to filter shadows or reduce lighting effects; and an output for displaying the output image. For use of the device in a vehicle, it is provided that the infrared camera is arranged in the dashboard so that the person is photographed from below and thus looks past the camera when looking at the road.In this context, it is suggested to train the neural network using frontal views or to correct the perspective of the recorded infrared images using image processing.

[0006] WO 2022 / 075184 A1 discloses a reflective film for reflecting infrared radiation. The reflective film can be arranged, in particular, on a vehicle's windshield to reflect infrared radiation emitted by a radiation source towards the driver and to reflect the radiation reflected there towards an IR sensor located below the windshield.

[0007] US Patent 2008 / 069403 A1 discloses a vehicle image acquisition system comprising an infrared source that generates an infrared beam along a field of view, as well as an infrared detector and a processing unit. The infrared detector is designed to detect infrared radiation reflected from objects within the field of view and to generate image signals from it. The processing unit is designed to perform facial feature recognition and, depending on the results, trigger a process within the vehicle. The infrared detector can be positioned below the windshield, which serves as a reflective surface.

[0008] US 2020 / 143184 A1 discloses a head-up display arrangement for a motor vehicle comprising the following: a windshield with a coating that reflects infrared energy; a light source arranged to emit a field of light such that the field of light is reflected by the windshield and appears to the driver as a virtual image outside the windshield; and an IR camera positioned and configured to receive infrared energy reflected from the driver's face and reflected by the windshield coating.

[0009] Against this background, it is an object of the present invention to provide an improved solution for video recording in a vehicle.

[0010] The problem is solved by a system according to claim 1.

[0011] In particular, the task is solved by a system for generating a stream of images in a vehicle. The system has the following features: - an IR radiation source; - an IR camera; - a windshield; and - an image processing unit.

[0012] The IR radiation source, the IR camera, and the windshield are arranged in such a way that the first IR radiation emitted by the IR radiation source hits a detection area, and the second IR radiation, which is created by reflection of the first IR radiation in the detection area, is reflected at the windshield and detected by the IR camera.

[0013] The IR camera is designed to generate an IR image stream containing a multitude of IR images by capturing the second IR radiation.

[0014] The image processing unit is designed to generate an RGB image stream based on the IR image stream, containing a multitude of RGB images.

[0015] Furthermore, the image processing unit is designed to generate one or more corresponding RGB images for one or more of the IR images.

[0016] Furthermore, the image processing unit is designed to capture position data from one or more reference points, in particular in a face, in at least one of the IR images and to generate at least one of the RGB images based on the position data.

[0017] One aspect of the present invention is to record the objects in the camera's detection range, for example the person in the driver's seat, not by detecting visible light, but by detecting infrared (IR) radiation with the IR camera.

[0018] In the context of this application, infrared radiation can be understood to mean, in particular, radiation with a wavelength in the range of 800 to 1500 nm.

[0019] To improve the quality of the captured IR images, the system uses an IR radiation source to illuminate the detection area with IR radiation. This occurs invisibly to the vehicle occupants and therefore not without some impairment. The IR radiation emitted by the source is reflected within the detection area, for example, off the driver's face, towards the windshield. In the context of this system, the radiation emitted by the source is referred to as first IR radiation, and the IR radiation reflected within the detection area is called second IR radiation.

[0020] Preferably, the windshield is essentially transparent to visible light and exhibits a high reflectivity for IR radiation on its inner surface, at least in one area. High reflectivity can be understood, for example, as a reflectance of 50% or higher. Optionally, the inner surface of the windshield may have a corresponding functional coating.

[0021] The reflectivity of the windshield allows the IR camera to be positioned and aligned in such a way that it captures the secondary IR radiation reflected from the windshield. In this way, the IR camera captures an IR image stream containing a multitude of IR images.

[0022] The IR radiation source can also be positioned so that the initial IR radiation is reflected off the windshield before entering the detection area. In particular, the IR radiation source can be integrated into the IR camera. Alternatively, the IR radiation source can be positioned separately and directed straight towards the detection area, for example, in the vehicle's rearview mirror.

[0023] Finally, the system according to the invention provides an image processing unit configured to generate an RGB image stream, i.e., an image stream with color information, based on the captured IR image stream. This can be achieved, for example, by colorizing the individual IR images. While the IR images only indicate intensity values ​​and thus have a grayscale appearance, the generated RGB image stream can have the appearance of a conventional color image stream, which can be used, for example, for video telephony or video conferencing.

[0024] In the context of this invention, an image stream can be understood to mean any data that specifies a temporally ordered sequence of images.

[0025] The inventive approach offers several advantages over a conventional system in which an RGB camera is located in the vehicle. Firstly, the detection of the reflected second IR radiation enables a virtual camera position that is directly in front of the detection area, i.e., in the driver's line of sight, even though the actual camera position is outside the driver's field of vision, for example, in an upper area of ​​the dashboard. In the detected IR image stream and the resulting RGB image stream, the driver thus appears to be looking directly at the camera, without their field of vision being obstructed by a camera positioned there. This increases driving safety and improves visual comfort in the vehicle.

[0026] Furthermore, the detection of IR radiation has the advantage that its spectrum lies outside the spectrum of visible light, making the captured IR images more robust against interfering light influences during driving (see above). The captured IR images and the RGB images generated from them are therefore of higher quality.

[0027] The image processing unit is designed to generate one or more corresponding RGB images for one or more of the IR images. In particular, the image processing unit can generate the RGB image stream on a frame-by-frame basis, i.e., for each IR image in the IR image stream, it can generate a corresponding RGB image within the RGB image stream. Thus, the image processing unit essentially processes individual IR images.

[0028] Alternatively, the image processing unit can also be designed to create an RGB image from several IR images, or to create several RGB images from a single IR image. This can improve the quality and / or realism of the generated RGB images.

[0029] Optionally, the image processing unit can perform further image optimization steps on the generated RGB images, such as adding depth of field, adjusting brightness and / or contrast, etc.

[0030] In one embodiment, the image processing unit can include an AI module and be configured to generate the RGB image stream using the AI ​​module.

[0031] Various techniques from the field of artificial intelligence (AI) are known to efficiently perform different image processing tasks (e.g., object recognition, image optimization, colorization) and deliver high-quality results. For example, the AI ​​module of the image processing unit can be an autoencoder that is adapted (trained) to generate a grayscale image from an RGB image.

[0032] The AI ​​module can be trained in particular by a supervised learning process, where the training data contains image pairs constructed as follows: Each image pair has an IR image from an IR camera as described above, and an RGB image from an RGB camera, which is positioned at the virtual camera position of the IR camera, i.e. frontally in front of the detection area.

[0033] If the RGB images or RGB image stream are generated by an AI module as described and trained above, the generated RGB image stream can exhibit high image quality and a high degree of realism.

[0034] In one embodiment, the image processing unit can be configured to generate at least one of the RGB images by colorizing at least one of the IR images.

[0035] In this embodiment, color information missing from the IR images of the IR image stream is subsequently added by colorizing one or more IR images. The colorizing, i.e., the conversion of the RGB image to a grayscale image, can be performed, in particular, by an AI module as described above.

[0036] In one embodiment, the image processing unit can be configured to capture an image area in at least one of the IR images and to generate at least one of the RGB images based on the image area.

[0037] The image area can be defined, in particular, by a bounding box (e.g., rectangular or elliptical) that encompasses a relevant portion of the IR image. This relevant area might, for example, include the head or upper body of the person in the driver's seat, but not background elements (e.g., the driver's seat, the vehicle interior). In subsequent image processing, the image processing unit can then process only the captured image area instead of the entire IR image.

[0038] For example, the image processing unit can be configured to extract the head or upper body area from an IR image and generate a color information area for the RGB image from this image area, i.e., to color only the captured image area. The RGB image can then be generated by placing the colored image area against a virtual background.

[0039] The above embodiment particularly exploits the fact that, due to the statically positioned IR camera, only some image elements change during recording (e.g., facial expressions and gestures of the person being recorded), while other image elements remain essentially the same (e.g., driver's seat, vehicle interior). By capturing a relevant image area and reducing image processing to this area, the amount of image data to be processed can be reduced. This leads to a more efficient system, especially with regard to resource requirements.

[0040] The image processing unit is designed to capture position data from one or more reference points (landmarks) in at least one of the IR images and to generate at least one of the RGB images based on the position data.

[0041] In particular, the reference points can specify prominent facial features such as the eyes, mouth, nose, and eyebrows. Such reference point detection can be achieved, for example, using the Dlib 68 software library, which detects 68 different marker points on the face and classifies them, optionally using a trained classifier.

[0042] The position data can be specified, in particular, as vectors in a two-dimensional coordinate system.

[0043] To generate a corresponding RGB image, an avatar (i.e., a virtual representation of a person) can be configured according to the captured positional data and, if necessary, placed in front of a virtual background. The avatar can then be adapted to the person shown in the IR image, for example, through appropriate transformations regarding facial expressions, position, and / or orientation, based on the captured positional data.

[0044] Alternatively, the RGB image can be generated by performing the transformations described above not on an avatar, but on a previously generated colored IR image. This preserves the realistic overall impression of the previously generated RGB image, with only adjustments made to facial expressions, position, and / or orientation.

[0045] According to this embodiment as well, the amount of data to be processed can be reduced, since the RGB image is generated based on the position data (vectors) and not on the basis of the complete image data of the IR image.

[0046] In one embodiment, the system can further include an RGB camera. The RGB camera can be configured to capture at least part of the detection area, possibly from a different angle, and to generate a second RGB image stream. In particular, the RGB camera can be positioned in the vehicle such that it is directed towards the driver but is outside their field of vision (e.g., in the rearview mirror).

[0047] The image processing unit can be configured to generate the RGB image stream using at least portions of the second RGB image stream. In particular, the image processing unit can colorize an IR image using a corresponding RGB image from the second RGB image stream. Preferably, the corresponding RGB image is acquired at essentially the same time as the IR image. This offers the advantage that the color information of the corresponding RGB image from the second image stream can be used to colorize the IR image. Thus, a higher-quality (more realistic) RGB image can be generated.

[0048] For all the image processing device features described above, it is true that (in corresponding embodiments) the AI ​​module described above can be configured to perform the corresponding image processing operations.

[0049] In one embodiment, the system can be implemented as a distributed system. In particular, the image processing unit can be provided by a (vehicle-external) server, which is connected to a (vehicle-internal) vehicle communication unit, especially via a mobile communication connection. The vehicle communication unit is configured to transmit the IR image stream completely or partially to the server.

[0050] This embodiment allows computationally intensive image processing operations to be offloaded to an external server, with the IR image stream being transmitted (in whole or in part) from the vehicle's communication unit to the server. This reduces the resource requirements (in terms of computing capacity) within the vehicle.

[0051] Preferably, in this embodiment, the system further comprises an image preprocessing unit in the vehicle. The image preprocessing unit can be configured to preprocess the IR image stream before transmission to the server. This preprocessing can include compressing the IR images, for example, using a suitable image data compression method. Compression can also be achieved by capturing reference points, as described above, and reducing the IR image stream to the position data of these reference points. In this case, only the position data, not the complete image data, is transmitted, while the server then generates complete image data based on the transmitted position data.

[0052] This approach can reduce the amount of data that needs to be transmitted via the communication link between the vehicle and the server.

[0053] A procedure can be executed, in particular, by a system as described above. The procedure comprises the following steps: - Detection of IR radiation to generate an IR image stream, in particular by an IR camera, wherein the IR image stream contains a plurality of IR images; - Generating a multitude of RGB images based on at least some of the IR images, in particular by an image processing unit, comprising determining positional data from one or more reference points in one of the IR images, in particular in a face; and - Generating an RGB image stream containing the multitude of RGB images, particularly through the image processing unit.

[0054] In one embodiment of the method, the multitude of RGB images can be generated by an AI module. The AI ​​module can, in particular, comprise an artificial neural network, for example, an autoencoder.

[0055] In one embodiment of the method, generating the multitude of RGB images may comprise at least one of the following steps: - Colorize at least one of the IR images; - Determining and / or generating an image section in at least one of the IR images; and - Determining positional data from one or more reference points in one of the IR images, especially in a face.

[0056] The method and the corresponding embodiments offer similar advantages and technical effects as those described in connection with the system and the corresponding embodiments.

[0057] A computer-readable storage medium contains instructions that, when executed by the at least one processor, cause at least one processor to implement a procedure as described above.

[0058] The computer-readable storage medium offers similar advantages and technical effects as those described in connection with the system and the method.

[0059] It goes without saying that the features and the advantages achievable with them, which have been described with regard to the system according to the invention, are applicable or transferable to the process and vice versa. Specifically, the components of the system, in the context of the present description of the invention, are designed to carry out the process steps according to the invention. Likewise, the functions of the components of the system according to the invention described above can be used as process steps of the process according to the invention.

[0060] The invention is described below with reference to exemplary embodiments, which are explained in more detail with reference to the figures. These show: Fig. 1: a system according to the invention in an exemplary embodiment; Fig. 2: a system according to the invention in an alternative embodiment; Fig. 3a: a flowchart of a process according to an exemplary embodiment; Fig. 3b: a single-image-based generation of an RGB image stream according to an exemplary embodiment; Fig. 4: generating an RGB image by colorizing an IR image; Fig. 5: generating an RGB image from an image area of ​​an IR image; Fig. 6: generating an RGB image by capturing reference points in an IR image.

[0061] In the following description, the same reference numbers are used for identical and equivalent parts.

[0062] Fig. Figure 1 shows an embodiment of a system according to the invention, which is arranged in a vehicle. The system comprises the IR radiation source 10, the IR camera 20, the windshield 30 and the image processing unit 40.

[0063] The IR radiation source 10 is positioned and oriented so that it illuminates the upper body of the driver 50 from the front. For this purpose, the IR radiation source 10 can, for example, be located in an interior rearview mirror (not shown) of the vehicle, so that it does not further impair the driver's 50 field of vision. It should be noted here that, unlike with a video camera mounted in the interior rearview mirror, a lateral offset from the driver's 50 line of sight is generally not a problem with the IR radiation source 10.

[0064] The first IR radiation R1 emitted by the IR radiation source 10 is reflected in the vehicle interior and by the driver's body 50 and strikes the windshield 30 as the second IR radiation R2. At the windshield 30, the second IR radiation R2 is reflected again and strikes a sensor of the IR camera 20, which is located in the area of ​​the dashboard.

[0065] This arrangement creates the impression in the image stream recorded by the IR camera 20 that the driver 50 is looking directly into the IR camera 20, even though his gaze is directed forward towards the windshield 30. At the same time, the IR camera 20 is not in the driver's 50 direct field of vision, so the driver 50 has an unobstructed view of the road through the windshield 30.

[0066] The IR camera 20 provides the captured IR image stream to the image processing unit 40. For this purpose, the IR camera 20 is connected to the image processing unit 40 via a data line in the vehicle. The image processing unit 40 has the AI ​​module 41, which is configured to generate the corresponding RGB images of the RGB image stream from the IR images of the IR image stream. This can be done, in particular, according to one of the methods described in the Fig. 3 to 6 are described.

[0067] Fig. Figure 2 shows a further embodiment of the system according to the invention. In contrast to the embodiment of the Fig. 1 Here, the IR radiation source 10 is integrated into the IR camera 20, which, as in the exemplary embodiment of the Fig. 1 is arranged in an area of ​​the dashboard. In this arrangement, the first IR radiation R1 emitted by the IR radiation source 10 is reflected at the windshield 30 in the direction of the driver 50. Through reflection of the first IR radiation R1 in the interior of the vehicle, second IR radiation R2 is generated, which is in turn reflected at the windshield 30, as in conjunction with the embodiment of the Fig. 1 explained. The other components of the system are as in the exemplary embodiment of the Fig. 1 configured and arranged.

[0068] Fig. Figure 3a shows a flowchart of the procedure according to an exemplary embodiment.

[0069] In step S1, (second) IR radiation is detected by an IR camera and the corresponding IR images A1, A2 and A3 (or the corresponding IR image stream) are stored in a memory.

[0070] In step S2, the IR image stream A, which contains the IR images A1, A2, and A3, is transferred to an image processing unit. In the exemplary embodiment of the Fig. 1 and Fig. 2. This can be achieved by transmitting the IR images A1, A2, A3 or the IR image stream A from the IR camera 20 to the image processing unit 40 via a data line in the vehicle. Alternatively, the image processing unit 40 can be implemented on a remote server. In this case, step S2 includes transmission to the server using an in-vehicle communication device, for example, via a mobile network.

[0071] In step S3, which is executed by the image processing unit, the RGB images B1, B2, and B3 are generated. According to this embodiment, one RGB image is generated for each of the IR images A1, A2, and A3 (as shown below using the following example). Fig. 3b explained).

[0072] In step S4, the generated RGB images B1, B2 and B3 are provided as RGB image stream B for further use, in particular to an interface of a video telephony module or a driver assistance system.

[0073] Fig. Figure 3b illustrates the single-frame generation of the RGB image stream B. According to the illustrated embodiment, the IR image stream A contains the three IR images A1, A2, and A3. The corresponding RGB images B1, B2, and B3 are generated for each IR image A1, A2, and A3. This can be achieved using one of the methods described below. Fig. Sections 4 to 6 will be explained. The RGB images B1, B2, and B3 generated in this way constitute the RGB image stream B.

[0074] Fig. Figure 4 shows the IR image A1 and the RGB image B1 generated by colorization. In this embodiment, the RGB image B1 was generated by an artificial neural network, which is adapted to provide a corresponding color image (RGB image) for a grayscale image, i.e., to colorize the grayscale image. As can be seen in the RGB image B1, the colorization gives at least the face of the depicted person a natural color tone.

[0075] Fig. Figure 5 shows another embodiment for generating the RGB image B1 from the IR image A1. According to this embodiment, the contour of the depicted person is first captured in the IR image A1, and the resulting image area A1' is extracted from the IR image A1. The image area A1' also represents an IR image, but its data volume is reduced compared to the IR image A1. A colored image area B1' is then generated from the image area A1, as described in the context of Fig. As described in section 4, the resulting RGB image B1 is obtained by superimposing the virtual background image C with the image area B1'. Preferably, the virtual background image C is an RGB image that shows a plausible background (e.g., a vehicle interior).

[0076] Fig. Figure 6 shows another embodiment for generating the RGB image B1 from the IR image A1. According to this embodiment, the position data of five reference points u, v, x, y and z are determined in the IR image A1, which indicate the position of the eyes, the corners of the mouth and the chin of the depicted person.

[0077] Based on the captured positional data, avatar D is configured, serving as a virtual representation of the person depicted in IR image A1. Avatar D can initially exist as a two- or three-dimensional model in a standard orientation. Configuring avatar D involves transformations so that the positions of the reference points u, v, x, y, and z in avatar D correspond to the captured positional data of the reference points u, v, x, y, and z in IR image A1. In this way, avatar D can be adapted to the person depicted in IR image A1 through configuration of facial expressions, orientation, and position.

[0078] Based on the configured avatar D, the RGB image B1 is generated by overlaying the virtual background image C with the configured avatar D (as described above in connection with Fig. 5 described).

[0079] The units described in this description by the terms "module", "unit", "component", etc., can in principle be implemented in either hardware or software, regardless of how this is described. Likewise, the described units can be combined or implemented separately without altering the essence of the invention.

[0080] It goes without saying that the systems and methods described above can be used not only to capture an image stream showing the person in the driver's seat, but also to capture the front interior of the vehicle in general.

[0081] For the sake of simplicity, the exemplary embodiments of the Fig. 1 and Fig. Two systems are described in which all components, including the image processing unit, are located in the vehicle. As previously described, the image processing unit can also be provided by a backend server. For this purpose, the data acquired in the vehicle, in particular the IR images, are transmitted to the backend server via a suitable vehicle communication device.

[0082] Further aspects of the invention are as follows: 1. System for generating a stream of images in a vehicle, wherein the system comprises the following: - an IR radiation source (10); - an IR camera (20); - a windshield (30); and - one image processing unit (40), wherein the IR radiation source (10), the IR camera (20) and the windshield (30) are arranged such that first IR radiation (R1) emitted by the IR radiation source (10) falls into a detection area and second IR radiation (R2) generated by reflection of the first IR radiation (R1) in the detection area is reflected at the windshield (30) and detected by the IR camera (20), wherein the IR camera (20) is configured to generate an IR image stream (A) by detecting the second IR radiation (R2) which contains a plurality of IR images (A1, A2, A3), and wherein the image processing unit (40) is configured to generate an RGB image stream (B) based on the IR image stream (A), which contains a plurality of RGB images (B1, B2, B3). 2. System according to aspect 1, wherein the image processing unit (40) is configured to generate one or more corresponding RGB images (B1, B2, B3) for one or more of the IR images (A1, A2, A3). 3. System according to one of the preceding aspects, wherein the image processing unit (40) has an AI module (41) and is configured to generate the RGB image stream (B) using the AI ​​module (41). 4. System according to one of the previous aspects, in particular according to aspect 2, wherein the image processing unit (40) is configured to produce at least one of the RGB images (B1, B2, B3) by colorizing at least one of the IR images (A1, A2, A3). 5. System according to one of the previous aspects, in particular according to aspect 2, wherein the image processing unit (40) is configured to detect an image area (A1') in at least one of the IR images (A1, A2, A3) and to generate at least one of the RGB images (B1, B2, B3) based on the image area (A1'). 6. System according to one of the previous aspects, in particular according to aspect 2, wherein the image processing unit (40) is configured to capture position data from one or more reference points (u, v, w, x, y, z), in particular in a face, in at least one of the IR images (A1, A2, A3) and to generate at least one of the RGB images (B1, B2, B3) based on the position data. 7. System according to one of the previous aspects, which furthermore includes an RGB camera designed to capture at least part of the detection area and to generate a second RGB image stream, and wherein the image processing unit (40) is configured to generate the RGB image stream (B) using at least parts of the second RGB image stream. 8. System according to one of the previous aspects, wherein the image processing unit (40) is provided by a server which is connected to a vehicle communication unit, in particular by means of a mobile communication connection, wherein the vehicle communication unit is located in the vehicle and is designed to transmit the IR image stream (A) completely or partially to the server. 9. System according to one of the previous aspects, in particular according to aspect 8, wherein the system further comprises an image preprocessing unit in the vehicle which is configured to perform preprocessing of the IR image stream (A) before transmission to the server, in particular by compressing at least one of the IR images (A1, A2, A3), for example by reducing to position information of reference points (u, v, w, x, y, z). 10. Method for generating an image stream in a vehicle, in particular by a system according to any of the preceding aspects, wherein the method comprises the following steps: - Detection of IR radiation to generate an IR image stream (A), in particular by an IR camera (20), wherein the IR image stream (A) contains a plurality of IR images (A1, A2, A3); - Generating a multitude of RGB images (B1, B2, B3) based on at least some of the IR images, in particular by an image processing unit (40); and - Generating an RGB image stream (B) containing the multitude of RGB images, in particular by the image processing unit (40). 11. Method according to aspect 10, wherein the multitude of RGB images (B1, B2, B3) is generated by an AI module (41), in particular an artificial neural network. 12. Method according to aspect 10 or 11, wherein generating the plurality of RGB images (B1, B2, B3) comprises at least one of the following steps: - Colorize at least one of the IR images (A1, A2, A3); - Determining and / or generating an image section (A1') in at least one of the IR images (A1, A2, A3); and - Determining positional data from one or more reference points (u, w, x, y, z) in one of the IR images (A1, A2, A3), especially in a face. 13. Computer-readable storage medium containing instructions that cause at least one processor to implement a method according to any of aspects 10 to 12 when the instructions are executed by the at least one processor.

[0083] It should be noted that all the parts described above, individually—even without additional features described in the respective context, even if these have not been explicitly identified as optional features in the respective context, e.g., by using: in particular, preferably, for example, e.g., parentheses, etc.—and in combination or any sub-combination, are to be regarded as independent embodiments or further developments of the invention as defined in particular in the introduction and the claims. Deviations from this are possible. Specifically, it should be noted that the word "in particular" or parentheses do not denote features that are mandatory in the respective context. Reference symbol list 10 IR radiation source 20 IR cameras 30 Windscreen 40 image processing units 41 Kl-Modul 50 drivers A IR image stream A1, A2, A3 IR image A1', B1' Image area B RGB image stream B1, B2, B3 RGB image C virtual background image The Avatar R1 first IR radiation R2 second IR radiation S1 Acquisition of the IR images of the IR image stream S2 Transferring the IR image stream to the image processing unit S3 Generating the RGB images S4 Providing the RGB image stream u, v, x, y, z reference points QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2018 216 806 A1

[0005] WO 2022 / 075184 A1

[0006] US 2008 / 069403 A1

[0007] US 2020 / 143184 A1

[0008]

Claims

[1] System for generating a stream of images in a vehicle, wherein the system comprises the following: - an IR radiation source (10); - an IR camera (20); - a windshield (30); and - one image processing unit (40), wherein the IR radiation source (10), the IR camera (20) and the windshield (30) are arranged such that first IR radiation (R1) emitted by the IR radiation source (10) falls into a detection area and second IR radiation (R2) generated by reflection of the first IR radiation (R1) in the detection area is reflected at the windshield (30) and detected by the IR camera (20), wherein the IR camera (20) is configured to generate an IR image stream (A) by detecting the second IR radiation (R2) which contains a plurality of IR images (A1, A2, A3), and wherein the image processing unit (40) is configured to generate an RGB image stream (B) based on the IR image stream (A), which contains a plurality of RGB images (B1, B2, B3), wherein the image processing unit (40) is configured to generate one or more corresponding RGB images (B1, B2, B3) for one or more of the IR images (A1, A2, A3), and wherein the image processing unit (40) is configured to capture position data from one or more reference points (u, v, w, x, y, z), in particular in a face, in at least one of the IR images (A1, A2, A3) and to generate at least one of the RGB images (B1, B2, B3) based on the position data. [2] System according to claim 1, wherein one or more reference points (u, v, w, x, y, z) indicate facial features such as eyes, mouth, nose and eyebrows. [3] System according to one of the preceding claims, wherein the RGB image (B1, B2, B3) is generated by configuring an avatar according to the captured position data, in particular by transformation with respect to facial expression, position and / or orientation according to the position data. [4] System according to one of the preceding claims, wherein the RGB image (B1, B2, B3) is generated by configuring a colored IR image (A1, A2, A3) according to the acquired position data, in particular by transformation with respect to expression, position and / or orientation according to the position data. [5] System according to any of the preceding claims, wherein the image processing unit (40) comprises an AI module (41) and is configured to generate the RGB image stream (B) using the AI ​​module (41). [6] System according to one of the preceding claims, wherein the image processing unit (40) is configured to generate at least one of the RGB images (B1, B2, B3) by colorizing at least one of the IR images (A1, A2, A3). [7] System according to one of the preceding claims, wherein the image processing unit (40) is configured to detect an image area (A1') in at least one of the IR images (A1, A2, A3) and to generate at least one of the RGB images (B1, B2, B3) based on the image area (A1'). [8] System according to any of the preceding claims, in particular according to claim 7, wherein the image area (A1') is defined by a bounding box which includes a head and / or upper body of a person. [9] System according to any of the preceding claims, in particular according to claim 7 or claim 8, wherein the image processing unit (40) processes exclusively the image area (A1') instead of the corresponding IR image (A1, A2, A3). [10] System according to any of the preceding claims, in particular according to any of claims 7 to 9, wherein the corresponding RGB image (B1, B2, B3) is created by coloring the image area (A1') and placing it in front of a virtual background. [11] System according to any of the preceding claims, which furthermore includes an RGB camera designed to capture at least part of the detection area and to generate a second RGB image stream, and wherein the image processing unit (40) is configured to generate the RGB image stream (B) using at least parts of the second RGB image stream. [12] System according to one of the preceding claims, in particular according to claim 11, wherein the image processing unit (40) is configured to perform colorization of an IR image (A1, A2, A3) using a corresponding RGB image from the second RGB image stream. [13] System according to any of the preceding claims, in particular according to claim 11 or 12, wherein the corresponding RGB image is recorded at the same time as the IR image. [14] System according to any of the preceding claims, wherein the image processing unit (40) is provided by a server which is connected to a vehicle communication unit, in particular by means of a mobile communication connection, wherein the vehicle communication unit is located in the vehicle and is designed to transmit the IR image stream (A) completely or partially to the server. [15] System according to one of the preceding claims, in particular according to claim 14, wherein the system further comprises an image preprocessing unit in the vehicle which is configured to perform preprocessing of the IR image stream (A) before transmission to the server, in particular by compressing at least one of the IR images (A1, A2, A3), for example by reducing to position information of reference points (u, v, w, x, y, z).

Images