Interior camera

Abstract

The invention relates to an interior camera (1) for monitoring a vehicle interior (2), having an image sensor (6) and a metalens (9) which interacts with the image sensor (6) in order to influence the optical imaging, and having an image processor (7) which interacts with the image sensor (6). The interior camera system according to the invention is characterised in that the metalens is designed to alter optical frequencies such that an image captured by the image sensor via the metalens can be manipulated by filtering-out, shifting and / or distorting in such a way that it is unrecognisable to a human eye and / or a machine-learning device. (Figure 1) 17

Description

[0001] Interior camera

[0002] The invention relates to an interior camera according to the type defined in the preamble of claim 1.

[0003] Interior cameras for monitoring vehicle interiors are well known in the art. Interior cameras are used for various purposes, such as detecting the presence of people inside the vehicle or their assignment to specific seats. Interior cameras are also used by vehicle assistance systems, which, for example, monitor the attention level of a person driving the vehicle or detect gestures that can be used to control functions in the vehicle.

[0004] As always in vehicle applications, the installation space required by the interior cameras plays a significant role. In this context, DE 10 2020 132 427 A1 describes a vehicle camera unit for the interior of a vehicle. The focus here is on keeping the size as small as possible, which is why at least one diffractive optical element is used instead of conventional optical elements. This allows the installation space to be minimized in the direction of the incident radiation. Further minimization is possible by using so-called metamaterials to form or supplement the diffractive optical element.

[0005] In addition to this vehicle-specific installation space requirement, another problem lies in protecting the privacy of the people inside the vehicle. The camera images are recorded and analyzed in the vehicle, possibly stored, and in some cases also transmitted to external servers, such as the backend server of a vehicle manufacturer, for analysis. An efficient way to ensure the privacy of the people inside the vehicle without having to compromise the quality of the functions that involve images from the vehicle's interior camera would therefore be desirable.

[0006] DE 1 02020 204 362 A1 discloses a method for evaluating a gesture of a vehicle occupant in an interior of a vehicle, in which the gesture is detected using a TOF (Time-of-Flight) camera in the interior.

[0007] From DE 102021 1 01 805 A1 a method for training an artificial neural network is known, which is designed to determine the level of attention of a driver of a motor vehicle.

[0008] US 2019 / 0 361 222 A1 discloses an optical system that uses an ultra-thin, flat metal lens to increase system functionality. The optical system is configured to modify and alter a captured image.

[0009] Accordingly, the object of the present invention is to provide an improved interior camera for monitoring a vehicle interior, which can combine high functionality with efficient protection of privacy.

[0010] According to the invention, this object is achieved by an interior camera having the features in claim 1, and in particular in the characterizing part of claim 1. Advantageous embodiments and further developments of the interior camera according to the invention emerge from the dependent claims.

[0011] The interior camera according to the invention for monitoring a vehicle interior comprises an image sensor and a metal lens interacting with this image sensor to influence the optical image, similar to the prior art mentioned above. The data captured by the image sensor is then further processed by an image processor.

[0012] In the interior camera, the metal lens is designed to change optical frequencies so that an image captured by the image sensor via the metal lens is falsified by filtering, shifting and / or distorting in such a way that it is unrecognizable for a human eye and / or a machine reading device, e.g. for automated personal identification.

[0013] A metalens is an optical element whose refractive properties are not determined by the transition of light from one medium to another, as is the case with conventional lenses, but rather by electromagnetic resonances on an artificially created grid of electrically conductive and dielectric elements. This grid can be precisely designed during development to achieve a specific optical effect. One design variant of such a metalens could, for example, be the targeted filtering out of edges (high-pass filter). Another design variant could be the targeted filtering out of shadows in the image (low-pass filter). To achieve this, the resonances of the elements of the metalens are then shifted to the corresponding optical frequencies.

[0014] Such a metal lens is used to modify the image captured by the image sensor in such a way that the privacy of the people inside the vehicle is maintained. Filtering out, shifting, or distorting the image distorts it to such an extent that people are no longer identifiable.

[0015] The interior camera according to the invention becomes particularly cost-effective and efficient if, according to a very advantageous development, its image processor is set up to convert a distorted image into an undistorted state by means of a transfer function of the metal lens that is stored in the image processor and optionally assigned to the level of distortion, and to output this image. The idea behind this is that the distortion via the metal lens occurs, for example, via a filter that only allows high frequencies, i.e. ultimately edges, or low frequencies, i.e. ultimately shadows, to pass through. Alternatively, a corresponding optical function could be stored in the metal lens, which overlays the undistorted image with a function implemented in the metal lens. If the filter property or this overlay, i.e. ultimately the function implemented with the metal lens orIf the transmission occurring at the selected level of corruption is known, it can be extracted from the corrupted image by deconvolution in the image processor, so that the image processor has an uncorrupted image. Using such an uncorrupted or, if necessary, only minimally corrupted image, even complex image content can then be reliably determined, for example, to enable the attention monitoring of a person driving the vehicle or to reliably capture complex gestures for controlling vehicle functions.

[0016] According to a further very advantageous embodiment of the interior camera according to the invention, it can now further be provided that the metal lens is designed to be switchable between at least a first state and at least a second state. In the first state, the captured image is distorted as described above; in the second state, the captured image is not distorted. This makes it possible to adapt the operation of the interior camera to different functions. For example, the first state can be present in standard operation, so that the images are distorted accordingly in order to adequately protect the privacy of the people in the vehicle interior during normal operation of the vehicle. In certain situations, however, this can be undesirable, for example if a conversation is to take place via video telephony.In this case, the metal lens can be switched to the second state, providing a clear image of the interior or the person captured by the interior camera for video telephony, and transmitting it as usual during video telephony. This ensures the protection of people during normal operation, as well as the usual functionality in situations where unadulterated images must be transmitted.

[0017] According to a very advantageous development of the interior camera according to the invention, the switchable metal lens can be designed in such a way that it can be switched by thermal influence, the application of an electric field, and / or the introduction of a front-mounted light modulator. Using such influence options, it is thus possible to switch between the first and second states accordingly, depending on the desired functionality.

[0018] A further advantageous embodiment can further provide for the first state to comprise multiple levels of corruption. This allows for varying degrees of corruption of the images, which can be preselected, for example, depending on the camera function, in order to reliably ensure certain functionalities and, at the same time, to optimally protect the privacy of the persons in the vehicle interior. Furthermore, the level of corruption, and thus ultimately the degree of privacy protection, can also be specified by the user. If the user sets the highest level of data security, this may lead to impairments in individual functions, which the user is then willing to accept.

[0019] According to a very advantageous development of this embodiment, the transfer function can be stored in encrypted form in the image processor or a memory module connected to it. This ensures that this transfer function cannot be easily read out in order to illegally convert the protected, falsified images into unfalsified images. The encrypted transfer function can, for example, be stored in a secure memory area of ​​the image processor to protect it from access by third parties.

[0020] In the interior camera according to the invention, it can further be provided that the metal lens is formed in a cover glass of the interior camera. Such a design of the metal lens in the cover glass allows for a very compact design of the interior camera, particularly in the direction of incidence of the light, since an additional conventional lens, which is mounted, for example, behind a cover glass, can be dispensed with. The metal lens can preferably be etched into the cover glass or vapor-deposited onto the cover glass.

[0021] According to another highly advantageous embodiment, the metal lens can also be combined with a metafilter, which allows only one color of the incident light to pass through to each pixel of the monochrome image sensor. The metal lens can preferably be applied to the front of a lens block, for example, etched or vapor-deposited. The metafilter can then be arranged on the back. The light can then be filtered into the individual colors red, green, blue, and infrared via it. The metafilter has significantly better properties in terms of characteristics, and in particular with regard to transmission, than polymer filters, which are typically applied to image sensors.By mounting the metafilter and the metal lens on a common lens block, significant additional costs can be saved and there is the physical advantage that these two elements are optimally aligned with each other without the need for realignment during assembly.

[0022] The metal lens itself can, as already mentioned above, be formed in a cover glass by being etched or vapor-deposited. Alternatively, it could also be designed as a metamirror or part of a metamirror, which directs an image area to be captured onto the image sensor. The mirror could, for example, be transparent to visible light and reflective to infrared light. It can, for example, be arranged on the windshield so that it can be used without requiring additional installation space and without impairing the view of a person using the vehicle. The image sensor could then, for example, be positioned below the windshield, in particular together with a display device for an overhead display or be integrated into it. This makes it possible to generate an image of the interior of the vehicle without having to position a camera that is visible to the people in the vehicle.

[0023] To improve illumination, light from an illumination source can also be projected into the camera's field of view. A metal lens can also be used here to influence the light according to the desired later imaging properties. Such an additional metal lens for illumination allows the light, for example the light from a laser or a light-emitting diode, to be projected through the metal lens precisely into the area of ​​the vehicle interior that is to be illuminated. Compared to the use of diffusers for illumination, the metal lens can be used to concentrate the radiation on the relevant area. This saves costs and energy. For example, the heads of the driver and front passenger should be illuminated much more brightly than the space between the seats in order to enable observation of the driver at night.Infrared lasers or LEDs used for this purpose can then be implemented more easily and energy-efficiently than if the entire vehicle interior had to be evenly illuminated, as is the case with diffusers. An extremely advantageous embodiment of the interior camera according to the invention provides for a trainable neural network to be implemented in the image processor for recognizing captured images depicting gestures. Such a trainable neural network can therefore identify corresponding gestures in the images and then make them available, for example, for gesture control of certain vehicle functions. The trainable neural network has the advantage that it can be pre-trained accordingly before being implemented in the vehicle, so that it already has a certain level of training to recognize the corresponding gestures.In an advantageous embodiment, this is further trained over the course of use and “gets used” to the execution of the corresponding gestures by those people who primarily use the vehicle.

[0024] According to a very advantageous further development, the neural network can be configured to recognize gestures in distorted images, whereby the neural network is trained with data from distorted images or pre-trained in the sense described above. These distorted images can be generated from unadulterated images, preferably using the transfer function of the metal lens, by calculating them from the unadulterated images through convolution. Here, the reverse approach is taken to train or pre-train the neural network: the actually captured images of unadulterated gestures are artificially distorted using the transfer function of the metal lens, so that the neural network can then recognize such gestures directly in the distorted images during subsequent operation. This saves, at least for the affected gesture control, the need to calculate back from the distorted captured images to their unadulterated content.In addition to the computing effort that can be saved, it also increases security, since the transmission functions no longer have to be stored in the image processor, preferably encrypted and secured, so that the risk of unauthorized reading by third parties can be further minimized.

[0025] If the metalens is switchable between the first and second states as described above, it can also be provided that a first neural network is trained or pre-trained with data from distorted images and a second neural network is trained or pre-trained with data from unadulterated images, so that the first neural network is activated in the first state of the metalens, and the second neural network is activated in the second state. Depending on the state of the metalens, a separate neural network is then available, optimally configured to detect gestures, regardless of whether distorted or unadulterated images reach the image processor.

[0026] Alternatively, the embodiment described above can of course also be operated with an unfolding of the falsified images into the unfalsified state, so that the image processor unfolds the falsified images in each case in order to then feed the images converted into the unfalsified state to a neural network which is trained accordingly with such unfalsified images.

[0027] Further advantageous embodiments of the interior camera according to the invention and its use also emerge from the exemplary embodiment, which is described in more detail below with reference to the figures.

[0028] Showing:

[0029] Fig. 1 shows a schematic arrangement of an interior camera in a possible embodiment according to the invention;

[0030] Fig. 2 shows a comparison of two image capture devices of an interior camera according to the prior art and an interior camera according to the invention with regard to the installation space;

[0031] Fig. 3 shows a comparison of two image capture devices of an interior camera according to the prior art and an interior camera according to the invention with regard to the passage height through the cover glass; and

[0032] Fig. 4 shows an alternative embodiment of the image capture device of the interior camera according to the invention.

[0033] The illustration in Figure 1 shows a system referred to as an interior camera 1, which is designed to capture a vehicle interior 2 (not explicitly shown). In this vehicle interior 2, a person designated by 3 is schematically indicated. An image capture device 4, purely exemplary with an optics 5 and an image sensor 6, serves to capture the vehicle interior 2 or the person 3 accordingly. The data from the image sensor 6 is transmitted to an image processor 7 connected to the image sensor 6. This image processor 7 can then, if necessary, transfer the image content it has recognized to a vehicle control system, designated here by 8, which further processes this image data or content recognized in the image data, for example recognized gestures for controlling vehicle functions, accordingly.

[0034] The interior camera 1 according to the invention now comprises a metal lens designated 9, which can be combined with a conventional optics 5 or, as will be shown later, can also replace it. The metal lens 9 is now designed as described above and configured such that, for example, only the edges of objects, in this case the person 3, in the vehicle interior 2 are retained. Any alternative filter function that obscures the image of the vehicle interior 2 captured by the image sensor 6 to such an extent that it is no longer usefully recognizable by the human eye or conventional machine person recognition would, however, also be conceivable. The metal lens 9 ultimately distorts the image of the vehicle interior 2 and thus of the person 3 to such an extent that the privacy of the person 3 is preserved and, for example, cannot be recognized by other people in the captured images.This makes it much more difficult for third parties to misuse the captured images.

[0035] Preferably, the transmission properties of the metal lens 9 can be switched accordingly via control electronics 10. The control electronics 10 can therefore switch the metal lens 9, for example, between a first state in which the image captured by the image sensor 6 is correspondingly distorted, and a second state in which this is not the case. For this purpose, for example, an electrical control signal a is transmitted from the control electronics 10 to the metal lens 9. In principle, various switching options are conceivable. On the one hand, a change in the transmission properties of the metal lens 9 can be achieved by applying an electrical and / or magnetic field. A further embodiment would be the combination of the metal lens 9 with a light modulator (not shown here) or an influence through temperature, for example by a PTC element in the edge region of the metamaterial.The switching then essentially occurs through a temperature-dependent phase transition of the material used, which is responsible for the electromagnetic resonance on the metamaterial element of the metal lens 9. Regardless of which of these options for influencing the transfer function of the metal lens 9 is used, an effect can be achieved in which the image captured by the image sensor 6 of the vehicle interior 2 and thus of the person 3 is distorted. This can also be done in individual stages. If the image should not be distorted, for example, during a video call, this can also be done according to a request from the vehicle control system 8 to the control electronics 10.Once the video telephony has ended, the system switches back from the unadulterated, clear image to the image distorted by the metal lens 9, and this information is transmitted to the image processor 7 in accordance with the signal b shown here.

[0036] Image processor 7 now offers various options. It can recalculate the image back to the unadulterated representation; this is called deconvolution if the transfer function is known. For example, with a design-specific transfer function, this transfer function can be stored in the image processor, preferably encrypted and in a secure storage location. Based on this transfer function, an unadulterated image can then be recalculated from the distorted image, which can be used to capture gestures, for example, or to determine the attention of a person driving the vehicle to the traffic situation.

[0037] Alternatively, it would also be conceivable to use a neural network in the image processor to directly recognize the content in the distorted image content. For example, to detect gestures, the neural network could be pre-trained with distorted images. These can be artificially distorted or convolved from unaltered images with a known transfer function in order to make the unrecognizable images available. The neural network is then able to recognize the gestures of the people using the vehicle through pre-training before use in the vehicle and later further training in the vehicle based on the distorted images. It is also able to increasingly recognize individual adaptations to the gestures by individual people. This way, the functionality of a gesture control system can be ensured, for example, even without first unfolding the distorted images.

[0038] When switching between distorted and unaltered image content, as described above, two neural networks could be used, for example: one for the first state with distorted images and a second for the second state with unaltered images. If different levels of distortion are used within the first state, it would also be conceivable, in principle, to have additional different neural networks for this purpose.

[0039] Alternatively, the transfer function can always be used to unfold the distorted image accordingly and convert it into an unadulterated image, which can then be recognized by a neural network that is trained or pre-trained with unadulterated images.

[0040] Intermediate solutions can also be chosen, especially when using different levels of falsification. For example, a neural network can be trained relatively easily to a relatively low level of falsification. If the person operating the vehicle now uses a very high level of falsification to maximize their privacy, the known transfer function could be used to expand from a falsification of, for example, a relatively high and secure level to a falsification of a lower level.

[0041] For example, a corruption of level 3 in a three-stage process could be reduced to a corruption of level 1, which would then make the recognition of gestures in the images corrupted by level 1 using a neural network easier and thus more reliable than if they had to be recognized in the highly corrupted level 3. Nevertheless, there are no completely uncorrupted images in any area of ​​the vehicle, unless this is explicitly desired, for example, for video telephony, so the protection of the person's privacy still remains relatively high.

[0042] In the illustration in Figure 2, on the left and designated by a), a structure according to the prior art can be seen. The image capture device 4 of the interior camera 1 is integrated into a display device 11. The display device 11 essentially consists of a display panel 12, a backlight 13 and a cover glass 14, which, in addition to the display panel 12, also covers the image capture device 4. This comprises the conventional optics 5 and an image sensor 6. In the direction of the incident light, i.e. from left to right in the illustration in Figure 2, the structure of this image capture device 4 extends significantly beyond that of the rest of the display device 11, resulting in a very large and deep structure overall.

[0043] In the illustration of Figure 2, on the right and labeled b), the alternative design with the metal lens 9 can be seen. In the embodiment shown here, this is located directly beneath the cover glass 14. The image sensor 6 is then directly connected to it, so that the image capture device 4 not only enables the properties described above thanks to the metal lens 9 but also can dispense with the conventional optics 5, thus realizing a very compact design.

[0044] The structure according to Figure 2 b) could be made even more compact if the metal lens 9 were, for example, introduced into the cover glass 4 or applied directly to it. This could be done, for example, by etching and / or vapor deposition, so that only the thickness of the cover glass 14 is required, for example with the metal lens 9 on the front side with regard to the incident light and a metafilter on the other side directly in front of the image sensor 6, which is then connected to the cover plate 14 and, if applicable, the filter. If the backlight 13 can also be omitted from the display element 11, for example by using a self-luminous display technology such as OLED or micro-LED, the result is an extremely thin structure, i.e. one that is very compact in the direction of the incident light.

[0045] Figure 3 takes up the illustration from Figure 2 again. Here, too, figure a) on the left shows the structure according to the prior art, and figure b) on the right shows the corresponding structure in an embodiment according to the invention. The conventional optics 5 in the illustration in Figure 3a) requires a certain passage height D through the cover glass 14 in order to capture a predetermined viewing angle, which is shown here by a triangle. This passage height D is of crucial importance for the construction of the camera 4 integrated into the display device 11. The cover glass 14 is required continuously in the area of ​​this passage height D, and the entire passage height D, or, viewed in plan, a circular passage opening with the diameter of the passage height D, must be free of further components.The greater the penetration height D for a given viewing angle, the more expensive the construction of the display element 11 becomes. If the construction can be shortened in the direction of the incident light by using the metal lens 9 in the illustrated embodiment directly behind the cover glass 14, this also results in a shift of the apex of the viewing angle away from the center of the conventional optics 5 to the center of the metal lens 9. The penetration height d shown in Figure 3b) is thus significantly lower than the penetration height D in the construction according to the prior art. The metal lens 9 therefore enables cost savings to be realized for the display element 11.

[0046] Figure 4 shows a further alternative embodiment of the image capture device 4 of the interior camera 1. The schematic view is a side view showing the person 3 or their head looking towards a windshield 15 of the vehicle, not shown in its entirety. In the area of ​​this windshield 15 is the metal lens 9, which here is designed in the form of a mirror. This metamirror 9 captures the person 3, as shown by the dashed light path, and reflects this image to the image sensor 6, which here is positioned, for example, on a dashboard (not shown). The metal lens 9, even if it is designed as a metamirror 9, provides the image with the intended distortion, either permanently or, in the case of a switchable metamirror 9, in the first state.

[0047] The metamirror 9 can be designed such that it is transparent to visible light but reflective to infrared light. Person 3 can thus be detected and observed in the infrared range, while the metamirror 9 does not obstruct the view in the area of ​​the windshield 15 in visible light. By redirecting the light via the metamirror 9, it can also be redirected to the image sensor 6 below the windshield, so that no camera is visible to person 3. Rather, the image sensor 6 can be integrated, for example, into the projection unit of an overhead display in order to simply and efficiently combine the interior camera 1 according to the invention and a display unit for a frequently already present overhead display into a compact and efficient unit.

Claims

Patent claims Interior camera (1) for monitoring a vehicle interior (2) with an image sensor (6) and with a metal lens (9) which interacts with the image sensor (6) for influencing the optical image, and with an image processor (7) which interacts with the image sensor (6), wherein the metal lens is designed to change optical frequencies so that an image captured by the image sensor via the metal lens is falsified by filtering out, shifting and / or distorting in such a way that it is unrecognizable for a human eye and / or machine reading device, characterized in that the image processor (7) is designed to convert a falsified image into an unfalsified state by means of a transfer function which is stored in the image processor (7) or a storage element connected to it and is assigned to the metal lens (9) and, if applicable, to the level of falsification, and to output said image.Interior camera (1) according to claim 1, characterized in that the metal lens (9) is designed to be switchable between at least a first state, which distorts the captured image, and a second state, which does not distort the captured image. Interior camera (1) according to claim 2, characterized in that the switchable metal lens (9) is designed such that it can be switched by thermal influence, application of an electric field, and / or the introduction of an upstream light modulator. Interior camera (1) according to claim 2 or 3, characterized in that the first state comprises several stages of falsification. Interior camera (1) according to claim 4, characterized in that the transfer function is stored in encrypted form in the image processor (7) or the memory element connected thereto. Interior camera (1) according to one of claims 1 to 5, characterized by a design of the metal lens (9) in a cover glass (14). Interior camera (1) according to one of claims 1 to 6, characterized in that the metal lens (9) is combined with at least one metafilter that allows only one color of the incident light to pass through to each pixel of the monochrome image sensor (6). Interior camera (1) according to one of claims 1 to 7, characterized in that a trainable neural network for recognizing captured images representing gestures is implemented in the image processor (7).Interior camera (1) according to claim 8, characterized in that the neural network is configured to recognize gestures in images distorted by the metal lens (9), wherein the neural network is trained with data from distorted images generated by convolution from undistorted images using the transfer function of the metal lens (9). Interior camera (1) according to claims 2 and 8, characterized in that a first neural network is trained with data from distorted images and a second neural network is trained with data from distorted images. A neural network is trained with data from unadulterated images, wherein the first neural network is activated in the first state of the metal lens (9) and the second neural network is activated in the second state of the metal lens (9). Interior camera (1) according to claims 5 and 8, characterized in that the image processor (7) is configured to convert the images of gestures distorted by the metal lens (9) into an unadulterated state by means of the transfer function by deconvolution and to feed these to a neural network trained with data from unadulterated images.

Images