HARDWARE SETUP FOR COMBINED IMAGES WITH AND WITHOUT INTERFERENCE ARTIFACTS FOR A VCSEL-BASED IMAGE SENSOR UNDER A DISPLAY

Abstract

An imaging system is provided comprising at least one sensor, at least one first emitter and one second emitter, and a partially reflective structure, wherein the partially reflective structure is arranged in the optical path of the sensor; wherein the partially reflective structure is arranged in the optical path of the first emitter and is configured to be at least partially transparent to first electromagnetic radiation emittable by the first emitter, wherein the second emitter is arranged such that the partially reflective structure has no or fewer reflective interfaces than with respect to the first emitter, and wherein the partially reflective structure is configured to be at least partially transparent to second electromagnetic radiation emittable by the second emitter, and wherein the sensor is configured to detect the first electromagnetic radiation and the second electromagnetic radiation.

Description

[0001] This disclosure generally relates to a 3D imaging system, such as a 3D facial recognition system.

[0002] Security systems, for example in smartphones or as door access control, can use biometric facial recognition (Face ID). Biometric facial recognition uses a dot emitter (also known as a "dot projector") that projects a grid of infrared dots onto the user's face. An infrared sensor captures the infrared dot pattern reflected from the face and compares the captured dot pattern with a previously stored dot pattern to confirm or deny access. The dot emitter can, for example, be a vertical cavity surface emitting laser (VCSEL). However, if a display, such as an organic light-emitting diode display, is placed in front of the VCSEL emitter, which emits light in the near-infrared (NIR) range, numerous interference effects can occur due to the coherent light from the VCSEL emitter.This interference can cause artifacts in the images captured by the infrared sensor, degrading their quality. Captured images with artifacts are then classified as corrupted and discarded.

[0003] A known approach to solving this problem is to capture both clean and corrupted images and then use an algorithm to clean up the artifacts in the corrupted images. A related technique uses two separate sensors to capture the corresponding clean and corrupted images. However, since the two sensors inherently have different viewing angles of the user's face, the algorithm must take these different viewing angles into account when cleaning up the corrupted images, making the process more complex.

[0004] It is an object of this invention to provide an improved 3D imaging system that can be used, for example, as a facial recognition system.

[0005] In one aspect, an imaging system is provided, comprising at least one sensor, at least a first emitter and a second emitter, a partially reflective structure, wherein the partially reflective structure is arranged in the optical path of the sensor; wherein the partially reflective structure is arranged in the optical path of the first emitter and is arranged to be at least partially transmissive to first electromagnetic radiation emittable by the first emitter, wherein the second emitter is arranged such that the partially reflective structure has no or fewer reflective interfaces than with respect to the first emitter, and .the partially reflective structure is configured to be at least partially transmissive to second electromagnetic radiation that can be emitted by the second emitter, and wherein the sensor is configured to detect the first electromagnetic radiation and the second electromagnetic radiation.

[0006] This allows both flawless and corrupted images to be captured by the same sensor. This simplifies the correction of corrupted images. This allows for an improved imaging system to provide, for example, improved facial recognition.

[0007] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis generally being placed upon illustrating the principles of the invention. In the following description, various aspects of the invention are described with reference to the following drawings, in which: Fig. 1 illustrates a 3D imaging system, and Fig. 2 images of the 3D imaging system of the Fig. 1 illustrates.

[0008] The following detailed description refers to the accompanying drawings which, by way of illustration, show specific details and aspects in which the disclosure may be practiced. One or more aspects are described in sufficient detail to enable those skilled in the art to practice the disclosure. Other aspects may be used, and structural, logical, and electrical changes may be made without departing from the scope of the disclosure. The various aspects described herein are not necessarily mutually exclusive, as some aspects may be combined with one or more other aspects to form new aspects. Various aspects are described in connection with methods, and various aspects are described in connection with devices. However, it is to be understood that aspects described in connection with methods may also apply to devices, and vice versa.In the drawings, it is noted that identical or similar elements, features, and structures are represented by identical reference numbers. In the drawings, it should be noted that proportions may not be to scale and that the size of features may be exaggerated for clarity.

[0009] Fig. 1 illustrates a schematic cross-sectional view and Fig. Figure 2 illustrates a schematic top view of an imaging system 100 according to various embodiments. The imaging system 100 includes at least one sensor 106, for example, a camera, and at least one first emitter 104 and one second emitter 110. Furthermore, the imaging system 100 includes a partially reflective structure 108, for example, a display, such as an organic light-emitting diode display.

[0010] The first emitter 104 and the second emitter 110 emit electromagnetic radiation 124-1, 126-1 into the scene of the imaging system. An object 112, for example, the face of a user, reflects at least a portion of the incident electromagnetic radiation 124-1, 126-1 toward the sensor 106. The reflected electromagnetic radiation 124-2, 126-2 is detected by the sensor 108. A portion of the electromagnetic radiation 124-3 emitted by the first emitter 104 is reflected by the partially reflective structure 108 and is incident on the sensor 108 as reflected electromagnetic radiation 124-4. The superposition of reflected electromagnetic radiations 124-2, 124-4 of the first emitter 104 from the partially reflective structure 108 or the object 112 can lead to a corrupted image of the object 112 in the sensor 110 due to interference.

[0011] The image of object 112 created by electromagnetic radiation 126-2 through second emitter 110 can be error-free because partially reflective structure 108 is not arranged in the direct optical path of second emitter 110, or is arranged with reduced reflectivity. This makes it possible to determine the proportion of the electromagnetic radiation 124-2 reflected by reflective structure 108 in the corrupted image. As a result, the image of object 112 captured by first electromagnetic radiation 124-2 can be corrected.

[0012] The reflective structure 108 is arranged in the optical path of the sensor 106 and is configured to be at least partially transparent to the electromagnetic radiation 124-2, 124-4, 126-2 detected by the sensor 106.

[0013] The reflective structure 108 is further arranged in the optical path of the first emitter 104 and is configured to be at least partially transmissive to first electromagnetic radiation 124-1, 124-2, 124-3, 124-4 emitted by the first emitter 104.

[0014] The second emitter 110 is arranged on the partially reflective structure 108 such that the reflective structure 108 is not directly in the optical path of the second emitter 110. Alternatively, the second emitter 110 may be integrated into the partially reflective structure 108 such that the reflective structure 108 has no or fewer reflective interfaces than with respect to the first emitter 104. For example, the partially reflective structure 108 may be a display with a plurality of active (sub-)pixels, and the second emitter 110 may be arranged between the (sub-)pixels. Alternatively, the second emitter 110 may extend through the reflective structure 108. For example, a lens or a cover of the second emitter 110 may extend through an opening in the partially reflective structure 108.Alternatively, the reflective structure 108 may have an opening within which the second emitter 110 is arranged, such that the direct optical path of the second emitter 110 is free of the reflective structure 108. Alternatively, the reflective structure 108 may be a display with a plurality of active (sub-)pixels, and the second emitter 110 may be the display or a (sub)set of the plurality of (sub-)pixels.

[0015] The reflective structure 108 is configured to be at least partially transmissive to the second electromagnetic radiation 126-2 emitted by the second emitter 110.

[0016] The sensor 106 is configured to detect the first electromagnetic radiation 124-2, 124-4 and the second electromagnetic radiation 126-2.

[0017] The first electromagnetic radiation 124-1, 124-3 and the second electromagnetic radiation 126-1 may be the same or different.

[0018] For example, the first electromagnetic radiation 124-1, 124-3, and the second electromagnetic radiation 126-1 may differ in wavelength. For example, one emitter of the first emitter 104 and the second emitter 110 may emit visible light, and the other emitter may emit infrared light, for example, near-infrared light. Alternatively, one emitter of the first emitter 104 and the second emitter 110 may emit near-infrared (NIR) light, and the other emitter may emit mid-infrared (MIR) light.

[0019] For example, the first electromagnetic radiation 124-1, 124-3 and the second electromagnetic radiation 126-1 may differ in at least one of the polarization, the chirality and / or the polarization direction and / or have a phase difference.

[0020] The scene of the imaging system 100 can be irradiated or illuminated alternately or simultaneously by the first emitter 104 and the second emitter 110, depending on the application. The first electromagnetic radiation 124-1, 124-3 and the second electromagnetic radiation 126-1 should be configured to be distinguishable from one another, so that the sensor 106 can assign the detected electromagnetic radiation 124-2, 124-2, 126-2 to the first emitter 104 or the second emitter 110, respectively. In the event that the first electromagnetic radiation 124-1, 124-3 is identical to the second electromagnetic radiation 126-1, the distinction can be made using different emission timings by transmitting a clock signal or trigger signal to the sensor 106 from the first emitter 104 or the second emitter 110, respectively, when they emit electromagnetic radiation (pulsed).

[0021] At least one emitter of the first emitter 104 and the second emitter 110 may be configured, for example, as a surface emitter.

[0022] The first emitter 104 and the sensor 106 can, for example, be arranged on a common substrate 102. In other words, the first emitter 104 and the sensor 106 can share a substrate 102. The substrate 102 can, for example, be a printed circuit board (PCB).

[0023] Optionally, the second emitter 110 may be arranged on the same substrate 102 as the first emitter 104 and / or the sensor 106.

[0024] Optionally, the first emitter 104, the second emitter 110 and the sensor 106 can be arranged on different substrates, for example on different circuit boards.

[0025] The first emitter 104 can be arranged at a first lateral distance a (see Fig. 2) to the sensor 106. The second emitter 110 can be arranged at a second lateral distance a (see Fig. 2) to the sensor 106. The first distance and the second distance may be identical. For example, the sensor 106 may be arranged laterally between the first emitter 104 and the second emitter 110, for example, symmetrically. Optionally, the first emitter 104 and the second emitter 110 may be arranged relative to the sensor 106 such that light from a reflective object 112 in the scene of the imaging system 100 is incident on the sensor 106 at approximately the same angle. In other words, the first distance and the second distance may be different if the first emitter 104 is arranged in a different plane relative to the sensor 106 than the second emitter 110.

[0026] The imaging system may further comprise a control device configured to capture a first image of an object 112 in the scene or the scene of the imaging system using the first emitter 104, and to capture a second image of the same object 112 in the scene or the scene of the imaging system using the second emitter 104. The control device may further be configured to determine a correction factor for one or more regions of the first image based on a difference between the first image and the second image. The control device may correct the first image based on the at least one correction factor, thereby providing a corrected first image.

[0027] The one or more regions of the first image may be contiguous or separate from each other.

[0028] The difference may be a systematic error or a systematic artifact of the first image caused by the portion of the first electromagnetic radiation 124-4 reflected by the partially reflective structure 108. Since the second image is free of this artifact or exhibits this artifact to a different extent than the first image, it is possible to determine the artifact from the difference between the images.

[0029] The difference can be determined, for example, using artificial intelligence, such as a self-learning computer algorithm.

[0030] Furthermore, a non-volatile memory may be provided, which contains instructions that cause a processor to capture a first image of an object 112 in the scene or the scene of the imaging system using the first emitter 104, and to capture a second image of the same object 112 in the scene or the scene of the imaging system using the second emitter 104. The control device may further be configured to determine a correction factor for one or more regions of the first image based on a difference between the first image and the second image.

[0031] The non-volatile memory can be any information storage device, such as a CD-ROM, a USB stick, a flash memory, cloud storage, a hard disk, server storage, etc.

[0032] This enables nearly simultaneous data retrieval. Furthermore, there is no display (e.g., a partially reflective structure) positioned on the illumination (e.g., the second emitter) and the sensor (e.g., a camera) (defect-free image). The display on the illumination and the camera can result in illumination fringes caused by the first emitter. This allows the double fringe effect to be investigated. Furthermore, the imaging system enables the elimination of misalignments between defect-free images and corrupted images (frayed data points).

[0033] In other words, the imaging system 100 clearly comprises a camera 106 (also referred to as a sensor) and two projectors 104, 108 (also referred to as emitters). The camera 106 and a VCSEL projector 104 are placed, for example, under a screen 110 (as a partially reflective substrate), which is a typical approach for facial recognition in smartphones. To illuminate the scene, the imaging system 100 comprises an LED as a second emitter 110, which is not arranged under the screen 110.

[0034] The first emitter 104 can be, for example, an EGA 2000 Flood Illuminator from OSRAM, and the second emitter 110 can be, for example, an OSLON Black IR light source from OSRAM, which is arranged on a circuit board. The sensor 108 can be, for example, a MIRA130 NIR camera.

[0035] The LED 108 and the VCSEL projector are activated alternately.

[0036] To capture error-free and corrupted (e.g., falsified) data (e.g., images) from the same viewing angle, the imaging system 100 can be configured as follows: an NIR camera 106 is placed beneath an OLED display 108. A VCSEL NIR projector 104 is placed near the camera 106 beneath an OLED display 108. An LED NIR projector 108 is placed near the camera 106 without an OLED display 108. The LED and VCSEL NIR projectors 104, 108 are activated alternately with a very short delay (typically < 5 ms). The NIR camera 106 is synchronized with both emitters 104, 108 so that the images are captured alternately. The camera 108 captures a corrupted (distorted, faulty) image corresponding to the scene illuminated by the VCSEL NIR projector 104, which has interference, and a fault-free (clean) image corresponding to the scene illuminated by the LED NIR projector 108.

[0037] The reflected illumination 126-2 from the LED NIR projector 108 is not affected by the OLED display 108, so that error-free images can be captured for reference purposes without interference.

[0038] If the display 108 has a dim viewing area, the camera 106 and the projectors 104, 108 can be placed to fit within that area.

[0039] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any example or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other examples or designs.

[0040] The words "plurality" and "multiple" in the description or claims expressly refer to a quantity greater than one. The terms "group of," "set of," "collection of," "series of," "sequence of," "grouping of," etc., and the like in the description or claims refer to a quantity equal to or greater than one, i.e., one or more. Any term expressed in the plural that is not expressly stated as "plurality" or "multiple" also refers to a quantity equal to or greater than one.

[0041] The term "connected" can be understood in the sense of a (e.g., mechanical, optical, and / or electrical) connection and / or interaction, e.g., direct or indirect. For example, multiple elements can be connected mechanically, so that they are physically held together (e.g., a plug connected to a socket), and electrically, so that they have an electrically conductive path (e.g., signal paths exist along a communication chain).

[0042] While the above descriptions and accompanying illustrations depict the components of the optical device as separate elements, those skilled in the art will appreciate the various possibilities for combining or integrating discrete optical functions into a single element. This may include combining two or more components from a single component. Conversely, those skilled in the art will recognize the possibility of dividing a single element into two or more separate elements, such as splitting a single component into two or more separate components.

[0043] It is understood that the methods described herein are exemplary and may therefore be implemented in a corresponding device. It is also understood that implementations of the devices described herein may be implemented as a corresponding method. It is therefore understood that a device corresponding to a method described herein may include one or more components configured to perform each aspect of the corresponding method.

[0044] All acronyms defined in the above description also apply to all claims contained herein.

[0045] Although the disclosure has been particularly shown and described with reference to particular embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the disclosure as defined by the appended claims. The scope of the disclosure is therefore indicated by the appended claims, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced. LIST OF REFERENCE SYMBOLS 100 imaging system 102 Substrat 104, 110 emitters 106 Sensor 108 partially reflective structure, e.g. display 112 Object in the scene of the imaging system 124-1, 124-2, 124-3, 124-4, 126-1, 126-2 electromagnetic radiation

Claims

[1] Imaging system comprising: at least one sensor, at least a first emitter and a second emitter, a partially reflective structure, wherein the partially reflective structure is arranged in the optical path of the sensor; wherein the partially reflective structure is arranged in the optical path of the first emitter and is at least partially transmissive to first electromagnetic radiation that can be emitted by the first emitter, wherein the second emitter is arranged such that the partially reflective structure has no or fewer reflective interfaces than with respect to the first emitter, and .the partially reflective structure is arranged to be at least partially permeable to second electromagnetic radiation that can be emitted by the second emitter, and wherein the sensor is configured to detect the first electromagnetic radiation and the second electromagnetic radiation. [2] The imaging system of claim 1, wherein the second emitter is disposed on the partially reflective structure. [3] Imaging system according to claim 1 or 2, wherein the sensor is a camera. [4] Imaging system according to one of claims 1 to 3, wherein the first emitter is arranged to emit electromagnetic radiation in the near-infrared range. [5] Imaging system according to one of claims 1 to 4, wherein at least one emitter of the first emitter and the second emitter is configured as a surface emitter. [6] Imaging system according to one of claims 1 to 5, further comprising a control device which is arranged to record a first image of an object in the scene or the scene of the imaging system by means of the first emitter, and to record a second image of the same object in the scene or the scene of the imaging system by means of the second emitter, wherein the control device is further configured to determine a correction factor for one or more regions of the first image based on a difference between the first image and the second image; and correct the first image based on the at least one correction factor so that a corrected first image is provided. [7] Imaging system according to one of claims 1 to 6, wherein the determination of the difference is determined by means of a self-learning computer algorithm. [8] Imaging system according to one of claims 1 to 7, wherein the scene of the imaging system is alternately illuminated by the first emitter and the second emitter. [9] Imaging system according to one of claims 1 to 8, wherein the first emitter is arranged at a first lateral distance from the sensor and the second emitter is arranged at a second lateral distance from the sensor such that electromagnetic radiation from the scene of the imaging system is incident on the sensor at approximately the same angle. [10] A smartphone comprising an imaging system according to any one of claims 1 to 9, wherein the partially reflective structure is a light-emitting display of the smartphone.

Images